Repository navigation
Run chosen SQL benchmark queries from CI and post to the PR - #11195
joseph-isaacs wants to merge 2 commits into
Conversation
Dispatching `PR SQL Benchmarks` now takes `queries` (e.g. `1,3,5-7`) and an optional `pr_number`, on top of the `benchmarks` input. With a PR number the run benchmarks that PR's head and posts the usual comparison comment to it; without one it benchmarks the dispatched ref and leaves the comparison in the run summary. A dispatched run has no `github.event.pull_request`, so the existing `fork == false` checks evaluated false and silently dropped the S3 cache, AWS, the S3-backed benchmarks and the PR comment. The matrix workflow now resolves the head commit, PR number and whether the code is trusted once, in `resolve-matrix`, and every job reads those outputs. Label and develop runs resolve to the same values as before. `pr-bench-sql.yml` looks the PR up itself, since develop and nightly callers grant no `pull-requests` permission, and refuses fork PRs: a dispatch runs with repository secrets, so forks keep using labels. A query subset gets its own comment tag, so it does not overwrite a full run's comment, and its title names the queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCEvae3twg9WNHrnma Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The previous commit rewrote every `github.event.pull_request` check on the assumption that a dispatched run fails them. It does not: GitHub expressions coerce `null == false` to true, so dispatched runs already took the trusted path. Only the PR comment broke, for lack of a PR. Restore the original checks and keep just what is needed: - `queries` input, passed to `vx-bench run --queries`. - A `find-pr` job for dispatched runs that uses `pr_number`, or else the open PR for the dispatched branch, and refuses fork PRs. Label runs still take the PR from the pull_request event. - The head commit falls back through `pr_head_sha`, the event, then `github.sha`, and the comment steps post to the resolved PR or are skipped when there is none. - A query subset keeps its own comment tag and title. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCEvae3twg9WNHrnma Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will degrade performance by 7.45%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
Summary
Stacked on #10996, which adds benchmark selection. This adds query selection, and lets a hand-dispatched run post its comparison to a PR the way label runs do. Once #10996 merges, this PR retargets to
develop.Changes
queriesinput onPR SQL Benchmarks(workflow_dispatchandworkflow_call) and onsql-bench-matrix.yml, passed tovx-bench run --queries. It takes numbers and ranges such as1,3,5-7and applies them to each selected benchmark.pull_requestevent. A dispatched run has no PR in its event, so a newfind-prjob usespr_numberif given, or else the open PR for the dispatched branch. It passes the PR number and head commit to the matrix. With no PR, the comparison only appears in the run summary; before this, the comment step failed for lack of a PR. Fork PRs are refused, because a dispatched run has the repository's secrets.pr_head_sha, then the event's head commit, thengithub.sha. The existing fork checks are unchanged: in GitHub expressionsnull == falseis true, so dispatched runs already took the trusted path.docs/developer-guide/benchmarking.md.Branch history: the first commit rewrote every
github.event.pull_requestcheck, on the wrong assumption that dispatched runs fail them. The second commit undoes that and slims the change to what is listed above. Squash on merge.Testing:
yamllint --strictpasses.typosandgit diff --checkare clean.parallel:steps that Run a single SQL benchmark from CI #10996's branch already has, which use syntax newer than 1.7.7.find-prstep: run locally against the real GitHub API. An explicit PR number,#-prefixed or not, resolves to Run a single SQL benchmark from CI #10996. A branch with an open PR resolves to that PR, a branch without one leaves the results in the run summary, a fork PR (perf(spark): append rows straight into the Arrow batch #11152) is refused, and a non-numeric value fails.parse_queries("1,3,5-7")returns[1, 3, 5, 6, 7], andvx-bench matrix pr --only tpch-nvmeselects that benchmark.compare-benchmark-jsons.pyon TPC-H q1 and q6 results taken from the real baseline. It compares the subset against the full baseline, and the title names the queries.Generated by Claude Code