Skip to content

Run chosen SQL benchmark queries from CI and post to the PR - #11195

Open
joseph-isaacs wants to merge 2 commits into
ji/nice-bohr-7bkxmvfrom
ji/admiring-fermat-ppu486
Open

joseph-isaacs wants to merge 2 commits into
ji/nice-bohr-7bkxmvfrom
ji/admiring-fermat-ppu486

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

Stacked on #10996, which adds benchmark selection. This adds query selection, and lets a hand-dispatched run post its comparison to a PR the way label runs do. Once #10996 merges, this PR retargets to develop.

Changes

  • queries input on PR SQL Benchmarks (workflow_dispatch and workflow_call) and on sql-bench-matrix.yml, passed to vx-bench run --queries. It takes numbers and ranges such as 1,3,5-7 and applies them to each selected benchmark.
  • Finding the PR. Label runs still take it from the pull_request event. A dispatched run has no PR in its event, so a new find-pr job uses pr_number if given, or else the open PR for the dispatched branch. It passes the PR number and head commit to the matrix. With no PR, the comparison only appears in the run summary; before this, the comment step failed for lack of a PR. Fork PRs are refused, because a dispatched run has the repository's secrets.
  • Head commit. It now falls back through pr_head_sha, then the event's head commit, then github.sha. The existing fork checks are unchanged: in GitHub expressions null == false is true, so dispatched runs already took the trusted path.
  • Comments. A query subset gets its own comment tag, so it doesn't overwrite a full run's comment, and its title names the queries.
  • Docs in docs/developer-guide/benchmarking.md.
gh workflow run pr-bench-sql.yml --ref my-branch -f benchmarks=tpch-nvme -f queries=1,6

Branch history: the first commit rewrote every github.event.pull_request check, on the wrong assumption that dispatched runs fail them. The second commit undoes that and slims the change to what is listed above. Squash on merge.

Testing:

  • Linting: yamllint --strict passes. typos and git diff --check are clean.
  • actionlint 1.7.7: reports nothing new. Its only errors are about the parallel: steps that Run a single SQL benchmark from CI #10996's branch already has, which use syntax newer than 1.7.7.
  • find-pr step: run locally against the real GitHub API. An explicit PR number, #-prefixed or not, resolves to Run a single SQL benchmark from CI #10996. A branch with an open PR resolves to that PR, a branch without one leaves the results in the run summary, a fork PR (perf(spark): append rows straight into the Arrow batch #11152) is refused, and a non-numeric value fails.
  • Orchestrator: parse_queries("1,3,5-7") returns [1, 3, 5, 6, 7], and vx-bench matrix pr --only tpch-nvme selects that benchmark.
  • Comparison: I ran compare-benchmark-jsons.py on TPC-H q1 and q6 results taken from the real baseline. It compares the subset against the full baseline, and the title names the queries.
  • Not yet run: a real CI run on the benchmark machines.

Generated by Claude Code

joseph-isaacs and others added 2 commits November 3, 2026 15:33
Dispatching `PR SQL Benchmarks` now takes `queries` (e.g. `1,3,5-7`) and
an optional `pr_number`, on top of the `benchmarks` input. With a PR
number the run benchmarks that PR's head and posts the usual comparison
comment to it; without one it benchmarks the dispatched ref and leaves
the comparison in the run summary.

A dispatched run has no `github.event.pull_request`, so the existing
`fork == false` checks evaluated false and silently dropped the S3
cache, AWS, the S3-backed benchmarks and the PR comment. The matrix
workflow now resolves the head commit, PR number and whether the code is
trusted once, in `resolve-matrix`, and every job reads those outputs.
Label and develop runs resolve to the same values as before.

`pr-bench-sql.yml` looks the PR up itself, since develop and nightly
callers grant no `pull-requests` permission, and refuses fork PRs: a
dispatch runs with repository secrets, so forks keep using labels.

A query subset gets its own comment tag, so it does not overwrite a full
run's comment, and its title names the queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBCEvae3twg9WNHrnma
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The previous commit rewrote every `github.event.pull_request` check on
the assumption that a dispatched run fails them. It does not: GitHub
expressions coerce `null == false` to true, so dispatched runs already
took the trusted path. Only the PR comment broke, for lack of a PR.

Restore the original checks and keep just what is needed:

- `queries` input, passed to `vx-bench run --queries`.
- A `find-pr` job for dispatched runs that uses `pr_number`, or else the
  open PR for the dispatched branch, and refuses fork PRs. Label runs
  still take the PR from the pull_request event.
- The head commit falls back through `pr_head_sha`, the event, then
  `github.sha`, and the comment steps post to the resolved PR or are
  skipped when there is none.
- A query subset keeps its own comment tag and title.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBCEvae3twg9WNHrnma
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed

codspeed Bot commented Nov 4, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 7.45%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 2303 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation filter_powerlaw_by_random[10000] 586.6 ns 999.3 ns -41.29%
⚡ CPU Simulation encode_primitives[u8, (2000, 4)] 162.9 µs 134.8 µs +20.82%
⚡ CPU Simulation sixteen_vertex_lines 666.9 µs 596.6 µs +11.78%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/admiring-fermat-ppu486 (05c0fd8) with ji/nice-bohr-7bkxmv (e9ec2d5)

Open in CodSpeed

This branch was successfully deployed

1 active deployment
vercel-docs — 05c0fd86 Deployed Nov 4, 2026 by joseph-isaacs via Publish preview / Deploy #8087
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant