Skip to content

Show SQL benchmark comments as one row per query - #11686

Merged
gatesn merged 4 commits into
developfrom
ngates/bench-comment-query-rows
Nov 20, 2026
Merged

gatesn merged 4 commits into
developfrom
ngates/bench-comment-query-rows

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

The SQL benchmark comment puts each engine and format in its own folded table, with cells that read PR / base / %diff in raw nanoseconds. To compare targets on one query, a reader has to open several tables and line up long numbers. This PR puts SQL queries in one table: a row per query and a column per target. Columns are named the way targets are: datafusion:vortex, duckdb:parquet, and the datafusion:vortex [scan-api] row from #11685. A caption names the develop and PR commits. Each cell reads develop → PR (PR ÷ develop) in a readable unit, and a change that clears the suite's threshold is bold and marked 🟢 or 🔴. The geomean row comes first and compares each target's geomean with develop's; it does not compare targets with each other.

Before and after, rendered from the same synthetic TPC-H records (two of the targets):

Before: one folded table per target.

datafusion / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/datafusion:vortex-file-compressed 87833278 / 90159186 / -2.6% 125145409 / 125145409 / +0.0% 0.70 / 0.72 / -2.6%
tpch_q02/datafusion:vortex-file-compressed 112659530 / 113366578 / -0.6% 159958503 / 156822061 / +2.0% 0.70 / 0.72 / -2.6%
tpch_q03/datafusion:vortex-file-compressed 136791072 / 111099878 / +23.1% 🔴 193062386 / 154449909 / +25.0% 🔴 0.71 / 0.72 / -1.5%
tpch_q04/datafusion:vortex-file-compressed 111880212 / 112304032 / -0.4% 155703690 / 155703690 / +0.0% 0.72 / 0.72 / -0.4%
tpch_q05/datafusion:vortex-file-compressed 200848570 / 197554972 / +1.7% 276464572 / 276464572 / +0.0% 0.73 / 0.71 / +1.7%
datafusion / parquet / ns (0.876x ✅, 3↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/datafusion:parquet 31091121 / 31149476 / -0.2% 43681438 / 43681438 / +0.0% 0.71 / 0.71 / -0.2%
tpch_q02/datafusion:parquet 94678361 / 119916454 / -21.0% 🟢 133927604 / 167409505 / -20.0% 🟢 0.71 / 0.72 / -1.3%
tpch_q03/datafusion:parquet 87632686 / 88181504 / -0.6% 123322755 / 123322755 / +0.0% 0.71 / 0.72 / -0.6%
tpch_q04/datafusion:parquet 156403569 / 188456318 / -17.0% 🟢 216479072 / 270598840 / -20.0% 🟢 0.72 / 0.70 / +3.7%
tpch_q05/datafusion:parquet 148194943 / 186915351 / -20.7% 🟢 210449132 / 263061415 / -20.0% 🟢 0.70 / 0.71 / -0.9%

After: the hot-run table folds under a summary of its geomean (🟢 or 🔴 past the threshold, ➖ otherwise) and how many cells moved each way, with the cold-run table (same layout) folded below it. The headline summary above both is unchanged.

Hot runs: geomean 0.952x ➖ · 5 queries × 5 targets (9↑ 3↓)

Each cell is develop base0000 → this PR pr000000 (hot runs), with PR ÷ develop in parentheses: below 1.00 is faster. Changes past ±10% are bold. The geomean row compares each target's geomean over its queries.

query datafusion:vortex datafusion:parquet
geomean 120 → 125 ms (1.04) 103 → 90.2 ms (0.88) 🟢
tpch_q01 90.2 → 87.8 ms (0.97) 31.1 → 31.1 ms (1.00)
tpch_q02 113 → 113 ms (0.99) 120 → 94.7 ms (0.79) 🟢
tpch_q03 111 → 137 ms (1.23) 🔴 88.2 → 87.6 ms (0.99)
tpch_q04 112 → 112 ms (1.00) 188 → 156 ms (0.83) 🟢
tpch_q05 198 → 201 ms (1.02) 187 → 148 ms (0.79) 🟢

When a [scan-api] column is present, a note says it runs the V1 scan and should be read against its own develop times. The SQL tables drop the per-row hot/cold factor. Random-access and compression results keep their per-target tables, and the headline summary is unchanged.

🤖 Generated with Claude Code

The PR benchmark comment split SQL results into one folded table per
engine and format, each cell reading `PR / base / %diff`, so comparing
targets on a query meant opening several tables. SQL queries now share
one table: a row per query, a column per target such as
`datafusion:vortex`, each cell `PR time (PR/base)` in a readable unit,
bold and marked once it clears the suite's threshold, under a geomean
row. Cold runs get the same table, folded. Random-access and compression
results keep their existing tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 3 commits November 20, 2026 09:58
A bare PR time with a ratio did not say what it was compared with. Each
cell now reads `develop → PR (PR ÷ develop)`, a caption names both commits,
and the geomean row compares each target's geomeans. A note explains
`[scan-api]` columns when one is present, and query numbers stored as
floats still count as SQL queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The hot-run table now folds by default like the cold-run table, each under
a summary of its size and how many cells cleared the threshold, with the
headline summary above them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn marked this pull request as ready for review November 20, 2026 15:07
@gatesn
gatesn enabled auto-merge (squash) November 20, 2026 15:13
@codspeed

codspeed Bot commented Nov 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 10.56%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 2 improved benchmarks
✅ 1980 untouched benchmarks
⏩ 57 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime compare_int_eq_neon 4.9 µs 4.4 µs +11.11%
⚡ Walltime decode_avx512[DeltaU8] 11.1 µs 10 µs +10.02%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ngates/bench-comment-query-rows (906d995) with develop (ecff38f)

Open in CodSpeed

Footnotes

  1. 57 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@gatesn
gatesn requested a review from robert3005 November 20, 2026 16:05
@gatesn
gatesn merged commit fb5ba8a into develop Nov 20, 2026
131 checks passed
@gatesn
gatesn deleted the ngates/bench-comment-query-rows branch November 20, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants