Repository navigation
Show SQL benchmark comments as one row per query - #11686
Merged
Merged
Conversation
The PR benchmark comment split SQL results into one folded table per engine and format, each cell reading `PR / base / %diff`, so comparing targets on a query meant opening several tables. SQL queries now share one table: a row per query, a column per target such as `datafusion:vortex`, each cell `PR time (PR/base)` in a readable unit, bold and marked once it clears the suite's threshold, under a geomean row. Cold runs get the same table, folded. Random-access and compression results keep their existing tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A bare PR time with a ratio did not say what it was compared with. Each cell now reads `develop → PR (PR ÷ develop)`, a caption names both commits, and the geomean row compares each target's geomeans. A note explains `[scan-api]` columns when one is present, and query numbers stored as floats still count as SQL queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The hot-run table now folds by default like the cold-run table, each under a summary of its size and how many cells cleared the threshold, with the headline summary above them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn
marked this pull request as ready for review
November 20, 2026 15:07
gatesn
enabled auto-merge (squash)
November 20, 2026 15:13
Merging this PR will improve performance by 10.56%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
connortsui20
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SQL benchmark comment puts each engine and format in its own folded table, with cells that read
PR / base / %diffin raw nanoseconds. To compare targets on one query, a reader has to open several tables and line up long numbers. This PR puts SQL queries in one table: a row per query and a column per target. Columns are named the way targets are:datafusion:vortex,duckdb:parquet, and thedatafusion:vortex [scan-api]row from #11685. A caption names the develop and PR commits. Each cell readsdevelop → PR (PR ÷ develop)in a readable unit, and a change that clears the suite's threshold is bold and marked 🟢 or 🔴. Thegeomeanrow comes first and compares each target's geomean with develop's; it does not compare targets with each other.Before and after, rendered from the same synthetic TPC-H records (two of the targets):
Before: one folded table per target.
datafusion / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.876x ✅, 3↑ 0↓)
After: the hot-run table folds under a summary of its geomean (🟢 or 🔴 past the threshold, ➖ otherwise) and how many cells moved each way, with the cold-run table (same layout) folded below it. The headline summary above both is unchanged.
Hot runs: geomean 0.952x ➖ · 5 queries × 5 targets (9↑ 3↓)
Each cell is develop
base0000→ this PRpr000000(hot runs), with PR ÷ develop in parentheses: below 1.00 is faster. Changes past ±10% are bold. The geomean row compares each target's geomean over its queries.When a
[scan-api]column is present, a note says it runs the V1 scan and should be read against its own develop times. The SQL tables drop the per-row hot/cold factor. Random-access and compression results keep their per-target tables, and the headline summary is unchanged.🤖 Generated with Claude Code