Repository navigation
Benchmark Vortex scans by integration and scan engine - #11685
Conversation
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
The SQL benchmark comment puts each engine and format in its own folded table, with cells that read `PR / base / %diff` in raw nanoseconds. To compare targets on one query, a reader has to open several tables and line up long numbers. This PR puts SQL queries in one table: a row per query and a column per target. Columns are named the way targets are: `datafusion:vortex`, `duckdb:parquet`, and the `datafusion:vortex [scan-api]` row from #11685. A caption names the develop and PR commits. Each cell reads `develop → PR (PR ÷ develop)` in a readable unit, and a change that clears the suite's threshold is bold and marked 🟢 or 🔴. The `geomean` row comes first and compares each target's geomean with develop's; it does not compare targets with each other. Before and after, rendered from the same synthetic TPC-H records (two of the targets): **Before**: one folded table per target. <details> <summary>datafusion / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)</summary> | name | hot ns (PR / base / %diff) | cold ns (PR / base / %diff) | hot/cold (PR / base / %diff) | |---|---|---|---| | tpch_q01/datafusion:vortex-file-compressed | 87833278 / 90159186 / -2.6% | 125145409 / 125145409 / +0.0% | 0.70 / 0.72 / -2.6% | | tpch_q02/datafusion:vortex-file-compressed | 112659530 / 113366578 / -0.6% | 159958503 / 156822061 / +2.0% | 0.70 / 0.72 / -2.6% | | tpch_q03/datafusion:vortex-file-compressed | 136791072 / 111099878 / +23.1% 🔴 | 193062386 / 154449909 / +25.0% 🔴 | 0.71 / 0.72 / -1.5% | | tpch_q04/datafusion:vortex-file-compressed | 111880212 / 112304032 / -0.4% | 155703690 / 155703690 / +0.0% | 0.72 / 0.72 / -0.4% | | tpch_q05/datafusion:vortex-file-compressed | 200848570 / 197554972 / +1.7% | 276464572 / 276464572 / +0.0% | 0.73 / 0.71 / +1.7% | </details> <details> <summary>datafusion / parquet / ns (0.876x ✅, 3↑ 0↓)</summary> | name | hot ns (PR / base / %diff) | cold ns (PR / base / %diff) | hot/cold (PR / base / %diff) | |---|---|---|---| | tpch_q01/datafusion:parquet | 31091121 / 31149476 / -0.2% | 43681438 / 43681438 / +0.0% | 0.71 / 0.71 / -0.2% | | tpch_q02/datafusion:parquet | 94678361 / 119916454 / -21.0% 🟢 | 133927604 / 167409505 / -20.0% 🟢 | 0.71 / 0.72 / -1.3% | | tpch_q03/datafusion:parquet | 87632686 / 88181504 / -0.6% | 123322755 / 123322755 / +0.0% | 0.71 / 0.72 / -0.6% | | tpch_q04/datafusion:parquet | 156403569 / 188456318 / -17.0% 🟢 | 216479072 / 270598840 / -20.0% 🟢 | 0.72 / 0.70 / +3.7% | | tpch_q05/datafusion:parquet | 148194943 / 186915351 / -20.7% 🟢 | 210449132 / 263061415 / -20.0% 🟢 | 0.70 / 0.71 / -0.9% | </details> **After**: the hot-run table folds under a summary of its geomean (🟢 or 🔴 past the threshold, ➖ otherwise) and how many cells moved each way, with the cold-run table (same layout) folded below it. The headline summary above both is unchanged. <details> <summary>Hot runs: geomean 0.952x ➖ · 5 queries × 5 targets (9↑ 3↓)</summary> Each cell is develop `base0000` → this PR `pr000000` (hot runs), with PR ÷ develop in parentheses: below 1.00 is faster. Changes past ±10% are bold. The geomean row compares each target's geomean over its queries. | query | datafusion:vortex | datafusion:parquet | |---|--:|--:| | **geomean** | 120 → 125 ms (1.04) | **103 → 90.2 ms (0.88)** 🟢 | | tpch_q01 | 90.2 → 87.8 ms (0.97) | 31.1 → 31.1 ms (1.00) | | tpch_q02 | 113 → 113 ms (0.99) | **120 → 94.7 ms (0.79)** 🟢 | | tpch_q03 | **111 → 137 ms (1.23)** 🔴 | 88.2 → 87.6 ms (0.99) | | tpch_q04 | 112 → 112 ms (1.00) | **188 → 156 ms (0.83)** 🟢 | | tpch_q05 | 198 → 201 ms (1.02) | **187 → 148 ms (0.79)** 🟢 | </details> When a `[scan-api]` column is present, a note says it runs the V1 scan and should be read against its own develop times. The SQL tables drop the per-row hot/cold factor. Random-access and compression results keep their per-target tables, and the headline summary is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add an integration axis to benchmark targets. `datafusion:vortex [scan-api]` runs `VortexTable` over `MultiFileDataSource` on the same Vortex files as the FileOpener target, selected with `VORTEX_USE_SCAN_API=1` and `--scan-v1 true`, which develop and the Scan API stack both honour. The orchestrator appends ` [scan-api]` to the format its results record, so the rows stay apart from the default target in comparisons and the benchmarks database. The target runs the V1 scan whatever the suite's engine, because `MultiFileDataSource` opens `LayoutReader`s. It runs on ClickBench and TPC-H SF=1 NVMe in the PR presets and on develop, and on TPC-H NVMe nightly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Replace the `VORTEX_USE_SCAN_API` environment variable in `datafusion-bench` with a `--scan-api` flag, defined as #11666 defines it, and have the orchestrator pass it for Scan API targets. PR benchmarks compare against the benchmarks database rather than a base build, so the flag only needs to exist on develop once this lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
At SF=100, develop's VortexTable path could run past the job timeout, and a strict target failure would lose the default targets' results with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A Vortex benchmark target now names both axes: the integration (`file` through the engine's file format, `datasource` through DataFusion's `VortexTable`) and the scan engine (`v1` LayoutReader, `v2` plans). The scan engine was implicit per suite; suites that cannot run v2 yet are now only a constraint, not the selector. Results carry both axes as `integration` and `scan_engine` fields, and every combination but the file integration on the suite's default scan appends `[integration, scan]` to its format identity, so existing history continues. Validation rejects datasource-v2 (needs #11626) and DuckDB datasource (no table function yet). PR and develop run every valid combination on ClickBench and TPC-H SF1 NVMe; pr-full runs them on every suite, and nightly runs the other scans in companion jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A target failure fails its job before the results are written, so a new scan sharing a develop job with the default targets could lose their history. Develop now runs ClickBench and TPC-H SF1 NVMe's other scans in `-scans` jobs beside them, as nightly does; PR presets keep every scan in one job so the comment shows them side by side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
454402f to
40d9594
Compare
Every valid integration and scan engine combination now runs as a
matrix entry of its own, `{case}-{engine}-{integration}-{scan}`, beside
the case's default entry, so covering them adds machines rather than
wall time and a failing scan cannot lose another entry's results. Only
the default entry records file sizes and trees. A comment job merges a
case's entries, fetches one baseline and posts one table, naming any
entry that did not report.
`action/bench-sql` (`pr`) is back to the default targets. The new
`action/bench-scan` label runs the `pr-scan` preset: every scan on
ClickBench and TPC-H SF=1 NVMe. Develop runs every scan on those two
suites, so PR comparisons have a baseline for each; pr-full and nightly
run every scan on all of their suites.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The extra scans cost a machine each, so pull requests get them only through `action/bench-scan`; `pr-full` and `pr-all` stay at 16 entries. The comment job fetches the baseline, which needs the PR head's full history to find the newest reachable develop commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
Hot runs: geomean 0.997x ➖ · 22 queries × 7 targets (0↑ 0↓)Each cell is develop
Cold runs (first iteration): geomean 1.001x ➖ · 22 queries × 7 targets (0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
Hot runs: geomean 0.997x ➖ · 43 queries × 7 targets (0↑ 0↓)Each cell is develop
Cold runs (first iteration): geomean 0.983x ➖ · 43 queries × 7 targets (4↑ 2↓)
No file size changes detected. |
robert3005
left a comment
There was a problem hiding this comment.
Seems to be working, I think we will want to add tpc-ds (better coverage for dynamic filters) and maybe sf=10? SF=1 effectively measure constant overhead at this point
The bench's VORTEX_USE_SCAN_API fallback is dropped: develop's #11685 removed it, so only --scan-api reaches VortexTable now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Plan(replacingLayoutReaders) #9062Vortex benchmarks only measured one path per suite: DataFusion's
ListingTableand DuckDB'sMultiFileReader, on whichever scan the suite defaulted to. V1 had quietly stopped running on most suites, and DataFusion'sVortexTablewas invisible. A Vortex target now names two axes. The integration isfile(ListingTable,MultiFileReader) ordatasource(VortexTableover aDataSource), and the scan engine isv1(LayoutReader) orv2(plans). Labels readdatafusion:vortex [file, v2], and results recordintegrationandscan_engineas fields of their own. File on the suite's default scan keeps the bare format identity, so existing history continues; the other combinations append the axes, e.g.vortex-file-compressed [datasource, v1].Valid today are DataFusion file-v1, file-v2 and datasource-v1, and DuckDB file-v1 and file-v2. Validation rejects datasource-v2 until #11626 lands and DuckDB datasource until it has a table function, each with a message saying so. Suites that read list layouts run v1 only.
datafusion-benchselectsVortexTablewith--scan-api true, replacingVORTEX_USE_SCAN_API.Each combination runs as a matrix entry of its own,
{case}-{engine}-{integration}-{scan}, on its own machine beside the case's default entry. Extra scans add machines rather than wall time, and a failing scan cannot take another entry's results with it. Only the default entry records file sizes and file trees. A new comment job merges a case's entries, fetches one baseline and posts one comment, so every scan sits side by side and any entry that did not report is named.action/bench-sqlruns theprpreset as before, on the default scans only. The newaction/bench-scanlabel runspr-scan: every valid combination on ClickBench and TPC-H SF=1 NVMe, 8 machines. Its wall time is that of the default entries, around 4 minutes of benchmarking on the last PR run.action/bench-sql-extendedandaction/bench-allare unchanged. Develop runs every combination on those two suites so PR comparisons have a baseline for each, taking a develop push from 15 SQL machines to 21. Nightly runs every combination on all of its suites, 14 machines to 34. Wall time stays bounded by today's default entries. Every PR SQL comment now also waits on a small comment job, which adds about a minute.Land this before #11666 and #11664 so develop has history for every scan, and their
action/bench-scanruns show their own before and after. Theaction/bench-scanlabel still needs creating. None of the new entries or the comment job has run in CI yet. Before merging, check that thequery_measurements.formatcolumn accepts the new strings: the DDL is not in this repo, and a rejection would fail a develop entry's whole ingest.🤖 Generated with Claude Code