Skip to content

Benchmark Vortex scans by integration and scan engine - #11685

Merged
gatesn merged 7 commits into
developfrom
ngates/bench-scan-api-target-4e945c
Nov 20, 2026
Merged

gatesn merged 7 commits into
developfrom
ngates/bench-scan-api-target-4e945c

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Vortex benchmarks only measured one path per suite: DataFusion's ListingTable and DuckDB's MultiFileReader, on whichever scan the suite defaulted to. V1 had quietly stopped running on most suites, and DataFusion's VortexTable was invisible. A Vortex target now names two axes. The integration is file (ListingTable, MultiFileReader) or datasource (VortexTable over a DataSource), and the scan engine is v1 (LayoutReader) or v2 (plans). Labels read datafusion:vortex [file, v2], and results record integration and scan_engine as fields of their own. File on the suite's default scan keeps the bare format identity, so existing history continues; the other combinations append the axes, e.g. vortex-file-compressed [datasource, v1].

Valid today are DataFusion file-v1, file-v2 and datasource-v1, and DuckDB file-v1 and file-v2. Validation rejects datasource-v2 until #11626 lands and DuckDB datasource until it has a table function, each with a message saying so. Suites that read list layouts run v1 only. datafusion-bench selects VortexTable with --scan-api true, replacing VORTEX_USE_SCAN_API.

Each combination runs as a matrix entry of its own, {case}-{engine}-{integration}-{scan}, on its own machine beside the case's default entry. Extra scans add machines rather than wall time, and a failing scan cannot take another entry's results with it. Only the default entry records file sizes and file trees. A new comment job merges a case's entries, fetches one baseline and posts one comment, so every scan sits side by side and any entry that did not report is named.

action/bench-sql runs the pr preset as before, on the default scans only. The new action/bench-scan label runs pr-scan: every valid combination on ClickBench and TPC-H SF=1 NVMe, 8 machines. Its wall time is that of the default entries, around 4 minutes of benchmarking on the last PR run. action/bench-sql-extended and action/bench-all are unchanged. Develop runs every combination on those two suites so PR comparisons have a baseline for each, taking a develop push from 15 SQL machines to 21. Nightly runs every combination on all of its suites, 14 machines to 34. Wall time stays bounded by today's default entries. Every PR SQL comment now also waits on a small comment job, which adds about a minute.

Land this before #11666 and #11664 so develop has history for every scan, and their action/bench-scan runs show their own before and after. The action/bench-scan label still needs creating. None of the new entries or the comment job has run in CI yet. Before merging, check that the query_measurements.format column accepts the new strings: the DDL is not in this repo, and a rejection would fail a develop entry's whole ingest.

🤖 Generated with Claude Code

@gatesn
gatesn requested a review from robert3005 November 20, 2026 16:06
@gatesn
gatesn marked this pull request as ready for review November 20, 2026 16:06
@codspeed

codspeed Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 1982 untouched benchmarks
⏩ 58 skipped benchmarks1


Comparing ngates/bench-scan-api-target-4e945c (e68a347) with develop (3a66a6b)

Open in CodSpeed

Footnotes

  1. 58 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

gatesn added a commit that referenced this pull request Nov 20, 2026
The SQL benchmark comment puts each engine and format in its own folded
table, with cells that read `PR / base / %diff` in raw nanoseconds. To
compare targets on one query, a reader has to open several tables and
line up long numbers. This PR puts SQL queries in one table: a row per
query and a column per target. Columns are named the way targets are:
`datafusion:vortex`, `duckdb:parquet`, and the `datafusion:vortex
[scan-api]` row from #11685. A caption names the develop and PR commits.
Each cell reads `develop → PR (PR ÷ develop)` in a readable unit, and a
change that clears the suite's threshold is bold and marked 🟢 or 🔴. The
`geomean` row comes first and compares each target's geomean with
develop's; it does not compare targets with each other.

Before and after, rendered from the same synthetic TPC-H records (two of
the targets):

**Before**: one folded table per target.

<details>
<summary>datafusion / vortex-file-compressed / ns (1.038x ➖, 0↑
1↓)</summary>

| name | hot ns (PR / base / %diff) | cold ns (PR / base / %diff) |
hot/cold (PR / base / %diff) |
|---|---|---|---|
| tpch_q01/datafusion:vortex-file-compressed | 87833278 / 90159186 /
-2.6% | 125145409 / 125145409 / +0.0% | 0.70 / 0.72 / -2.6% |
| tpch_q02/datafusion:vortex-file-compressed | 112659530 / 113366578 /
-0.6% | 159958503 / 156822061 / +2.0% | 0.70 / 0.72 / -2.6% |
| tpch_q03/datafusion:vortex-file-compressed | 136791072 / 111099878 /
+23.1% 🔴 | 193062386 / 154449909 / +25.0% 🔴 | 0.71 / 0.72 / -1.5% |
| tpch_q04/datafusion:vortex-file-compressed | 111880212 / 112304032 /
-0.4% | 155703690 / 155703690 / +0.0% | 0.72 / 0.72 / -0.4% |
| tpch_q05/datafusion:vortex-file-compressed | 200848570 / 197554972 /
+1.7% | 276464572 / 276464572 / +0.0% | 0.73 / 0.71 / +1.7% |

</details>
<details>
<summary>datafusion / parquet / ns (0.876x ✅, 3↑ 0↓)</summary>

| name | hot ns (PR / base / %diff) | cold ns (PR / base / %diff) |
hot/cold (PR / base / %diff) |
|---|---|---|---|
| tpch_q01/datafusion:parquet | 31091121 / 31149476 / -0.2% | 43681438 /
43681438 / +0.0% | 0.71 / 0.71 / -0.2% |
| tpch_q02/datafusion:parquet | 94678361 / 119916454 / -21.0% 🟢 |
133927604 / 167409505 / -20.0% 🟢 | 0.71 / 0.72 / -1.3% |
| tpch_q03/datafusion:parquet | 87632686 / 88181504 / -0.6% | 123322755
/ 123322755 / +0.0% | 0.71 / 0.72 / -0.6% |
| tpch_q04/datafusion:parquet | 156403569 / 188456318 / -17.0% 🟢 |
216479072 / 270598840 / -20.0% 🟢 | 0.72 / 0.70 / +3.7% |
| tpch_q05/datafusion:parquet | 148194943 / 186915351 / -20.7% 🟢 |
210449132 / 263061415 / -20.0% 🟢 | 0.70 / 0.71 / -0.9% |

</details>

**After**: the hot-run table folds under a summary of its geomean (🟢 or
🔴 past the threshold, ➖ otherwise) and how many cells moved each way,
with the cold-run table (same layout) folded below it. The headline
summary above both is unchanged.

<details>
<summary>Hot runs: geomean 0.952x ➖ · 5 queries × 5 targets (9↑
3↓)</summary>

Each cell is develop `base0000` → this PR `pr000000` (hot runs), with PR
÷ develop in parentheses: below 1.00 is faster. Changes past ±10% are
bold. The geomean row compares each target's geomean over its queries.

| query | datafusion:vortex | datafusion:parquet |
|---|--:|--:|
| **geomean** | 120 → 125 ms (1.04) | **103 → 90.2 ms (0.88)** 🟢 |
| tpch_q01 | 90.2 → 87.8 ms (0.97) | 31.1 → 31.1 ms (1.00) |
| tpch_q02 | 113 → 113 ms (0.99) | **120 → 94.7 ms (0.79)** 🟢 |
| tpch_q03 | **111 → 137 ms (1.23)** 🔴 | 88.2 → 87.6 ms (0.99) |
| tpch_q04 | 112 → 112 ms (1.00) | **188 → 156 ms (0.83)** 🟢 |
| tpch_q05 | 198 → 201 ms (1.02) | **187 → 148 ms (0.79)** 🟢 |

</details>

When a `[scan-api]` column is present, a note says it runs the V1 scan
and should be read against its own develop times. The SQL tables drop
the per-row hot/cold factor. Random-access and compression results keep
their per-target tables, and the headline summary is unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
gatesn and others added 3 commits November 20, 2026 11:29
Add an integration axis to benchmark targets. `datafusion:vortex [scan-api]`
runs `VortexTable` over `MultiFileDataSource` on the same Vortex files as the
FileOpener target, selected with `VORTEX_USE_SCAN_API=1` and `--scan-v1 true`,
which develop and the Scan API stack both honour. The orchestrator appends
` [scan-api]` to the format its results record, so the rows stay apart from
the default target in comparisons and the benchmarks database.

The target runs the V1 scan whatever the suite's engine, because
`MultiFileDataSource` opens `LayoutReader`s. It runs on ClickBench and TPC-H
SF=1 NVMe in the PR presets and on develop, and on TPC-H NVMe nightly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Replace the `VORTEX_USE_SCAN_API` environment variable in `datafusion-bench`
with a `--scan-api` flag, defined as #11666 defines it, and have the
orchestrator pass it for Scan API targets. PR benchmarks compare against the
benchmarks database rather than a base build, so the flag only needs to exist
on develop once this lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
At SF=100, develop's VortexTable path could run past the job timeout, and
a strict target failure would lose the default targets' results with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 2 commits November 20, 2026 11:42
A Vortex benchmark target now names both axes: the integration (`file`
through the engine's file format, `datasource` through DataFusion's
`VortexTable`) and the scan engine (`v1` LayoutReader, `v2` plans).
The scan engine was implicit per suite; suites that cannot run v2 yet
are now only a constraint, not the selector.

Results carry both axes as `integration` and `scan_engine` fields, and
every combination but the file integration on the suite's default scan
appends `[integration, scan]` to its format identity, so existing
history continues. Validation rejects datasource-v2 (needs #11626) and
DuckDB datasource (no table function yet).

PR and develop run every valid combination on ClickBench and TPC-H SF1
NVMe; pr-full runs them on every suite, and nightly runs the other
scans in companion jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A target failure fails its job before the results are written, so a
new scan sharing a develop job with the default targets could lose
their history. Develop now runs ClickBench and TPC-H SF1 NVMe's other
scans in `-scans` jobs beside them, as nightly does; PR presets keep
every scan in one job so the comment shows them side by side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/bench-scan-api-target-4e945c branch from 454402f to 40d9594 Compare November 20, 2026 16:47
@gatesn gatesn changed the title Benchmark DataFusion through the Scan API beside FileOpener Benchmark every Vortex integration and scan engine combination Nov 20, 2026
gatesn and others added 2 commits November 20, 2026 11:52
Every valid integration and scan engine combination now runs as a
matrix entry of its own, `{case}-{engine}-{integration}-{scan}`, beside
the case's default entry, so covering them adds machines rather than
wall time and a failing scan cannot lose another entry's results. Only
the default entry records file sizes and trees. A comment job merges a
case's entries, fetches one baseline and posts one table, naming any
entry that did not report.

`action/bench-sql` (`pr`) is back to the default targets. The new
`action/bench-scan` label runs the `pr-scan` preset: every scan on
ClickBench and TPC-H SF=1 NVMe. Develop runs every scan on those two
suites, so PR comparisons have a baseline for each; pr-full and nightly
run every scan on all of their suites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The extra scans cost a machine each, so pull requests get them only
through `action/bench-scan`; `pr-full` and `pr-all` stay at 16 entries.
The comment job fetches the baseline, which needs the PR head's full
history to find the newest reachable develop commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn changed the title Benchmark every Vortex integration and scan engine combination Benchmark Vortex scans by integration and scan engine Nov 20, 2026
@robert3005 robert3005 added the action/bench-scan Run scan benchmarks label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-scan Run scan benchmarks label Nov 20, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done e68a347 clickbench-nvme-duckdb-file-v1 1 Explore Profiling Data
🟢 Done e68a347 clickbench-nvme-datafusion-file-v1 1 Explore Profiling Data
🟢 Done e68a347 clickbench-nvme-datafusion-datasource-v1 1 Explore Profiling Data
🟢 Done e68a347 clickbench-nvme 1 Explore Profiling Data
🟢 Done e68a347 tpch-nvme-datafusion-file-v1 1 Explore Profiling Data
🟢 Done e68a347 tpch-nvme-datafusion-datasource-v1 1 Explore Profiling Data
🟢 Done e68a347 tpch-nvme-duckdb-file-v1 1 Explore Profiling Data
🟢 Done e68a347 tpch-nvme 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: TPC-H SF=1 on NVME 📖

Commits: PR e68a347f vs base fb5ba8af
Verdict: No clear signal (low confidence)
Attributed Vortex impact: +0.6%
Engines: DataFusion No clear signal (+0.3%, low confidence) · DuckDB No clear signal (+0.9%, low confidence)
Vortex (geomean): hot 1.000x ➖ · cold 1.002x ➖
Parquet (geomean): hot 0.994x ➖ · cold 1.000x ➖
Shifts: Parquet (control) -0.6% · Median polish -0.3%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
  • Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
  • Query tables: One row per query and one column per target. Each cell reads develop → PR (PR ÷ develop), so (0.80) is 20% faster than develop. A change that clears this suite's threshold is bold and marked 🔴 slower or 🟢 faster; anything smaller is noise here and is left plain. The geomean row compares each target's geomean over its queries with develop's. Hot runs are shown first and cold runs are folded below them.
  • Other tables: Each cell reads PR / base / %diff, marked the same way.

Hot runs: geomean 0.997x ➖ · 22 queries × 7 targets (0↑ 0↓)

Each cell is develop fb5ba8af → this PR e68a347f (hot runs), with PR ÷ develop in parentheses: below 1.00 is faster. Changes past ±10% are bold. The geomean row compares each target's geomean over its queries.
Vortex columns name the integration and the scan engine: file reads through the engine's file format (DataFusion ListingTable, DuckDB MultiFileReader), datasource through DataFusion's VortexTable over a Vortex DataSource; v1 and v2 are the LayoutReader and plan scans. Each column compares with its own develop times.

query datafusion:vortex [file, v2] datafusion:vortex [file, v1] datafusion:vortex [datasource, v1] datafusion:parquet duckdb:vortex [file, v2] duckdb:vortex [file, v1] duckdb:parquet
geomean 20.1 → 20.1 ms (1.00) — → 21.2 ms — → 63.9 ms 77.8 → 77.7 ms (1.00) 15.5 → 15.5 ms (1.00) — → 15.6 ms 56.3 → 55.7 ms (0.99)
tpch_q01 15.3 → 15.7 ms (1.03) — → 16.2 ms — → 254 ms 71.3 → 71.2 ms (1.00) 10.0 → 10.1 ms (1.01) — → 9.53 ms 47.6 → 47.3 ms (0.99)
tpch_q02 17.4 → 17.7 ms (1.02) — → 17.7 ms — → 37.0 ms 75.7 → 75.7 ms (1.00) 15.3 → 15.6 ms (1.02) — → 14.8 ms 48.4 → 48.2 ms (0.99)
tpch_q03 19.7 → 19.1 ms (0.97) — → 19.3 ms — → 52.4 ms 81.8 → 81.3 ms (0.99) 18.9 → 18.4 ms (0.98) — → 18.8 ms 60.3 → 59.9 ms (0.99)
tpch_q04 10.4 → 10.1 ms (0.97) — → 9.76 ms — → 56.8 ms 39.5 → 39.7 ms (1.01) 12.5 → 12.2 ms (0.97) — → 11.7 ms 34.8 → 34.6 ms (0.99)
tpch_q05 37.9 → 37.6 ms (0.99) — → 37.6 ms — → 162 ms 111 → 110 ms (1.00) 20.4 → 20.1 ms (0.98) — → 20.4 ms 67.3 → 66.7 ms (0.99)
tpch_q06 4.51 → 4.48 ms (0.99) — → 5.99 ms — → 7.43 ms 31.6 → 31.3 ms (0.99) 4.43 → 4.78 ms (1.08) — → 4.88 ms 21.7 → 21.5 ms (0.99)
tpch_q07 52.6 → 53.1 ms (1.01) — → 52.1 ms — → 192 ms 113 → 113 ms (1.00) 24.1 → 24.4 ms (1.01) — → 24.0 ms 64.2 → 63.6 ms (0.99)
tpch_q08 52.4 → 52.2 ms (1.00) — → 52.2 ms — → 89.1 ms 117 → 117 ms (1.00) 20.7 → 20.9 ms (1.01) — → 21.2 ms 75.9 → 75.5 ms (0.99)
tpch_q09 47.8 → 47.3 ms (0.99) — → 49.6 ms — → 193 ms 137 → 137 ms (1.00) 22.8 → 22.1 ms (0.97) — → 22.2 ms 108 → 107 ms (0.99)
tpch_q10 17.5 → 18.1 ms (1.03) — → 18.5 ms — → 85.9 ms 103 → 104 ms (1.01) 28.3 → 28.1 ms (0.99) — → 27.9 ms 107 → 106 ms (0.99)
tpch_q11 15.0 → 14.7 ms (0.99) — → 15.1 ms — → 28.4 ms 61.4 → 62.4 ms (1.02) 8.08 → 8.15 ms (1.01) — → 7.83 ms 29.5 → 29.3 ms (0.99)
tpch_q12 18.7 → 18.9 ms (1.01) — → 22.2 ms — → 47.1 ms 64.3 → 64.6 ms (1.01) 14.2 → 14.5 ms (1.03) — → 15.6 ms 37.4 → 37.2 ms (1.00)
tpch_q13 19.8 → 18.8 ms (0.95) — → 19.1 ms — → 35.4 ms 144 → 143 ms (1.00) 19.7 → 19.0 ms (0.96) — → 20.1 ms 175 → 174 ms (1.00)
tpch_q14 14.4 → 14.2 ms (0.99) — → 14.8 ms — → 23.0 ms 47.9 → 47.7 ms (1.00) 11.1 → 11.1 ms (1.00) — → 11.3 ms 42.5 → 42.2 ms (0.99)
tpch_q15 18.3 → 18.4 ms (1.00) — → 18.8 ms — → 36.8 ms 73.9 → 74.4 ms (1.01) 8.39 → 8.25 ms (0.98) — → 8.34 ms 32.8 → 32.6 ms (0.99)
tpch_q16 18.1 → 18.5 ms (1.02) — → 18.1 ms — → 31.1 ms 43.2 → 43.7 ms (1.01) 16.5 → 16.4 ms (0.99) — → 16.7 ms 48.1 → 46.9 ms (0.98)
tpch_q17 29.3 → 29.6 ms (1.01) — → 28.3 ms — → 216 ms 124 → 123 ms (0.99) 12.9 → 12.8 ms (0.99) — → 13.3 ms 48.1 → 47.9 ms (1.00)
tpch_q18 36.2 → 36.1 ms (1.00) — → 36.4 ms — → 278 ms 121 → 121 ms (0.99) 24.7 → 24.2 ms (0.98) — → 25.4 ms 74.5 → 73.8 ms (0.99)
tpch_q19 9.58 → 9.80 ms (1.02) — → 15.0 ms — → 17.8 ms 57.0 → 56.6 ms (0.99) 11.4 → 11.6 ms (1.03) — → 11.8 ms 50.5 → 48.1 ms (0.95)
tpch_q20 20.9 → 21.0 ms (1.00) — → 20.9 ms — → 51.6 ms 78.6 → 78.1 ms (0.99) 14.5 → 14.5 ms (1.00) — → 14.6 ms 63.8 → 63.3 ms (0.99)
tpch_q21 34.9 → 35.8 ms (1.03) — → 39.2 ms — → 226 ms 114 → 113 ms (0.99) 48.2 → 48.1 ms (1.00) — → 48.0 ms 107 → 106 ms (0.99)
tpch_q22 10.7 → 10.9 ms (1.02) — → 12.1 ms — → 35.0 ms 52.0 → 51.3 ms (0.99) 16.9 → 16.8 ms (0.99) — → 16.8 ms 50.6 → 50.4 ms (1.00)
Cold runs (first iteration): geomean 1.001x ➖ · 22 queries × 7 targets (0↑ 0↓)
query datafusion:vortex [file, v2] datafusion:vortex [file, v1] datafusion:vortex [datasource, v1] datafusion:parquet duckdb:vortex [file, v2] duckdb:vortex [file, v1] duckdb:parquet
geomean 47.0 → 47.2 ms (1.00) — → 46.5 ms — → 91.6 ms 100 → 101 ms (1.00) 29.2 → 29.1 ms (1.00) — → 28.3 ms 65.0 → 64.7 ms (1.00)
tpch_q01 44.8 → 43.4 ms (0.97) — → 44.1 ms — → 272 ms 89.1 → 89.7 ms (1.01) 23.2 → 21.8 ms (0.94) — → 20.0 ms 54.3 → 54.1 ms (1.00)
tpch_q02 39.6 → 38.8 ms (0.98) — → 35.7 ms — → 53.6 ms 92.5 → 94.5 ms (1.02) 25.1 → 24.9 ms (0.99) — → 24.6 ms 55.8 → 55.5 ms (0.99)
tpch_q03 44.6 → 47.9 ms (1.07) — → 45.4 ms — → 78.5 ms 105 → 103 ms (0.98) 34.6 → 36.0 ms (1.04) — → 34.1 ms 68.1 → 69.0 ms (1.01)
tpch_q04 34.7 → 35.2 ms (1.01) — → 31.5 ms — → 73.1 ms 58.3 → 58.5 ms (1.00) 25.5 → 25.3 ms (0.99) — → 24.7 ms 43.3 → 43.3 ms (1.00)
tpch_q05 66.1 → 66.0 ms (1.00) — → 66.9 ms — → 191 ms 139 → 138 ms (0.99) 36.8 → 36.2 ms (0.98) — → 37.9 ms 78.1 → 76.6 ms (0.98)
tpch_q06 26.2 → 26.5 ms (1.01) — → 27.4 ms — → 22.8 ms 48.3 → 48.4 ms (1.00) 15.0 → 14.3 ms (0.95) — → 14.4 ms 27.7 → 27.4 ms (0.99)
tpch_q07 81.8 → 81.0 ms (0.99) — → 79.0 ms — → 221 ms 138 → 138 ms (1.00) 39.9 → 41.3 ms (1.04) — → 40.1 ms 73.9 → 72.8 ms (0.99)
tpch_q08 84.0 → 83.9 ms (1.00) — → 78.8 ms — → 114 ms 143 → 143 ms (1.00) 37.5 → 37.2 ms (0.99) — → 36.4 ms 86.9 → 86.8 ms (1.00)
tpch_q09 80.3 → 76.5 ms (0.95) — → 80.8 ms — → 221 ms 167 → 166 ms (0.99) 40.1 → 39.5 ms (0.99) — → 38.0 ms 120 → 119 ms (0.99)
tpch_q10 45.5 → 49.2 ms (1.08) — → 46.9 ms — → 119 ms 129 → 129 ms (1.00) 45.9 → 45.4 ms (0.99) — → 42.6 ms 118 → 117 ms (0.99)
tpch_q11 33.9 → 35.7 ms (1.05) — → 32.0 ms — → 47.7 ms 80.5 → 82.4 ms (1.02) 16.8 → 18.0 ms (1.07) — → 16.8 ms 34.7 → 34.8 ms (1.00)
tpch_q12 45.0 → 41.6 ms (0.92) — → 43.6 ms — → 69.4 ms 85.3 → 85.0 ms (1.00) 25.5 → 27.9 ms (1.09) — → 27.7 ms 44.5 → 44.7 ms (1.00)
tpch_q13 41.3 → 41.6 ms (1.01) — → 42.9 ms — → 50.5 ms 162 → 165 ms (1.02) 31.1 → 32.1 ms (1.03) — → 31.4 ms 183 → 183 ms (1.00)
tpch_q14 38.1 → 38.9 ms (1.02) — → 39.4 ms — → 44.7 ms 70.6 → 70.7 ms (1.00) 24.1 → 24.5 ms (1.02) — → 22.3 ms 51.3 → 50.7 ms (0.99)
tpch_q15 45.7 → 46.5 ms (1.02) — → 45.0 ms — → 54.6 ms 93.7 → 94.4 ms (1.01) 20.1 → 21.3 ms (1.06) — → 20.4 ms 41.4 → 41.5 ms (1.00)
tpch_q16 39.4 → 40.0 ms (1.02) — → 38.0 ms — → 49.4 ms 60.0 → 61.4 ms (1.02) 23.1 → 23.4 ms (1.01) — → 23.1 ms 53.3 → 52.0 ms (0.98)
tpch_q17 57.6 → 58.7 ms (1.02) — → 58.3 ms — → 241 ms 145 → 146 ms (1.01) 28.5 → 28.3 ms (0.99) — → 25.8 ms 57.2 → 58.1 ms (1.02)
tpch_q18 58.7 → 57.0 ms (0.97) — → 56.1 ms — → 307 ms 139 → 143 ms (1.03) 34.1 → 33.5 ms (0.98) — → 34.0 ms 79.1 → 79.6 ms (1.01)
tpch_q19 38.3 → 36.2 ms (0.95) — → 41.8 ms — → 38.8 ms 83.8 → 81.2 ms (0.97) 26.7 → 25.1 ms (0.94) — → 25.7 ms 59.3 → 57.9 ms (0.98)
tpch_q20 44.9 → 48.2 ms (1.07) — → 44.7 ms — → 74.2 ms 104 → 102 ms (0.98) 32.2 → 30.4 ms (0.94) — → 28.4 ms 74.1 → 74.2 ms (1.00)
tpch_q21 65.7 → 64.6 ms (0.98) — → 66.7 ms — → 266 ms 138 → 139 ms (1.01) 64.8 → 62.7 ms (0.97) — → 61.9 ms 117 → 115 ms (0.98)
tpch_q22 30.7 → 31.5 ms (1.03) — → 30.3 ms — → 52.2 ms 67.2 → 69.2 ms (1.03) 26.9 → 26.4 ms (0.98) — → 26.7 ms 59.4 → 58.8 ms (0.99)

No file size changes detected.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Clickbench on NVME 📖

Commits: PR e68a347f vs base fb5ba8af
Verdict: No clear signal (low confidence)
Attributed Vortex impact: -0.5%
Engines: DataFusion No clear signal (-0.1%, low confidence) · DuckDB No clear signal (-0.9%, low confidence)
Vortex (geomean): hot 0.994x ➖ · cold 0.983x ➖
Parquet (geomean): hot 0.999x ➖ · cold 0.982x ➖
Shifts: Parquet (control) -0.1% · Median polish -0.4%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
  • Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
  • Query tables: One row per query and one column per target. Each cell reads develop → PR (PR ÷ develop), so (0.80) is 20% faster than develop. A change that clears this suite's threshold is bold and marked 🔴 slower or 🟢 faster; anything smaller is noise here and is left plain. The geomean row compares each target's geomean over its queries with develop's. Hot runs are shown first and cold runs are folded below them.
  • Other tables: Each cell reads PR / base / %diff, marked the same way.

Hot runs: geomean 0.997x ➖ · 43 queries × 7 targets (0↑ 0↓)

Each cell is develop fb5ba8af → this PR e68a347f (hot runs), with PR ÷ develop in parentheses: below 1.00 is faster. Changes past ±10% are bold. The geomean row compares each target's geomean over its queries.
Vortex columns name the integration and the scan engine: file reads through the engine's file format (DataFusion ListingTable, DuckDB MultiFileReader), datasource through DataFusion's VortexTable over a Vortex DataSource; v1 and v2 are the LayoutReader and plan scans. Each column compares with its own develop times.

query datafusion:vortex [file, v2] datafusion:vortex [file, v1] datafusion:vortex [datasource, v1] datafusion:parquet duckdb:vortex [file, v2] duckdb:vortex [file, v1] duckdb:parquet
geomean 41.4 → 41.1 ms (0.99) — → 42.3 ms — → 52.4 ms 65.0 → 64.8 ms (1.00) 35.9 → 35.7 ms (0.99) — → 34.8 ms 66.9 → 67.0 ms (1.00)
clickbench_q00 1.21 → 1.30 ms (1.07) — → 1.24 ms — → 2.49 ms 1.23 → 1.24 ms (1.01) 2.21 → 2.17 ms (0.98) — → 2.20 ms 9.37 → 9.24 ms (0.99)
clickbench_q01 8.89 → 9.14 ms (1.03) — → 8.68 ms — → 7.62 ms 16.0 → 16.2 ms (1.01) 7.00 → 7.04 ms (1.01) — → 7.08 ms 15.0 → 15.1 ms (1.01)
clickbench_q02 13.1 → 13.4 ms (1.03) — → 12.6 ms — → 9.55 ms 18.3 → 18.9 ms (1.04) 9.28 → 9.38 ms (1.01) — → 7.99 ms 22.2 → 22.4 ms (1.01)
clickbench_q03 14.0 → 14.1 ms (1.00) — → 13.4 ms — → 9.61 ms 17.3 → 16.7 ms (0.96) 13.9 → 13.9 ms (1.00) — → 13.6 ms 23.0 → 22.8 ms (0.99)
clickbench_q04 59.9 → 59.2 ms (0.99) — → 57.7 ms — → 69.9 ms 65.4 → 63.5 ms (0.97) 58.2 → 55.9 ms (0.96) — → 54.7 ms 61.7 → 61.2 ms (0.99)
clickbench_q05 81.0 → 81.7 ms (1.01) — → 78.6 ms — → 115 ms 88.5 → 87.8 ms (0.99) 65.2 → 64.0 ms (0.98) — → 62.0 ms 81.7 → 83.1 ms (1.02)
clickbench_q06 1.27 → 1.26 ms (0.99) — → 1.27 ms — → 8.75 ms 1.23 → 1.27 ms (1.04) 2.64 → 2.58 ms (0.98) — → 2.67 ms 20.4 → 20.1 ms (0.98)
clickbench_q07 12.3 → 12.3 ms (1.00) — → 12.2 ms — → 13.9 ms 16.8 → 16.7 ms (1.00) 9.56 → 8.95 ms (0.94) — → 8.32 ms 17.0 → 16.9 ms (0.99)
clickbench_q08 82.2 → 82.2 ms (1.00) — → 78.6 ms — → 99.8 ms 85.3 → 84.4 ms (0.99) 71.0 → 70.0 ms (0.99) — → 66.7 ms 75.4 → 75.0 ms (0.99)
clickbench_q09 175 → 173 ms (0.99) — → 175 ms — → 183 ms 185 → 186 ms (1.01) 94.8 → 94.4 ms (1.00) — → 88.8 ms 102 → 101 ms (0.99)
clickbench_q10 24.7 → 24.6 ms (1.00) — → 24.2 ms — → 26.0 ms 36.4 → 36.2 ms (0.99) 27.7 → 27.3 ms (0.98) — → 26.1 ms 42.0 → 41.5 ms (0.99)
clickbench_q11 27.2 → 26.5 ms (0.97) — → 26.8 ms — → 29.6 ms 43.3 → 42.6 ms (0.98) 30.5 → 30.1 ms (0.99) — → 28.8 ms 46.8 → 46.9 ms (1.00)
clickbench_q12 75.4 → 75.0 ms (0.99) — → 74.2 ms — → 100.0 ms 95.7 → 95.6 ms (1.00) 70.6 → 69.2 ms (0.98) — → 68.1 ms 92.0 → 92.6 ms (1.01)
clickbench_q13 119 → 121 ms (1.02) — → 121 ms — → 124 ms 147 → 147 ms (1.00) 119 → 119 ms (1.00) — → 116 ms 150 → 150 ms (1.00)
clickbench_q14 80.4 → 79.5 ms (0.99) — → 78.8 ms — → 105 ms 102 → 101 ms (0.99) 71.9 → 72.8 ms (1.01) — → 72.6 ms 101 → 100 ms (0.99)
clickbench_q15 68.1 → 66.1 ms (0.97) — → 63.5 ms — → 80.4 ms 72.5 → 72.5 ms (1.00) 76.8 → 76.0 ms (0.99) — → 74.6 ms 86.9 → 86.8 ms (1.00)
clickbench_q16 157 → 160 ms (1.02) — → 154 ms — → 211 ms 157 → 157 ms (1.00) 137 → 139 ms (1.01) — → 132 ms 164 → 161 ms (0.98)
clickbench_q17 157 → 158 ms (1.00) — → 156 ms — → 208 ms 157 → 157 ms (1.00) 141 → 136 ms (0.96) — → 135 ms 163 → 167 ms (1.03)
clickbench_q18 296 → 300 ms (1.01) — → 289 ms — → 329 ms 313 → 306 ms (0.98) 254 → 262 ms (1.03) — → 258 ms 293 → 293 ms (1.00)
clickbench_q19 9.76 → 9.98 ms (1.02) — → 9.26 ms — → 9.21 ms 19.0 → 19.2 ms (1.01) 9.35 → 9.10 ms (0.97) — → 9.26 ms 20.1 → 19.9 ms (0.99)
clickbench_q20 46.1 → 45.2 ms (0.98) — → 44.9 ms — → 41.8 ms 178 → 178 ms (1.00) 51.5 → 50.6 ms (0.98) — → 49.2 ms 182 → 188 ms (1.03)
clickbench_q21 58.6 → 56.9 ms (0.97) — → 91.8 ms — → 84.9 ms 175 → 173 ms (0.99) 58.0 → 57.7 ms (0.99) — → 85.9 ms 223 → 217 ms (0.97)
clickbench_q22 74.6 → 72.8 ms (0.98) — → 99.0 ms — → 93.1 ms 263 → 260 ms (0.99) 90.9 → 90.6 ms (1.00) — → 135 ms 308 → 315 ms (1.03)
clickbench_q23 106 → 100 ms (0.95) — → 126 ms — → 170 ms 1.03 → 1.01 s (0.99) 63.9 → 64.7 ms (1.01) — → 36.3 ms 172 → 176 ms (1.02)
clickbench_q24 9.12 → 8.87 ms (0.97) — → 15.4 ms — → 13.2 ms 29.8 → 29.8 ms (1.00) 16.8 → 17.3 ms (1.03) — → 13.5 ms 29.1 → 30.1 ms (1.03)
clickbench_q25 20.5 → 20.4 ms (1.00) — → 19.0 ms — → 17.1 ms 36.3 → 35.7 ms (0.98) 19.6 → 19.4 ms (0.99) — → 18.6 ms 41.5 → 41.5 ms (1.00)
clickbench_q26 15.8 → 14.8 ms (0.94) — → 15.9 ms — → 17.5 ms 30.4 → 29.9 ms (0.98) 14.0 → 14.1 ms (1.01) — → 11.3 ms 29.9 → 30.0 ms (1.00)
clickbench_q27 104 → 102 ms (0.98) — → 98.1 ms — → 100 ms 194 → 192 ms (0.99) 51.2 → 51.4 ms (1.00) — → 49.4 ms 201 → 196 ms (0.98)
clickbench_q28 565 → 564 ms (1.00) — → 574 ms — → 850 ms 619 → 619 ms (1.00) 519 → 515 ms (0.99) — → 503 ms 971 → 947 ms (0.97)
clickbench_q29 26.5 → 26.6 ms (1.00) — → 25.3 ms — → 21.8 ms 29.0 → 29.4 ms (1.01) 17.3 → 17.2 ms (0.99) — → 16.4 ms 23.8 → 24.1 ms (1.02)
clickbench_q30 61.9 → 59.4 ms (0.96) — → 61.2 ms — → 84.6 ms 89.7 → 89.4 ms (1.00) 60.7 → 60.2 ms (0.99) — → 58.6 ms 106 → 105 ms (0.99)
clickbench_q31 69.9 → 69.7 ms (1.00) — → 69.2 ms — → 77.1 ms 103 → 102 ms (0.99) 111 → 109 ms (0.99) — → 108 ms 148 → 159 ms (1.07)
clickbench_q32 296 → 291 ms (0.98) — → 288 ms — → 294 ms 310 → 304 ms (0.98) 369 → 361 ms (0.98) — → 376 ms 390 → 387 ms (0.99)
clickbench_q33 323 → 329 ms (1.02) — → 327 ms — → 464 ms 375 → 377 ms (1.01) 342 → 347 ms (1.02) — → 338 ms 470 → 471 ms (1.00)
clickbench_q34 326 → 324 ms (1.00) — → 330 ms — → 493 ms 380 → 375 ms (0.99) 357 → 356 ms (1.00) — → 349 ms 464 → 468 ms (1.01)
clickbench_q35 61.8 → 61.3 ms (0.99) — → 59.4 ms — → 83.8 ms 63.8 → 63.5 ms (1.00) 59.2 → 57.9 ms (0.98) — → 56.9 ms 61.1 → 61.6 ms (1.01)
clickbench_q36 36.9 → 37.1 ms (1.00) — → 37.3 ms — → 44.4 ms 74.1 → 73.6 ms (0.99) 11.2 → 11.2 ms (1.00) — → 11.3 ms 30.6 → 30.4 ms (0.99)
clickbench_q37 21.2 → 21.1 ms (1.00) — → 21.9 ms — → 28.2 ms 35.6 → 35.6 ms (1.00) 7.75 → 7.68 ms (0.99) — → 7.64 ms 23.4 → 23.2 ms (0.99)
clickbench_q38 13.1 → 12.8 ms (0.98) — → 11.8 ms — → 24.4 ms 48.4 → 47.8 ms (0.99) 8.03 → 8.27 ms (1.03) — → 8.33 ms 23.3 → 23.7 ms (1.02)
clickbench_q39 78.6 → 78.9 ms (1.00) — → 77.2 ms — → 107 ms 144 → 143 ms (0.99) 18.2 → 17.9 ms (0.98) — → 18.2 ms 52.7 → 53.3 ms (1.01)
clickbench_q40 10.3 → 10.6 ms (1.03) — → 11.2 ms — → 18.6 ms 19.9 → 19.9 ms (1.00) 7.15 → 6.87 ms (0.96) — → 6.78 ms 15.0 → 14.6 ms (0.97)
clickbench_q41 9.64 → 9.29 ms (0.96) — → 10.7 ms — → 17.2 ms 18.7 → 18.6 ms (0.99) 7.74 → 7.91 ms (1.02) — → 7.66 ms 15.3 → 15.7 ms (1.02)
clickbench_q42 9.85 → 9.78 ms (0.99) — → 10.8 ms — → 16.4 ms 19.4 → 19.3 ms (0.99) 7.10 → 7.14 ms (1.01) — → 7.14 ms 14.0 → 14.4 ms (1.03)
Cold runs (first iteration): geomean 0.983x ➖ · 43 queries × 7 targets (4↑ 2↓)
query datafusion:vortex [file, v2] datafusion:vortex [file, v1] datafusion:vortex [datasource, v1] datafusion:parquet duckdb:vortex [file, v2] duckdb:vortex [file, v1] duckdb:parquet
geomean 108 → 105 ms (0.97) — → 104 ms — → 116 ms 159 → 157 ms (0.99) 84.3 → 83.7 ms (0.99) — → 83.4 ms 84.1 → 82.3 ms (0.98)
clickbench_q00 23.7 → 22.8 ms (0.96) — → 24.6 ms — → 26.2 ms 50.9 → 50.3 ms (0.99) 24.6 → 24.3 ms (0.99) — → 24.0 ms 14.1 → 13.9 ms (0.99)
clickbench_q01 38.0 → 38.4 ms (1.01) — → 37.8 ms — → 30.8 ms 70.9 → 73.8 ms (1.04) 25.8 → 28.5 ms (1.10) 🔴 — → 28.1 ms 20.3 → 20.9 ms (1.03)
clickbench_q02 50.9 → 48.3 ms (0.95) — → 40.8 ms — → 33.3 ms 80.2 → 80.6 ms (1.01) 36.5 → 34.8 ms (0.95) — → 34.4 ms 31.3 → 31.3 ms (1.00)
clickbench_q03 51.8 → 50.5 ms (0.98) — → 46.5 ms — → 45.0 ms 80.9 → 76.1 ms (0.94) 46.5 → 43.5 ms (0.94) — → 54.2 ms 39.7 → 33.2 ms (0.84) 🟢
clickbench_q04 102 → 102 ms (1.00) — → 90.5 ms — → 106 ms 126 → 124 ms (0.98) 90.5 → 86.8 ms (0.96) — → 89.5 ms 75.1 → 73.5 ms (0.98)
clickbench_q05 135 → 125 ms (0.93) — → 121 ms — → 206 ms 154 → 154 ms (1.00) 95.2 → 92.7 ms (0.97) — → 89.6 ms 94.8 → 94.7 ms (1.00)
clickbench_q06 24.7 → 23.4 ms (0.95) — → 24.4 ms — → 30.3 ms 49.9 → 50.1 ms (1.00) 32.9 → 32.0 ms (0.97) — → 29.1 ms 30.6 → 29.0 ms (0.95)
clickbench_q07 37.2 → 36.4 ms (0.98) — → 38.7 ms — → 36.5 ms 67.8 → 69.0 ms (1.02) 30.9 → 29.6 ms (0.96) — → 28.8 ms 25.2 → 23.6 ms (0.94)
clickbench_q08 124 → 124 ms (1.00) — → 117 ms — → 162 ms 160 → 154 ms (0.96) 101 → 101 ms (0.99) — → 108 ms 90.9 → 89.2 ms (0.98)
clickbench_q09 216 → 212 ms (0.98) — → 206 ms — → 248 ms 266 → 262 ms (0.98) 125 → 127 ms (1.02) — → 129 ms 120 → 118 ms (0.98)
clickbench_q10 66.1 → 64.4 ms (0.98) — → 63.5 ms — → 56.4 ms 101 → 99.2 ms (0.98) 68.5 → 62.6 ms (0.91) — → 66.2 ms 52.5 → 53.8 ms (1.03)
clickbench_q11 69.4 → 64.1 ms (0.92) — → 64.0 ms — → 65.2 ms 109 → 109 ms (1.00) 71.0 → 64.1 ms (0.90) — → 67.0 ms 62.8 → 63.8 ms (1.02)
clickbench_q12 128 → 124 ms (0.97) — → 120 ms — → 151 ms 163 → 159 ms (0.98) 99.3 → 97.6 ms (0.98) — → 93.6 ms 107 → 106 ms (0.99)
clickbench_q13 184 → 186 ms (1.01) — → 188 ms — → 176 ms 227 → 226 ms (1.00) 166 → 158 ms (0.95) — → 157 ms 169 → 169 ms (1.00)
clickbench_q14 132 → 132 ms (0.99) — → 121 ms — → 147 ms 175 → 168 ms (0.96) 102 → 99.6 ms (0.98) — → 97.8 ms 119 → 117 ms (0.98)
clickbench_q15 106 → 101 ms (0.96) — → 96.2 ms — → 146 ms 132 → 130 ms (0.99) 108 → 109 ms (1.02) — → 111 ms 101 → 99.2 ms (0.99)
clickbench_q16 219 → 216 ms (0.99) — → 209 ms — → 300 ms 238 → 233 ms (0.98) 169 → 168 ms (0.99) — → 174 ms 182 → 176 ms (0.97)
clickbench_q17 227 → 222 ms (0.98) — → 216 ms — → 271 ms 241 → 237 ms (0.98) 168 → 174 ms (1.04) — → 174 ms 181 → 181 ms (1.00)
clickbench_q18 384 → 401 ms (1.04) — → 389 ms — → 437 ms 419 → 417 ms (0.99) 304 → 298 ms (0.98) — → 306 ms 318 → 315 ms (0.99)
clickbench_q19 47.5 → 42.4 ms (0.89) 🟢 — → 46.6 ms — → 40.2 ms 76.5 → 76.1 ms (0.99) 33.8 → 34.8 ms (1.03) — → 36.9 ms 28.0 → 26.6 ms (0.95)
clickbench_q20 237 → 240 ms (1.01) — → 223 ms — → 205 ms 282 → 277 ms (0.98) 157 → 156 ms (1.00) — → 161 ms 233 → 231 ms (0.99)
clickbench_q21 278 → 279 ms (1.00) — → 277 ms — → 260 ms 297 → 291 ms (0.98) 177 → 183 ms (1.03) — → 255 ms 271 → 273 ms (1.01)
clickbench_q22 378 → 389 ms (1.03) — → 343 ms — → 329 ms 429 → 413 ms (0.96) 296 → 297 ms (1.00) — → 353 ms 354 → 354 ms (1.00)
clickbench_q23 354 → 374 ms (1.06) — → 461 ms — → 582 ms 1.20 → 1.17 s (0.98) 172 → 181 ms (1.06) — → 84.9 ms 245 → 246 ms (1.00)
clickbench_q24 63.0 → 56.5 ms (0.90) 🟢 — → 65.8 ms — → 58.8 ms 93.6 → 87.9 ms (0.94) 41.2 → 44.9 ms (1.09) — → 35.9 ms 47.6 → 43.1 ms (0.91)
clickbench_q25 64.7 → 61.7 ms (0.95) — → 60.0 ms — → 49.2 ms 103 → 101 ms (0.97) 52.5 → 51.4 ms (0.98) — → 47.3 ms 56.4 → 54.7 ms (0.97)
clickbench_q26 67.0 → 63.3 ms (0.94) — → 63.8 ms — → 52.4 ms 89.0 → 87.3 ms (0.98) 46.5 → 45.4 ms (0.98) — → 35.5 ms 49.3 → 45.9 ms (0.93)
clickbench_q27 250 → 254 ms (1.02) — → 242 ms — → 230 ms 288 → 283 ms (0.98) 169 → 169 ms (1.00) — → 213 ms 234 → 226 ms (0.97)
clickbench_q28 679 → 638 ms (0.94) — → 622 ms — → 968 ms 703 → 697 ms (0.99) 561 → 554 ms (0.99) — → 577 ms 935 → 941 ms (1.01)
clickbench_q29 59.9 → 59.5 ms (0.99) — → 51.8 ms — → 48.9 ms 91.8 → 89.9 ms (0.98) 46.2 → 45.2 ms (0.98) — → 41.3 ms 31.0 → 30.4 ms (0.98)
clickbench_q30 112 → 114 ms (1.01) — → 109 ms — → 128 ms 174 → 166 ms (0.96) 110 → 101 ms (0.92) — → 108 ms 128 → 128 ms (1.00)
clickbench_q31 159 → 155 ms (0.97) — → 153 ms — → 162 ms 197 → 192 ms (0.97) 173 → 183 ms (1.05) — → 192 ms 202 → 175 ms (0.87) 🟢
clickbench_q32 394 → 381 ms (0.97) — → 378 ms — → 409 ms 427 → 429 ms (1.01) 399 → 400 ms (1.00) — → 418 ms 413 → 410 ms (0.99)
clickbench_q33 435 → 430 ms (0.99) — → 448 ms — → 625 ms 478 → 473 ms (0.99) 401 → 399 ms (0.99) — → 433 ms 452 → 447 ms (0.99)
clickbench_q34 439 → 453 ms (1.03) — → 443 ms — → 631 ms 481 → 472 ms (0.98) 421 → 416 ms (0.99) — → 446 ms 453 → 467 ms (1.03)
clickbench_q35 95.8 → 97.3 ms (1.02) — → 90.4 ms — → 133 ms 123 → 125 ms (1.02) 90.6 → 88.5 ms (0.98) — → 90.8 ms 73.2 → 75.1 ms (1.03)
clickbench_q36 68.9 → 65.4 ms (0.95) — → 68.7 ms — → 81.4 ms 136 → 134 ms (0.99) 34.1 → 33.9 ms (1.00) — → 31.8 ms 36.3 → 35.3 ms (0.97)
clickbench_q37 48.5 → 45.9 ms (0.95) — → 50.0 ms — → 55.3 ms 93.9 → 91.9 ms (0.98) 29.6 → 30.0 ms (1.01) — → 26.9 ms 29.2 → 30.2 ms (1.04)
clickbench_q38 42.7 → 40.6 ms (0.95) — → 41.0 ms — → 48.8 ms 108 → 107 ms (0.99) 29.7 → 28.4 ms (0.96) — → 31.7 ms 31.0 → 29.2 ms (0.94)
clickbench_q39 115 → 112 ms (0.98) — → 109 ms — → 154 ms 212 → 207 ms (0.98) 44.8 → 43.0 ms (0.96) — → 42.0 ms 60.2 → 59.1 ms (0.98)
clickbench_q40 36.2 → 34.7 ms (0.96) — → 37.7 ms — → 41.9 ms 77.8 → 78.0 ms (1.00) 28.6 → 28.0 ms (0.98) — → 27.4 ms 19.8 → 19.0 ms (0.96)
clickbench_q41 37.9 → 35.0 ms (0.92) — → 37.7 ms — → 39.0 ms 76.2 → 76.8 ms (1.01) 28.9 → 29.8 ms (1.03) — → 29.1 ms 20.4 → 20.0 ms (0.98)
clickbench_q42 37.6 → 35.0 ms (0.93) — → 36.5 ms — → 37.8 ms 75.1 → 76.1 ms (1.01) 26.5 → 29.3 ms (1.10) 🔴 — → 27.2 ms 20.8 → 20.2 ms (0.97)

No file size changes detected.

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be working, I think we will want to add tpc-ds (better coverage for dynamic filters) and maybe sf=10? SF=1 effectively measure constant overhead at this point

@gatesn
gatesn merged commit ee414c6 into develop Nov 20, 2026
124 of 125 checks passed
@gatesn
gatesn deleted the ngates/bench-scan-api-target-4e945c branch November 20, 2026 17:33
gatesn added a commit that referenced this pull request Nov 20, 2026
The bench's VORTEX_USE_SCAN_API fallback is dropped: develop's #11685 removed it, so only
--scan-api reaches VortexTable now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch was successfully deployed

1 active deployment
vercel-docs — e68a347f Deployed Nov 20, 2026 by gatesn via Publish preview / Deploy #11510
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants