Skip to content

Enable the unstable edition in every benchmark - #11534

Merged
joseph-isaacs merged 20 commits into
developfrom
claude/unstable-encodings-benchmarks-audit-79g1tu
Nov 18, 2026
Merged

joseph-isaacs merged 20 commits into
developfrom
claude/unstable-encodings-benchmarks-audit-79g1tu

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Benchmark sessions now enable the unstable edition. Since #11595, that edition holds only fastlanes.delta. Every benchmark that writes Vortex files therefore writes them with delta available, and PR benchmark runs compare against the base with it.

joseph-isaacs and others added 3 commits November 17, 2026 11:19
Benchmarks can now write Vortex files whose compressor may also choose the
array encodings of the unstable edition (bitpacked.v2, for.v2, delta,
decimal_byte_parts.v2, and patched when VORTEX_EXPERIMENTAL_PATCHED_ARRAY=1).
The files are written by a separate UNSTABLE_SESSION, so the existing
vortex and vortex-compact formats are unchanged. The unstable layouts and
aggregates stay disabled so every scan can read the files.

The format is wired through data-gen, tpchgen, the DataFusion, DuckDB and
compress benchmarks, and the bench orchestrator, but is not added to any CI
preset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Signed-off-by: Claude <noreply@anthropic.com>
The vortex-unstable benchmark format now enables only fastlanes.delta on
top of the default editions. The audit found the other unstable array
encodings not ready to benchmark: blocked bitpacked.v2 and for.v2 have no
slice, filter or take kernels, patched is gated by an environment
variable, and nothing produces union.

The pr-full and develop presets run vortex-unstable on their NVMe targets
for DataFusion and DuckDB. The file-size reporting scripts know the new
format directory.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.1

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 6 improved benchmarks
✅ 1969 untouched benchmarks
⏩ 53 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime decode_avx2[DeltaU8] 34.9 µs 11.7 µs ×3
⚡ Walltime decode_avx2[BitPackedU8] 32.3 µs 12.4 µs ×2.6
⚡ Walltime decode_neon[DeltaU8] 46.3 µs 19.3 µs ×2.4
⚡ Walltime decode_avx512[DeltaU8] 19.7 µs 11.1 µs +78.22%
⚡ Walltime decode_neon[BitPackedU8] 41.9 µs 23.9 µs +75.28%
⚡ Walltime decode_avx512[BitPackedU8] 18.3 µs 10.9 µs +68.1%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/unstable-encodings-benchmarks-audit-79g1tu (a23764f) with develop (5dd206d)2

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (4504cce) during the generation of this report, so 5dd206d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Every SQL target set that runs vortex now also runs vortex-unstable, in
every preset, including S3, nightly and the per-target TPC-DS SF=100 jobs.
A matrix test asserts that pairing.

The compression and random access benchmarks also run vortex-unstable:
each random access dataset writes a -unstable file, and the split scripts
and develop workflow pass the format.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
@joseph-isaacs joseph-isaacs added the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Nov 17, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Nov 17, 2026
@joseph-isaacs joseph-isaacs added the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Nov 17, 2026
@github-actions github-actions Bot removed the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Nov 17, 2026
The Rust (docs) job rejects public docs that link private items.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 2707db2 vortex-queries 1 Explore Profiling Data
🟢 Done 2707db2 clickbench-sorted-nvme 1 Explore Profiling Data
🟢 Done 2707db2 random-access-bench 1 Explore Profiling Data
🟢 Done 2707db2 clickbench-nvme 1 Explore Profiling Data
🟢 Done 2707db2 statpopgen 1 Explore Profiling Data
🟢 Done 2707db2 appian-nvme 1 Explore Profiling Data
🟢 Done 2707db2 job-nvme 1 Explore Profiling Data
🟢 Done 2707db2 tpch-s3-10 1 Explore Profiling Data
🟢 Done 2707db2 tpch-nvme-10 1 Explore Profiling Data
🟢 Done 2707db2 polarsignals 1 Explore Profiling Data
🟢 Done 2707db2 tpcds-nvme-10 1 Explore Profiling Data
🟢 Done 2707db2 tpch-s3 1 Explore Profiling Data
🟢 Done 2707db2 tpch-nvme 1 Explore Profiling Data
🟢 Done 2707db2 fineweb 1 Explore Profiling Data
🟢 Done 2707db2 tpch-nested-nvme 1 Explore Profiling Data
🟢 Done 2707db2 fineweb-s3 1 Explore Profiling Data
🟢 Done 2707db2 tpcds-nvme 1 Explore Profiling Data
🟢 Done 2707db2 compress-bench 1 Explore Profiling Data
🟢 Done 2707db2 string-bench 1 Explore Profiling Data
Previous Runs (6)
Status Commit Job Attempt Link
🟢 Done 2593fb2 random-access-bench 1 Explore Profiling Data
🟢 Done ae1091f string-bench 1 Explore Profiling Data
🟢 Done ae1091f compress-bench 1 Explore Profiling Data
🟢 Done 3c64441 random-access-bench 1 Explore Profiling Data
🟢 Done 3c64441 compress-bench 1 Explore Profiling Data
🟢 Done 3c64441 string-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR 2707db27 vs base bdb4af51


vortex / vortex-file-compressed / ns (1.065x ➖, 1↑ 4↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 454391806 / 453228909 / +0.3% 452591987 / 451135291 / +0.3% 1.00 / 1.00 / -0.1%
compress time/bimbo 2352206020 / 2344914870 / +0.3% 2323236100 / 2323916629 / -0.0% 1.01 / 1.01 / +0.3%
compress time/cmsprovider 1055993724 / 1056515440 / -0.0% 1049397081 / 1044362373 / +0.5% 1.01 / 1.01 / -0.5%
compress time/euro2016 179520876 / 169984487 / +5.6% 178654692 / 171419290 / +4.2% 1.00 / 0.99 / +1.3%
compress time/food 208281443 / 207127844 / +0.6% 213492676 / 211904003 / +0.7% 0.98 / 0.98 / -0.2%
compress time/hashtags 375668210 / 361057005 / +4.0% 370431308 / 359882401 / +2.9% 1.01 / 1.00 / +1.1%
compress time/sparse-wide-flat-runs 90286462 / 90552151 / -0.3% 90066317 / 85527357 / +5.3% 1.00 / 1.06 / -5.3%
compress time/sparse-wide-lists-runs 679879580 / 674406832 / +0.8% 669572540 / 671804791 / -0.3% 1.02 / 1.00 / +1.1%
compress time/sparse-wide-lists-shuffled 300971346 / 290392286 / +3.6% 287440194 / 312935181 / -8.1% 1.05 / 0.93 / +12.8%
compress time/sparse-wide-nested-child-nulls 615329092 / 624100086 / -1.4% 611942135 / 607445032 / +0.7% 1.01 / 1.03 / -2.1%
compress time/sparse-wide-nested-runs 636414418 / 612761886 / +3.9% 603267827 / 629194781 / -4.1% 1.05 / 0.97 / +8.3%
compress time/statpopgen 3386500734 / 546701180 / +519.4% 🔴 3339429758 / 564285708 / +491.8% 🔴 1.01 / 0.97 / +4.7%
compress time/taxi 208456128 / 204423886 / +2.0% 208604492 / 207232583 / +0.7% 1.00 / 0.99 / +1.3%
compress time/tpc-h l_comment canonical 109253261 / 106619455 / +2.5% 109995885 / 108019528 / +1.8% 0.99 / 0.99 / +0.6%
compress time/tpc-h l_comment chunked 109164700 / 106669539 / +2.3% 110045205 / 107919613 / +2.0% 0.99 / 0.99 / +0.4%
compress time/tpc-h sf1 lineitem 403641615 / 401679336 / +0.5% 401600619 / 401474409 / +0.0% 1.01 / 1.00 / +0.5%
compress time/tpc-h sf1 orders 88530924 / 86667730 / +2.1% 89264963 / 86760384 / +2.9% 0.99 / 1.00 / -0.7%
compress time/tpc-h sf1 partsupp 59301998 / 57187790 / +3.7% 59725723 / 58510463 / +2.1% 0.99 / 0.98 / +1.6%
compress time/wide table cols=100 chunks=1 rows=1000 5714714 / 5719680 / -0.1% 7313434 / 7370802 / -0.8% 0.78 / 0.78 / +0.7%
compress time/wide table cols=100 chunks=50 rows=1000 5762489 / 5678599 / +1.5% 6907676 / 7216599 / -4.3% 0.83 / 0.79 / +6.0%
compress time/wide table cols=1000 chunks=1 rows=1000 55919820 / 55458560 / +0.8% 56652445 / 60224616 / -5.9% 0.99 / 0.92 / +7.2%
compress time/wide table cols=1000 chunks=50 rows=1000 56574102 / 55707477 / +1.6% 57636839 / 58659185 / -1.7% 0.98 / 0.95 / +3.4%
compress time/wide table cols=10000 chunks=1 rows=1000 635773014 / 638669317 / -0.5% 627464918 / 636194526 / -1.4% 1.01 / 1.00 / +0.9%
compress time/wide table cols=10000 chunks=50 rows=1000 638922038 / 638133355 / +0.1% 625939685 / 644455319 / -2.9% 1.02 / 0.99 / +3.1%
decompress time/arade 15538952 / 14793316 / +5.0% 16339636 / 15957909 / +2.4% 0.95 / 0.93 / +2.6%
decompress time/bimbo 69333204 / 67022790 / +3.4% 74673971 / 69739939 / +7.1% 0.93 / 0.96 / -3.4%
decompress time/cmsprovider 40642238 / 38641593 / +5.2% 42162116 / 41264251 / +2.2% 0.96 / 0.94 / +2.9%
decompress time/euro2016 13013899 / 12042186 / +8.1% 13945176 / 12988880 / +7.4% 0.93 / 0.93 / +0.7%
decompress time/food 4417022 / 3869549 / +14.1% 🔴 5849013 / 4629295 / +26.3% 🔴 0.76 / 0.84 / -9.7%
decompress time/hashtags 61139124 / 41983186 / +45.6% 🔴 61460188 / 42124456 / +45.9% 🔴 0.99 / 1.00 / -0.2%
decompress time/sparse-wide-flat-runs 11611183 / 13830722 / -16.0% 🟢 13675950 / 11005538 / +24.3% 🔴 0.85 / 1.26 / -32.4%
decompress time/sparse-wide-lists-runs 16687884 / 16764272 / -0.5% 17348364 / 16970803 / +2.2% 0.96 / 0.99 / -2.6%
decompress time/sparse-wide-lists-shuffled 26610656 / 26108460 / +1.9% 27792207 / 27844761 / -0.2% 0.96 / 0.94 / +2.1%
decompress time/sparse-wide-nested-child-nulls 18881838 / 17110420 / +10.4% 🔴 17032398 / 16595419 / +2.6% 1.11 / 1.03 / +7.5%
decompress time/sparse-wide-nested-runs 16102115 / 17728824 / -9.2% 15839489 / 18083308 / -12.4% 🟢 1.02 / 0.98 / +3.7%
decompress time/statpopgen 721787897 / 721857096 / -0.0% 704661198 / 729515973 / -3.4% 1.02 / 0.99 / +3.5%
decompress time/taxi 10724270 / 10341106 / +3.7% 12491992 / 11541671 / +8.2% 0.86 / 0.90 / -4.2%
decompress time/tpc-h l_comment canonical 5199553 / 5027085 / +3.4% 5546056 / 5142471 / +7.8% 0.94 / 0.98 / -4.1%
decompress time/tpc-h l_comment chunked 5196942 / 5029376 / +3.3% 5880038 / 5700529 / +3.1% 0.88 / 0.88 / +0.2%
decompress time/tpc-h sf1 lineitem 15386232 / 15259899 / +0.8% 16329440 / 16410886 / -0.5% 0.94 / 0.93 / +1.3%
decompress time/tpc-h sf1 orders 4156407 / 3966221 / +4.8% 4548295 / 4345026 / +4.7% 0.91 / 0.91 / +0.1%
decompress time/tpc-h sf1 partsupp 3324087 / 3180260 / +4.5% 3624005 / 3169893 / +14.3% 🔴 0.92 / 1.00 / -8.6%
decompress time/wide table cols=100 chunks=1 rows=1000 1530154 / 1551466 / -1.4% 1682315 / 1657965 / +1.5% 0.91 / 0.94 / -2.8%
decompress time/wide table cols=100 chunks=50 rows=1000 1557210 / 1510074 / +3.1% 1653718 / 1717687 / -3.7% 0.94 / 0.88 / +7.1%
decompress time/wide table cols=1000 chunks=1 rows=1000 14004230 / 13815184 / +1.4% 14407455 / 14465531 / -0.4% 0.97 / 0.96 / +1.8%
decompress time/wide table cols=1000 chunks=50 rows=1000 13938488 / 13734911 / +1.5% 14430344 / 14338797 / +0.6% 0.97 / 0.96 / +0.8%
decompress time/wide table cols=10000 chunks=1 rows=1000 146023240 / 142523633 / +2.5% 148703885 / 143362267 / +3.7% 0.98 / 0.99 / -1.2%
decompress time/wide table cols=10000 chunks=50 rows=1000 146345584 / 142281994 / +2.9% 147953178 / 143825367 / +2.9% 0.99 / 0.99 / -0.0%
vortex / vortex-file-compressed / bytes (0.957x ➖, 4↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/arade 141323236 / 142108908 / -0.6%
vortex-file-compressed size/bimbo 410263628 / 425251924 / -3.5%
vortex-file-compressed size/cmsprovider 413118820 / 416254308 / -0.8%
vortex-file-compressed size/euro2016 156505100 / 164044372 / -4.6%
vortex-file-compressed size/food 37247664 / 38951184 / -4.4%
vortex-file-compressed size/hashtags 169752540 / 183585892 / -7.5%
vortex-file-compressed size/sparse-wide-flat-runs 1552424 / 1552392 / +0.0%
vortex-file-compressed size/sparse-wide-lists-runs 3933548 / 3991596 / -1.5%
vortex-file-compressed size/sparse-wide-lists-shuffled 6436508 / 6436476 / +0.0%
vortex-file-compressed size/sparse-wide-nested-child-nulls 1687396 / 1687364 / +0.0%
vortex-file-compressed size/sparse-wide-nested-runs 1298452 / 1298420 / +0.0%
vortex-file-compressed size/statpopgen 230798780 / 251914932 / -8.4%
vortex-file-compressed size/taxi 48553948 / 49091068 / -1.1%
vortex-file-compressed size/tpc-h l_comment canonical 64987212 / 75050436 / -13.4% 🟢
vortex-file-compressed size/tpc-h l_comment chunked 64987212 / 75050436 / -13.4% 🟢
vortex-file-compressed size/tpc-h sf1 lineitem 154809520 / 173478160 / -10.8% 🟢
vortex-file-compressed size/tpc-h sf1 orders 33372820 / 42977780 / -22.3% 🟢
vortex-file-compressed size/tpc-h sf1 partsupp 24526824 / 25940480 / -5.4%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934888 / 934856 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934888 / 934856 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9333688 / 9333656 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9333688 / 9333656 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93357688 / 93357656 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93357688 / 93357656 / +0.0%
vortex / parquet / ns (1.000x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 1152153101 / 1158818514 / -0.6% 1155178929 / 1163486147 / -0.7% 1.00 / 1.00 / +0.1%
compress time/bimbo 2124843836 / 2145331192 / -1.0% 2129956020 / 2147660794 / -0.8% 1.00 / 1.00 / -0.1%
compress time/cmsprovider 1661795160 / 1670107831 / -0.5% 1666993485 / 1670470841 / -0.2% 1.00 / 1.00 / -0.3%
compress time/euro2016 996691325 / 998554804 / -0.2% 999381975 / 996224058 / +0.3% 1.00 / 1.00 / -0.5%
compress time/food 405334141 / 407421958 / -0.5% 406413963 / 408789325 / -0.6% 1.00 / 1.00 / +0.1%
compress time/hashtags 538247366 / 538434768 / -0.0% 548265768 / 545645217 / +0.5% 0.98 / 0.99 / -0.5%
compress time/sparse-wide-flat-runs 41731016 / 42009648 / -0.7% 41764566 / 42469887 / -1.7% 1.00 / 0.99 / +1.0%
compress time/sparse-wide-lists-runs 80873074 / 80097288 / +1.0% 81379160 / 79681950 / +2.1% 0.99 / 1.01 / -1.1%
compress time/sparse-wide-lists-shuffled 156825720 / 154068852 / +1.8% 155306571 / 155686811 / -0.2% 1.01 / 0.99 / +2.0%
compress time/sparse-wide-nested-child-nulls 54264590 / 54644370 / -0.7% 55802787 / 54854472 / +1.7% 0.97 / 1.00 / -2.4%
compress time/sparse-wide-nested-runs 55140151 / 55253426 / -0.2% 54922937 / 55841406 / -1.6% 1.00 / 0.99 / +1.5%
compress time/statpopgen 1023929990 / 1025278871 / -0.1% 1027642041 / 1028283731 / -0.1% 1.00 / 1.00 / -0.1%
compress time/taxi 239324111 / 238482967 / +0.4% 239958851 / 240044646 / -0.0% 1.00 / 0.99 / +0.4%
compress time/tpc-h l_comment canonical 721924416 / 722077144 / -0.0% 723471485 / 724567874 / -0.2% 1.00 / 1.00 / +0.1%
compress time/tpc-h l_comment chunked 721008396 / 721941834 / -0.1% 722491979 / 723442086 / -0.1% 1.00 / 1.00 / +0.0%
compress time/tpc-h sf1 lineitem 1195443882 / 1204816240 / -0.8% 1195707366 / 1201750617 / -0.5% 1.00 / 1.00 / -0.3%
compress time/tpc-h sf1 orders 337879614 / 341169546 / -1.0% 343840201 / 347548512 / -1.1% 0.98 / 0.98 / +0.1%
compress time/tpc-h sf1 partsupp 305429557 / 306479372 / -0.3% 309497920 / 307039467 / +0.8% 0.99 / 1.00 / -1.1%
compress time/wide table cols=100 chunks=1 rows=1000 3654071 / 3748157 / -2.5% 4311176 / 4357753 / -1.1% 0.85 / 0.86 / -1.5%
compress time/wide table cols=100 chunks=50 rows=1000 3684403 / 3579330 / +2.9% 4174436 / 4432507 / -5.8% 0.88 / 0.81 / +9.3%
compress time/wide table cols=1000 chunks=1 rows=1000 37543371 / 37108730 / +1.2% 39318183 / 38513498 / +2.1% 0.95 / 0.96 / -0.9%
compress time/wide table cols=1000 chunks=50 rows=1000 37232176 / 36988776 / +0.7% 38428262 / 38314341 / +0.3% 0.97 / 0.97 / +0.4%
compress time/wide table cols=10000 chunks=1 rows=1000 377767676 / 374542726 / +0.9% 384844741 / 386971290 / -0.5% 0.98 / 0.97 / +1.4%
compress time/wide table cols=10000 chunks=50 rows=1000 377930472 / 373913318 / +1.1% 380972740 / 386424262 / -1.4% 0.99 / 0.97 / +2.5%
decompress time/arade 54630694 / 54691850 / -0.1% 54826699 / 54900227 / -0.1% 1.00 / 1.00 / +0.0%
decompress time/bimbo 92268449 / 92249456 / +0.0% 92240334 / 92825349 / -0.6% 1.00 / 0.99 / +0.7%
decompress time/cmsprovider 102932998 / 103646258 / -0.7% 103146492 / 103234977 / -0.1% 1.00 / 1.00 / -0.6%
decompress time/euro2016 159411564 / 159582122 / -0.1% 159655911 / 159779082 / -0.1% 1.00 / 1.00 / -0.0%
decompress time/food 26555712 / 26569544 / -0.1% 26814008 / 26770755 / +0.2% 0.99 / 0.99 / -0.2%
decompress time/hashtags 446453368 / 446883113 / -0.1% 446204155 / 447181563 / -0.2% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-flat-runs 14855346 / 14996378 / -0.9% 15376387 / 15549627 / -1.1% 0.97 / 0.96 / +0.2%
decompress time/sparse-wide-lists-runs 277151738 / 277539379 / -0.1% 277164209 / 278138400 / -0.4% 1.00 / 1.00 / +0.2%
decompress time/sparse-wide-lists-shuffled 318674263 / 318081632 / +0.2% 318499016 / 318854570 / -0.1% 1.00 / 1.00 / +0.3%
decompress time/sparse-wide-nested-child-nulls 52714964 / 52934085 / -0.4% 53128633 / 53213116 / -0.2% 0.99 / 0.99 / -0.3%
decompress time/sparse-wide-nested-runs 53000854 / 53160882 / -0.3% 53035587 / 53269786 / -0.4% 1.00 / 1.00 / +0.1%
decompress time/statpopgen 2905501410 / 2914725342 / -0.3% 2928531817 / 2991128481 / -2.1% 0.99 / 0.97 / +1.8%
decompress time/taxi 60793843 / 60843518 / -0.1% 60933269 / 61261427 / -0.5% 1.00 / 0.99 / +0.5%
decompress time/tpc-h l_comment canonical 35955481 / 36012709 / -0.2% 36040349 / 36123105 / -0.2% 1.00 / 1.00 / +0.1%
decompress time/tpc-h l_comment chunked 35923954 / 36389936 / -1.3% 36040816 / 36787430 / -2.0% 1.00 / 0.99 / +0.8%
decompress time/tpc-h sf1 lineitem 82090996 / 82349284 / -0.3% 82507447 / 82324422 / +0.2% 0.99 / 1.00 / -0.5%
decompress time/tpc-h sf1 orders 87965868 / 88154253 / -0.2% 88439462 / 88382563 / +0.1% 0.99 / 1.00 / -0.3%
decompress time/tpc-h sf1 partsupp 111184291 / 111246282 / -0.1% 111459749 / 111456927 / +0.0% 1.00 / 1.00 / -0.1%
decompress time/wide table cols=100 chunks=1 rows=1000 1954652 / 1954181 / +0.0% 2079827 / 2161001 / -3.8% 0.94 / 0.90 / +3.9%
decompress time/wide table cols=100 chunks=50 rows=1000 1950990 / 1949696 / +0.1% 2145822 / 2090301 / +2.7% 0.91 / 0.93 / -2.5%
decompress time/wide table cols=1000 chunks=1 rows=1000 22708751 / 22393368 / +1.4% 23690658 / 23271737 / +1.8% 0.96 / 0.96 / -0.4%
decompress time/wide table cols=1000 chunks=50 rows=1000 22398540 / 22296084 / +0.5% 23047627 / 23032924 / +0.1% 0.97 / 0.97 / +0.4%
decompress time/wide table cols=10000 chunks=1 rows=1000 235169383 / 231778432 / +1.5% 238713104 / 234295423 / +1.9% 0.99 / 0.99 / -0.4%
decompress time/wide table cols=10000 chunks=50 rows=1000 233327420 / 230557914 / +1.2% 237372483 / 232562402 / +2.1% 0.98 / 0.99 / -0.8%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/arade 210020851 / 210020851 / +0.0%
parquet size/bimbo 376187078 / 376187078 / +0.0%
parquet size/cmsprovider 364268159 / 364268159 / +0.0%
parquet size/euro2016 111035158 / 111035158 / +0.0%
parquet size/food 34903783 / 34903783 / +0.0%
parquet size/hashtags 126189203 / 126189203 / +0.0%
parquet size/sparse-wide-flat-runs 1388944 / 1388944 / +0.0%
parquet size/sparse-wide-lists-runs 4327813 / 4327813 / +0.0%
parquet size/sparse-wide-lists-shuffled 5710569 / 5710569 / +0.0%
parquet size/sparse-wide-nested-child-nulls 1266042 / 1266042 / +0.0%
parquet size/sparse-wide-nested-runs 1388944 / 1388944 / +0.0%
parquet size/statpopgen 81284006 / 81284006 / +0.0%
parquet size/taxi 52142534 / 52142534 / +0.0%
parquet size/tpc-h l_comment canonical 52204116 / 52204116 / +0.0%
parquet size/tpc-h l_comment chunked 52204116 / 52204116 / +0.0%
parquet size/tpc-h sf1 lineitem 153403836 / 153403836 / +0.0%
parquet size/tpc-h sf1 orders 37029542 / 37029542 / +0.0%
parquet size/tpc-h sf1 partsupp 29200612 / 29200612 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.999x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 95599606 / 95650421 / -0.1% 118920158 / 121132770 / -1.8% 0.80 / 0.79 / +1.8%
compress time/bimbo 595009513 / 605043864 / -1.7% 657065937 / 671791214 / -2.2% 0.91 / 0.90 / +0.5%
compress time/cmsprovider 376056934 / 379218000 / -0.8% 438038027 / 443221287 / -1.2% 0.86 / 0.86 / +0.3%
compress time/euro2016 38091934 / 38564980 / -1.2% 51787726 / 53301758 / -2.8% 0.74 / 0.72 / +1.7%
compress time/food 39911106 / 40366806 / -1.1% 53220051 / 54827942 / -2.9% 0.75 / 0.74 / +1.9%
compress time/hashtags 77676562 / 77793366 / -0.2% 105652480 / 106474860 / -0.8% 0.74 / 0.73 / +0.6%
compress time/sparse-wide-flat-runs 27745050 / 28074474 / -1.2% 36586585 / 37108833 / -1.4% 0.76 / 0.76 / +0.2%
compress time/sparse-wide-lists-runs 44069822 / 45259763 / -2.6% 51007800 / 52039181 / -2.0% 0.86 / 0.87 / -0.7%
compress time/sparse-wide-lists-shuffled 45997258 / 46560285 / -1.2% 49256500 / 50081844 / -1.6% 0.93 / 0.93 / +0.4%
compress time/sparse-wide-nested-child-nulls 40421400 / 40716394 / -0.7% 50157169 / 50729557 / -1.1% 0.81 / 0.80 / +0.4%
compress time/sparse-wide-nested-runs 40439936 / 40814886 / -0.9% 50147473 / 50773182 / -1.2% 0.81 / 0.80 / +0.3%
compress time/statpopgen 216243304 / 217058694 / -0.4% 262161013 / 263938039 / -0.7% 0.82 / 0.82 / +0.3%
compress time/taxi 58614156 / 59139214 / -0.9% 74738148 / 75953655 / -1.6% 0.78 / 0.78 / +0.7%
compress time/tpc-h l_comment canonical 22503040 / 22754496 / -1.1% 30511574 / 31310642 / -2.6% 0.74 / 0.73 / +1.5%
compress time/tpc-h l_comment chunked 22579888 / 22856664 / -1.2% 30515335 / 31386679 / -2.8% 0.74 / 0.73 / +1.6%
compress time/tpc-h sf1 lineitem 128484209 / 129862850 / -1.1% 162944645 / 171064479 / -4.7% 0.79 / 0.76 / +3.9%
compress time/tpc-h sf1 orders 24535718 / 24766214 / -0.9% 3416499 / 34774468 / -1.8% 0.72 / 0.71 / +0.8%
compress time/tpc-h sf1 partsupp 13147536 / 13158528 / -0.1% 19162518 / 19597870 / -2.2% 0.69 / 0.67 / +2.2%
compress time/wide table cols=100 chunks=1 rows=1000 194805.5 / 196655 / -0.9% 412534 / 598612 / -31.1% 🟢 0.47 / 0.33 / +43.7%
compress time/wide table cols=100 chunks=50 rows=1000 209584 / 194151.5 / +7.9% 370913 / 551020 / -32.7% 🟢 0.57 / 0.35 / +60.4%
compress time/wide table cols=1000 chunks=1 rows=1000 2337844 / 2346114 / -0.4% 2787020 / 3217685 / -13.4% 🟢 0.84 / 0.73 / +15.0%
compress time/wide table cols=1000 chunks=50 rows=1000 2300723 / 2316834 / -0.7% 2783097 / 3070438 / -9.4% 0.83 / 0.75 / +9.6%
compress time/wide table cols=10000 chunks=1 rows=1000 24881498 / 24967601 / -0.3% 31108820 / 31386644 / -0.9% 0.80 / 0.80 / +0.5%
compress time/wide table cols=10000 chunks=50 rows=1000 24886068 / 25154706 / -1.1% 31118866 / 31464157 / -1.1% 0.80 / 0.80 / +0.0%
decompress time/arade 216914628 / 210269144 / +3.2% 217056257 / 210867590 / +2.9% 1.00 / 1.00 / +0.2%
decompress time/bimbo 412030272 / 424998364 / -3.1% 412563330 / 425279442 / -3.0% 1.00 / 1.00 / -0.1%
decompress time/cmsprovider 692326250 / 695812920 / -0.5% 694326290 / 696532040 / -0.3% 1.00 / 1.00 / -0.2%
decompress time/euro2016 85648522 / 83596626 / +2.5% 85970193 / 83935867 / +2.4% 1.00 / 1.00 / +0.0%
decompress time/food 67089978 / 65559936 / +2.3% 67257451 / 65758514 / +2.3% 1.00 / 1.00 / +0.1%
decompress time/hashtags 164880520 / 163165782 / +1.1% 165056221 / 163489821 / +1.0% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-flat-runs 18901024 / 19117088 / -1.1% 20566852 / 20222901 / +1.7% 0.92 / 0.95 / -2.8%
decompress time/sparse-wide-lists-runs 80905612 / 81349588 / -0.5% 81146875 / 81493571 / -0.4% 1.00 / 1.00 / -0.1%
decompress time/sparse-wide-lists-shuffled 81522610 / 81548258 / -0.0% 81735277 / 81685740 / +0.1% 1.00 / 1.00 / -0.1%
decompress time/sparse-wide-nested-child-nulls 26749385 / 26748142 / +0.0% 27854267 / 28235901 / -1.4% 0.96 / 0.95 / +1.4%
decompress time/sparse-wide-nested-runs 26885444 / 26657264 / +0.9% 28455166 / 29477006 / -3.5% 0.94 / 0.90 / +4.5%
decompress time/statpopgen 462482354 / 462294284 / +0.0% 463221753 / 463348326 / -0.0% 1.00 / 1.00 / +0.1%
decompress time/taxi 51090320 / 50791668 / +0.6% 51384390 / 51099381 / +0.6% 0.99 / 0.99 / +0.0%
decompress time/tpc-h l_comment canonical 39756370 / 39880288 / -0.3% 40103635 / 40187356 / -0.2% 0.99 / 0.99 / -0.1%
decompress time/tpc-h l_comment chunked 39782449 / 39917294 / -0.3% 40117642 / 40228113 / -0.3% 0.99 / 0.99 / -0.1%
decompress time/tpc-h sf1 lineitem 188135938 / 191663555 / -1.8% 188407617 / 191914272 / -1.8% 1.00 / 1.00 / -0.0%
decompress time/tpc-h sf1 orders 38318263 / 39550973 / -3.1% 38407213 / 39683794 / -3.2% 1.00 / 1.00 / +0.1%
decompress time/tpc-h sf1 partsupp 13590547 / 13896142 / -2.2% 13789000 / 14005541 / -1.5% 0.99 / 0.99 / -0.7%
decompress time/wide table cols=100 chunks=1 rows=1000 317748 / 315480.5 / +0.7% 344500 / 344112 / +0.1% 0.92 / 0.92 / +0.6%
decompress time/wide table cols=100 chunks=50 rows=1000 316960.5 / 317716 / -0.2% 338171 / 343351 / -1.5% 0.94 / 0.93 / +1.3%
decompress time/wide table cols=1000 chunks=1 rows=1000 3757960 / 3751196 / +0.2% 3774510 / 3709212 / +1.8% 1.00 / 1.01 / -1.6%
decompress time/wide table cols=1000 chunks=50 rows=1000 3727906 / 3469296 / +7.5% 3791633 / 3717069 / +2.0% 0.98 / 0.93 / +5.3%
decompress time/wide table cols=10000 chunks=1 rows=1000 36418122 / 35490178 / +2.6% 37243792 / 36097190 / +3.2% 0.98 / 0.98 / -0.5%
decompress time/wide table cols=10000 chunks=50 rows=1000 36443648 / 36003994 / +1.2% 36810490 / 36835819 / -0.1% 0.99 / 0.98 / +1.3%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/arade 680639722 / 680639722 / +0.0%
arrow-ipc size/bimbo 3168319202 / 3168319202 / +0.0%
arrow-ipc size/cmsprovider 2394412322 / 2394412322 / +0.0%
arrow-ipc size/euro2016 359300042 / 359300042 / +0.0%
arrow-ipc size/food 252069442 / 252069442 / +0.0%
arrow-ipc size/hashtags 700364562 / 700364562 / +0.0%
arrow-ipc size/sparse-wide-flat-runs 245220370 / 245220370 / +0.0%
arrow-ipc size/sparse-wide-lists-runs 142046306 / 142046306 / +0.0%
arrow-ipc size/sparse-wide-lists-shuffled 142870882 / 142870882 / +0.0%
arrow-ipc size/sparse-wide-nested-child-nulls 251527506 / 251527506 / +0.0%
arrow-ipc size/sparse-wide-nested-runs 251527122 / 251527122 / +0.0%
arrow-ipc size/statpopgen 1890451826 / 1890451826 / +0.0%
arrow-ipc size/taxi 474481242 / 474481242 / +0.0%
arrow-ipc size/tpc-h l_comment canonical 184829074 / 184829074 / +0.0%
arrow-ipc size/tpc-h l_comment chunked 184829074 / 184829074 / +0.0%
arrow-ipc size/tpc-h sf1 lineitem 1021072290 / 1021072290 / +0.0%
arrow-ipc size/tpc-h sf1 orders 194812026 / 194812026 / +0.0%
arrow-ipc size/tpc-h sf1 partsupp 131286562 / 131286562 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 123192506 / 123192506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 123192506 / 123192506 / +0.0%

CompressionTimingMeasurement panicked on any format outside its fixed
list, so compress-bench failed in CI once vortex-unstable was added.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
@joseph-isaacs joseph-isaacs added action/bench-compress Run only the compression benchmark on this PR action/bench-string Run only the string encoding benchmark on this PR labels Nov 17, 2026 — with Claude
@github-actions github-actions Bot removed action/bench-string Run only the string encoding benchmark on this PR action/bench-compress Run only the compression benchmark on this PR labels Nov 17, 2026
random-access-bench rejected --formats vortex-unstable because the
format was missing from ALLOWED_FORMATS. Its measurement names also used
the file extension, which vortex-unstable shares with vortex, so the two
formats would have overwritten each other's results.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Replace the vortex-unstable benchmark format with
vortex_bench::enable_unstable_encodings, which enables a bench-unstable
edition holding only fastlanes.delta on a session. The benchmark SESSION
and the string benchmark sessions call it, so every benchmark's existing
vortex files may use delta, and PR runs compare them directly against
the base.

This reverts the vortex-unstable format plumbing in the benchmark
binaries, CI presets, orchestrator and reporting scripts.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
@joseph-isaacs joseph-isaacs changed the title Benchmark the unstable delta encoding in CI with a vortex-unstable format Write every benchmark's Vortex files with the unstable delta encoding Nov 17, 2026
@joseph-isaacs joseph-isaacs added the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Nov 17, 2026 — with Claude
@joseph-isaacs joseph-isaacs changed the title Write every benchmark's Vortex files with the unstable delta encoding Keep only delta in the unstable edition and benchmark with it Nov 18, 2026
bitpacked.v2, for.v2, decimal_byte_parts.v2, patched, union, the list
layout, the SBBF bloom filter and sum_v2 leave the unstable edition and
belong to no edition, so a writer that enforces editions no longer writes
them. An audit found them not ready: blocked bitpacked.v2 and for.v2 have
no slice, filter or take kernels, patched is gated by an environment
variable, nothing produces union, the V2 scan cannot read the list
layout, and the bloom filter is never used for pruning.

The edition record is regenerated with
`cargo run -p xtask -- generate-editions`.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs changed the title Keep only delta in the unstable edition and benchmark with it Enable the unstable edition in every benchmark Nov 18, 2026
@joseph-isaacs
joseph-isaacs changed the base branch from develop to ji/unstable-delta-only November 18, 2026 14:29
joseph-isaacs and others added 7 commits November 18, 2026 14:30
The unstable edition never freezes, so a test listing its members fails on
every membership change. Keep only the invariant that enabling it never
enables the vortex.all_* aggregates.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The unstable edition never freezes, so the test fails on every
membership change.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Drop the added doc comments and the btrblocks comment change, leaving the
one-line declaration change, the regenerated edition record, and the two
tests that hardcoded the old members.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The test hardcoded whether UNSTABLE permits decimal_byte_parts.v2, so it
broke whenever the unstable edition changed. It now checks whether the
session's enabled editions permit the v2 ID.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs added a commit that referenced this pull request Nov 18, 2026
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J
Base automatically changed from ji/unstable-delta-only to develop November 18, 2026 14:46
…codings-benchmarks-audit-79g1tu

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 18, 2026 14:47
## Summary

To estimate its ratio, `DeltaScheme` delta-encoded the whole array,
computed integer stats over every residual and took `max - min`. When
Delta won, `compress` then encoded the same array again. It was the only
sampled-style estimate that read every row.

Delta now delta-encodes 4 whole 1024-row chunks picked by
`sample_aligned_chunks`. Delta encodes each FastLanes chunk on its own,
with fresh bases. The residuals of a sampled aligned chunk are therefore
the same as the full encode's residuals for those rows. The one
exception is leading nulls, which are filled from the previous chunk.
The penalized ratio still uses the full length and padding.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs merged commit 3d6c8c2 into develop Nov 18, 2026
77 checks passed
@joseph-isaacs
joseph-isaacs deleted the claude/unstable-encodings-benchmarks-audit-79g1tu branch November 18, 2026 14:59
joseph-isaacs added a commit that referenced this pull request Nov 18, 2026
…chmarking

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This branch was successfully deployed

1 active deployment
vercel-docs — a23764f6 Deployed Nov 18, 2026 by joseph-isaacs via Publish preview / Deploy #10778
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants