Repository navigation
Enable the unstable edition in every benchmark - #11534
Conversation
Benchmarks can now write Vortex files whose compressor may also choose the array encodings of the unstable edition (bitpacked.v2, for.v2, delta, decimal_byte_parts.v2, and patched when VORTEX_EXPERIMENTAL_PATCHED_ARRAY=1). The files are written by a separate UNSTABLE_SESSION, so the existing vortex and vortex-compact formats are unchanged. The unstable layouts and aggregates stay disabled so every scan can read the files. The format is wired through data-gen, tpchgen, the DataFusion, DuckDB and compress benchmarks, and the bench orchestrator, but is not added to any CI preset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o Signed-off-by: Claude <noreply@anthropic.com>
The vortex-unstable benchmark format now enables only fastlanes.delta on top of the default editions. The audit found the other unstable array encodings not ready to benchmark: blocked bitpacked.v2 and for.v2 have no slice, filter or take kernels, patched is gated by an environment variable, and nothing produces union. The pr-full and develop presets run vortex-unstable on their NVMe targets for DataFusion and DuckDB. The file-size reporting scripts know the new format directory. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Merging this PR will improve performance by ×2.12 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Every SQL target set that runs vortex now also runs vortex-unstable, in every preset, including S3, nightly and the per-target TPC-DS SF=100 jobs. A matrix test asserts that pairing. The compression and random access benchmarks also run vortex-unstable: each random access dataset writes a -unstable file, and the split scripts and develop workflow pass the format. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
The Rust (docs) job rejects public docs that link private items. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Polar Signals Profiling ResultsLatest Run
Previous Runs (6)
Powered by Polar Signals Cloud |
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (1.065x ➖, 1↑ 4↓)
vortex / vortex-file-compressed / bytes (0.957x ➖, 4↑ 0↓)
vortex / parquet / ns (1.000x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.999x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
CompressionTimingMeasurement panicked on any format outside its fixed list, so compress-bench failed in CI once vortex-unstable was added. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
random-access-bench rejected --formats vortex-unstable because the format was missing from ALLOWED_FORMATS. Its measurement names also used the file extension, which vortex-unstable shares with vortex, so the two formats would have overwritten each other's results. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Replace the vortex-unstable benchmark format with vortex_bench::enable_unstable_encodings, which enables a bench-unstable edition holding only fastlanes.delta on a session. The benchmark SESSION and the string benchmark sessions call it, so every benchmark's existing vortex files may use delta, and PR runs compare them directly against the base. This reverts the vortex-unstable format plumbing in the benchmark binaries, CI presets, orchestrator and reporting scripts. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
bitpacked.v2, for.v2, decimal_byte_parts.v2, patched, union, the list layout, the SBBF bloom filter and sum_v2 leave the unstable edition and belong to no edition, so a writer that enforces editions no longer writes them. An audit found them not ready: blocked bitpacked.v2 and for.v2 have no slice, filter or take kernels, patched is gated by an environment variable, nothing produces union, the V2 scan cannot read the list layout, and the bloom filter is never used for pruning. The edition record is regenerated with `cargo run -p xtask -- generate-editions`. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The unstable edition never freezes, so a test listing its members fails on every membership change. Keep only the invariant that enabling it never enables the vortex.all_* aggregates. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The unstable edition never freezes, so the test fails on every membership change. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
Drop the added doc comments and the btrblocks comment change, leaving the one-line declaration change, the regenerated edition record, and the two tests that hardcoded the old members. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The test hardcoded whether UNSTABLE permits decimal_byte_parts.v2, so it broke whenever the unstable edition changed. It now checks whether the session's enabled editions permit the v2 ID. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDnhofVptQwrj9sj7o
…benchmarks-audit-79g1tu Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J
…codings-benchmarks-audit-79g1tu Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary To estimate its ratio, `DeltaScheme` delta-encoded the whole array, computed integer stats over every residual and took `max - min`. When Delta won, `compress` then encoded the same array again. It was the only sampled-style estimate that read every row. Delta now delta-encodes 4 whole 1024-row chunks picked by `sample_aligned_chunks`. Delta encodes each FastLanes chunk on its own, with fresh bases. The residuals of a sampled aligned chunk are therefore the same as the full encode's residuals for those rows. The one exception is leading nulls, which are filled from the previous chunk. The penalized ratio still uses the full length and padding. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…chmarking Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Benchmark sessions now enable the
unstableedition. Since #11595, that edition holds onlyfastlanes.delta. Every benchmark that writes Vortex files therefore writes them with delta available, and PR benchmark runs compare against the base with it.