Repository navigation
Estimate Delta from sampled FastLanes chunks - #11453
joseph-isaacs merged 11 commits into
Conversation
Adds a sampler that takes whole 1024-row FastLanes chunks, a temporary VX_SAMPLE_CHUNKS / VX_SAMPLE_CHUNK_SCHEMES switch, and a temporary sample_eval example to compare it against the current stratified sampler. The switch and example are experiment scaffolding and will be removed before this is opened as a PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The default compression sample joins 64-row runs from across the array, so a 1024-row FastLanes chunk of it mixes rows that were never adjacent. `sample_aligned_chunks` instead takes whole chunks that start at multiples of the chunk length, one from each equal partition of the array's full chunks, deterministically. Schemes that encode each chunk independently can estimate on it without that mixing. No scheme uses it yet. Switching the existing sampled schemes to it did not shrink files, so they keep the default sample. This replaces the earlier work-in-progress experiment commit, dropping its temporary VX_SAMPLE_CHUNKS switch, sample_eval example and their dev-dependencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Delta's estimate delta-encoded the whole array to measure the residual span, then `compress` encoded it again. It now delta-encodes four whole 1024-row chunks picked by `sample_aligned_chunks`. Delta encodes each chunk on its own with fresh bases, so those residuals match the full encode's for the same rows. Inside another scheme's sample, Delta still measures the array it is given, as before. A sample misses rare outliers. For series that restart mid-chunk, one large negative residual per restart used to widen the whole-array span to the full width, and Delta lost to FoR plus BitPacking. The cascade under Delta patches those few residuals, so the sampled span is the better guide. The new regression test covers that case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 12.18%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Every sampled scheme used to slice the same rows of an array, copy them, generate their stats, measure their size and check them for nulls again. The compressor now builds that sample once per array, caches it in the array's `ArrayAndStats`, and every sampled scheme, including the string dictionary callback, compresses the same `ArrayAndStats`. The sample keeps the rows and positions it had, and its stats use the merged options of every scheme the compressor considers, a superset of each scheme's own. Compressor output is unchanged; the golden snapshots do not move. Inside sample compression the array is already the sample, so it is used whole and shares the array's stats cache, but is not cached in it, which would make the cache own a handle to itself. `ArrayAndStats` is now `Clone`; clones share the stats cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…sample Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Integer and float arrays whose 1% sample spans at least eight 1024-row chunks now take those rows as whole chunks aligned to FastLanes chunk boundaries, rather than as 64-row runs joined from across the array. A sampled chunk then holds the rows the FastLanes encodings encode together. The sample keeps its size, about 1% of the rows. Shorter arrays, and every other dtype, keep the 64-row runs: one or a few contiguous chunks cover too little of a short array, and choosing one chunk instead of 16 spread runs for 10k-row columns made StatPopGen's small columns up to 18% larger. In practice this reaches arrays of about 800k rows or more, such as the elements of StatPopGen's per-sample list columns. Compressed sizes are unchanged on StatPopGen and on the AirQuality, r/place, TPC-H and synthetic test data, and StatPopGen compresses about 2% faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…sample Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Arrays of any dtype whose 1% sample spans at least eight 1024-row chunks now take that sample as whole aligned chunks, not just integer and float arrays. Shorter arrays keep the 64-row runs. Besides keeping FastLanes chunks whole, this cuts the sample from far fewer slices: about 330 instead of 5,300 for a 34M-row array. Slicing is costly when the array carries compressed validity, such as Sparse with Delta-encoded patch indices, where each slice binary-searches the indices and each probe decodes a Delta chunk. On StatPopGen with editions disabled, compression drops from 28.5 s to 11.6 s. Compressed sizes are unchanged on StatPopGen, with the default edition and with editions disabled, and on the AirQuality, r/place, TPC-H and synthetic test data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…sample Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…gned chunks (#11452) ## Summary 1. **One sample per array, shared by every scheme.** 2. **Long arrays are sampled as aligned chunks.** The sample is still about 1% of the rows. When those rows span at least eight 1024-row chunks, the sample takes them as whole chunks aligned to FastLanes boundaries. This applies to every dtype. Shorter arrays keep the current 3. **`sample_aligned_chunks`** is now public, for #11453. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J
Polar Signals Profiling ResultsLatest Run
Previous Runs (9)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -7.7% overall, 0↑ 1↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.017x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.981x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.8% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
File Size Changes (8 files changed, -12.3% overall, 2↑ 6↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
File Size Changes (16 files changed, -11.8% overall, 2↑ 14↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.990x ➖, 1↑ 0↓)
File Size Changes (24 files changed, -6.9% overall, 10↑ 14↓)
Totals:
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.048x ➖, 0↑ 4↓)
datafusion / parquet / ns (0.986x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -4.9% overall, 0↑ 100↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.912x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.953x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.893x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.968x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.044x ➖, 0↑ 17↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.032x ➖, 0↑ 10↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
File Size Changes (33 files changed, -10.2% overall, 7↑ 26↓)
Totals:
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (1.028x ➖, 0↑ 2↓)
vortex / vortex-file-compressed / bytes (0.956x ➖, 4↑ 0↓)
vortex / parquet / ns (0.997x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.996x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBA4uGoC1yCEQgJdM3J
a23764f
into
claude/unstable-encodings-benchmarks-audit-79g1tu
Delta's serialized form was settled in #11441, and with the fused decode and kernels (#11428) and the sampled estimator (#11453) it is cheap enough to offer for broad opt-in use. This moves `fastlanes.delta` from `unstable` into the draft `preview2026.08.0` edition, so Delta's wire format becomes fixed when that edition freezes. Writers that enable only `core` still never write Delta. Benchmark sessions now enable `preview` as well as `unstable` through `enable_unstable_encodings`, so every benchmark keeps writing Delta. This also enables `preview`'s other member, the `vortex.uncompressed_size_in_bytes` aggregate, in benchmark files. PR benchmarks against `develop`, which already writes Delta through `unstable`, are flat. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
To estimate its ratio,
DeltaSchemedelta-encoded the whole array, computed integer stats over every residual and tookmax - min. When Delta won,compressthen encoded the same array again. It was the only sampled-style estimate that read every row.Delta now delta-encodes 4 whole 1024-row chunks picked by
sample_aligned_chunks. Delta encodes each FastLanes chunk on its own, with fresh bases. The residuals of a sampled aligned chunk are therefore the same as the full encode's residuals for those rows. The one exception is leading nulls, which are filled from the previous chunk. The penalized ratio still uses the full length and padding.A sample can miss rare outliers. That turns out to help. Take series that restart in the middle of a chunk: each restart leaves one large negative residual, which used to widen the whole-array span to the full width, so Delta lost to FoR plus BitPacking. The cascade under Delta patches those few residuals instead of widening every residual. So the span of the sampled chunks predicts the compressed size better. This is the case #10233 went after by compressing the whole array through the cascade. This PR gets the same choice at a fraction of the cost.