Repository navigation
Add PatchesV2 with chunk-local patch indices - #9611
joseph-isaacs wants to merge 3 commits into
Conversation
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | cold_misaligned[(16, 64)] |
345.6 µs | 390.8 µs | -11.57% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.5 µs | 12.3 µs | +67.05% |
| 🆕 | Simulation | search_index_above_max_v2 |
N/A | 19.5 µs | N/A |
| 🆕 | Simulation | search_index_below_min_v2 |
N/A | 18.5 µs | N/A |
| 🆕 | Simulation | search_index_full_range_random_v2 |
N/A | 21.6 µs | N/A |
| 🆕 | Simulation | search_index_in_range_v2 |
N/A | 20.2 µs | N/A |
| 🆕 | Simulation | search_index_mixed_out_of_range_v2 |
N/A | 18.8 µs | N/A |
| 🆕 | Simulation | apply_full_range |
N/A | 9.5 µs | N/A |
| 🆕 | Simulation | apply_full_range_v2 |
N/A | 16.2 µs | N/A |
| 🆕 | Simulation | slice_unaligned |
N/A | 44.2 µs | N/A |
| 🆕 | Simulation | slice_unaligned_v2 |
N/A | 87.9 µs | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/patches-v2-local-indices (5827196) with develop (68e2aee)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
|
Don't we want to move to the world of PatchedArray and we can make all the changes there? |
Polar Signals Profiling ResultsLatest Run
Previous Runs (20)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -46.4% overall, 0↑ 3↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (10 files changed, -45.5% overall, 0↑ 10↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.937x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.018x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.990x ➖, 1↑ 0↓)
File Size Changes (201 files changed, -42.9% overall, 52↑ 149↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 3↑ 7↓)
duckdb / parquet / ns (1.006x ➖, 4↑ 6↓)
File Size Changes (25 files changed, -43.9% overall, 0↑ 25↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.022x ➖, 0↑ 1↓)
File Size Changes (10 files changed, -45.6% overall, 0↑ 10↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.999x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -37.7% overall, 1↑ 2↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.881x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.180x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.964x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.122x ➖, 1↑ 5↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 1↑ 1↓)
File Size Changes (101 files changed, -39.5% overall, 0↑ 101↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.976x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.971x ➖, 0↑ 0↓)
|
|
CodSpeed The Simulation-mode instruction-count regression (~-11%) persists on head Local wall-clock comparison (same machine, same commands, 100 samples each):
Median delta is ~1%, within run-to-run variance (slowest samples swing 50–68 µs on both builds). Conclusion: the instruction-count delta is a codegen/inlining reshuffle in shared dependencies (vortex-btrblocks is a direct dep of vortex-file), not a runtime regression on this benchmark. Generated by Claude Code |
`Patches` addresses outliers with global row indices, which forces a wide index child (u32/u64 for large arrays) and threads `offset` / `offset_within_chunk` saturating adjustments through every chunked lookup and slice. `PatchesV2` stores the same information addressed by chunk-local u16 indices: positions are local to each 1024-value chunk, `chunk_offsets` holds required u32 prefix patch counts rebased on every slice, and an `offset` within the first chunk keeps unaligned slices on the chunk grid. Lookups are an O(1) chunk select plus an in-chunk binary search, and slices are self-contained rather than carrying saturating adjustments. The container addresses patches by position only, so it carries values of any dtype. Tests cover that with randomized model-based cases (search, slice, nested slice, apply, and the global-index round trip against a reference model over fixed seeds) plus extension-typed values. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Wires the chunk-local form into two real decompression paths, behind `VORTEX_PATCHES_V2_SCATTER=1` / `force_patches_v2_scatter`: - BitPacked and FoR decompression scatter chunked patch sets through `PatchesV2::apply_each` instead of materializing global indices. - Sparse canonicalization of a constant fill scatters through `PatchesV2::apply_into` when the types match, i.e. when the result is a non-nullable primitive and there is no validity to patch, so the whole operation is a value scatter over the fill. Converting a global-index patch set per decompression costs a pass and allocations over the patch set, so both stay opt-in until the stored layout is chunk-local; the default paths are unchanged and the branch is kept out of line in the bitpacked scatter loop. Tests assert both paths agree with their defaults on real patched arrays, and that a `Sparse` array rebuilt through `PatchesV2` is identical to the original across every dtype `Sparse` supports: null, bool, the primitive widths, decimal, utf8 (VarBin and VarBinView), binary, list, fixed-size list and struct. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Patch indices are sorted and bounded by the array length, so they bitpack well. Compressing the index child narrows patch sets whose indices would otherwise be stored at full width. This stays opt-in behind `force_patch_index_bitpack` while the golden corpus is regenerated, so default output is unchanged. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
d3d6359 to
5827196
Compare
|
Do you think we want a single patches array with 2 options (global vs local indices) Or should we have two arrays one for each indices setup? |
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Summary
Patchesaddresses outliers with global row indices, which forces a wide index child (u32/u64for large arrays) and threadsoffset/offset_within_chunksaturating adjustments through every chunked lookup and slice. This PR addsPatchesV2, a patch container addressed by chunk-localu16indices, and exercises it on two real decompression paths.Changes
vortex-array/src/patches_v2/with thePatchesV2struct:indices:u16positions local to each 1024-value chunk (PATCH_CHUNK_SIZE), strictly sorted per chunk.chunk_offsets: required, non-nullableu32prefix patch counts with a leading zero — rebased on every slice, so slices are self-contained and lookups need no saturating-adjustment bookkeeping.offsetin0..PATCH_CHUNK_SIZEplaces logical element zero inside the first chunk, keeping unaligned slices on the chunk grid.try_new/new_unchecked,search_indexandget_patched(O(1) chunk select + in-chunk binary search),view()returning a resolved borrowedPatchesV2Viewfor hot loops (plain slice reads, no dispatch or allocation),apply_each/apply_into(the decompression scatter primitives), rebasingslice, andfrom_patches/to_patchesconversions.VORTEX_PATCHES_V2_SCATTER=1/force_patches_v2_scatter:apply_patches_to_uninit_range(BitPacked and FoR decompression) scatters chunked patch sets throughPatchesV2.Sparsecanonicalization of a constant fill scatters throughPatchesV2when the types match — a non-nullable primitive result, where there is no validity to patch and the operation is purely a value scatter over the fill.compress_patches(force_patch_index_bitpack), default output unchanged.patches_lookupbenchmark gains_v2search variants plus apply and slice comparisons.Converting a global-index patch set per decompression costs a pass and allocations over the patch set, so both scatter integrations stay opt-in until the stored layout is chunk-local.
Types it works for
The container addresses patches by position only, so nothing in it is dtype-specific.
encodings/sparse/tests/patches_v2_stand_in.rspins that down againstSparse: for each dtype it builds a constant-fillSparsearray, round-trips its patches throughPatchesV2, and asserts the rebuilt array is identical, plus that scalar lookups and slices agree.Covered:
null,bool,u8/i16/u32/i64/f32/f64,decimal,utf8(bothVarBinandVarBinView),binary,list,fixed_size_list,struct. Extension-typed values are covered at the container level invortex-array. The remaining dtypes —Map,Union,Variant,Extension— are gaps inSparse's own canonicalization (vortex_bail!/todo!()), not in this container.Tests
search_index,get_patched,apply_each,apply_into, nestedslice, and the global-index round trip, each checked against a reference model of the patched rows.Sparsetest also asserts the scatter count moved, so it cannot pass vacuously by silently taking the default path.Benchmarks
cargo bench -p vortex-array --bench patches_lookup(1M rows, 100 patches, 100 queries per iteration, medians):Search matches unchunked
Patcheswhile keeping O(1) chunk select, and is ~40× faster than the chunkedPatchespath (which pays per-query child slicing and generic dispatch). Apply and slice are behind v1 at this patch density because reconstructing positions walks ~1000 chunk offsets for only 100 patches; at realistic patch densities (patches ≥ chunks) the cursor walk amortizes. The v1 numbers also exclude the wider on-disk indices v1 pays for.This lands the container plus the scatter integrations; migrating
BitPacked's stored layout onto chunk-local indices is follow-up work.Checks run
cargo testforvortex-array(3459),vortex-sparse(141),vortex-fastlanes(321) andvortex-btrblocks(51);cargo clippy --all-targetson those four crates;cargo +nightly fmt --all --check;cargo check -p vortex-array --all-targetsat the first commit for bisectability.Not fixed here:
golden::golden_compact'scompact__list_of_int_runssnapshot fails locally under--features unstable_encodings,zstd,pco. It fails identically at this branch's merge base, so it is pre-existing and unrelated; it passes on CI.