Repository navigation
Conversation
gatesn
added a commit
that referenced
this pull request
Nov 3, 2026
Rebased onto `develop` at `8896ab27d0`, including the benchmark baseline merged in #11175. **Validation:** all 54 checks pass, including CodSpeed Performance Analysis. Only the human-review policy is pending. This optimizes read-time reconstruction of sparse lists. It reuses their existing elements and builds offsets directly, instead of rebuilding the lists through a builder. For example, suppose only two rows contain lists: ```text row: 0 1 2 3 4 value: [a,b] null null [c,d] null ``` The elements are already stored compactly as `[a,b,c,d]`. To represent all five rows as a native List, we only need offsets describing which elements belong to each row: ```text elements: [a, b, c, d] (reuse the existing array) offsets: [0, 2, 2, 2, 4, 4] (one boundary per row, plus the final boundary) ``` Previously, sparse decoding built the output by appending patch-list slices and fill gaps through a list builder. When the elements are a lazy gather, those appends can repeatedly execute the same gather indices. The new main path keeps the compact elements and constructs the offsets directly. Empty or null fill rows consume no elements, so their offsets repeat. Store those repetitions as runs: ```text patch positions: [0, 3] compact offsets: [0, 2, 4] offset run ends: [1, 4, 6] offset values: [0, 2, 4] decoded offsets: [0, 2, 2, 2, 4, 4] ``` This avoids rebuilding elements and keeps offsets compressed until a consumer needs them expanded. Arrow's native List export still materializes dense offsets. The optimization changes read-time reconstruction, not file compression or size. ## Implementation The main `empty_fill` function now follows the native-list path directly: reuse the compact patch elements, construct run-end offsets, and retain validity. `can_use_native_list` isolates the representability checks. `empty_fill_as_views` separately handles duplicate patch positions (last patch wins) and narrow view offsets whose terminal offset would overflow; it scatters independent offsets/sizes while sharing the elements. Nonempty fills and batches with at most one valid patch keep the existing builder path: there is no repeated element gather to avoid in the latter case. Duplicate positions still use the metadata path to preserve last-patch-wins semantics. Regression tests cover null/empty fills, repeated positions, genuine null patches, element-buffer sharing, the physical offset representation, and terminal-offset overflow. The sparse struct writer remains separate. ## Benchmark comparison ### CodSpeed CPU simulation [Final report](#11173 (comment)), comparing PR `c981287` with develop `1012d0c` (which includes the #11175 baseline): | Benchmark | Baseline µs | PR µs | Baseline / PR | | --- | ---: | ---: | ---: | | Null-fill list `(512, 7, 4)` | 423.4 | 200.3 | **2.11×** | | Null-fill list `(1024, 17, 8)` | 421.9 | 227.5 | **1.85×** | CodSpeed reports two improvements and no regressions. Both compressed-child union-take cases were measured in this run and no longer flag regressions. Keeping the builder for a single valid list avoids offset setup when there is no repeated gather to eliminate. These results measure CPU simulation. The local wall-clock measurements below use a different method and platform. ### Local wall-clock Local macOS arm64 measurement, Rust 1.99.1, `release_debug` with the repository's default codegen settings. Baseline `8896ab27d0` versus this PR at `c981287d20`; identical benchmark source from #11175. Five alternating/reversed baseline/candidate pairs, at least 1,000 samples per case and a minimum of 0.25 seconds per case, one thread. Values are medians of the five per-run medians. | Case | Rows / patch stride / list size | Baseline µs | PR µs | Speedup | | --- | --- | ---: | ---: | ---: | | Null-fill list | 512 / 7 / 4 | 11.240 | 2.666 | **4.22×** | | Null-fill list | 1024 / 17 / 8 | 10.620 | 2.874 | **3.70×** | | Nonempty-fill list control | 512 / 7 / 4 | 10.740 | 10.700 | 1.00× | | Nonempty-fill list control | 1024 / 17 / 8 | 9.040 | 9.082 | 1.00× | | Fixed-size-list control | 512 / 7 / 4 | 11.200 | 11.160 | 1.00× | | Fixed-size-list control | 1024 / 17 / 8 | 10.870 | 10.830 | 1.00× | The null-fill cases improve substantially; this microbenchmark's control changes are within 1%. With the single-valid-patch guard, five alternating local union-take pairs measure 6.040 → 5.874 µs and 7.291 → 7.207 µs for the compressed-child cases; dense-child controls are unchanged. The final CodSpeed report also flags no union-take regressions. This measures top-level canonicalization with primitive patch elements, not Arrow export or the separate lazy-gather workload. Both binaries were built in the same dedicated checkout, preserving the baseline executable before checking out the candidate. Each was run with: ```sh <benchmark-binary> --bench --color never --threads 1 --sample-count 1000 --min-time 0.25 ``` Raw local results and the comparison script are saved at `/private/tmp/vortex-11173-benchmark-results`. ## Validation - `cargo clippy -p vortex-sparse --all-targets --all-features -- -D warnings` - `cargo nextest run -p vortex-sparse --lib --all-features` — 129 passed, including single-valid-patch and duplicate-position coverage. - `cargo +nightly-2026-09-10 fmt -p vortex-sparse --check` - `git diff --check` Targets `develop` directly. Merge before #11174: that dictionary/sparse-code rewrite exposes the expensive list path addressed here. #11172 independently avoids rebuilding null-only constant list batches. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn
force-pushed
the
ngates/push-dict-through-sparse-codes
branch
from
November 6, 2026 16:18
cd178b9 to
9b6a01b
Compare
Register the dictionary-over-sparse-codes rewrite with the session's parent-reduce rules, and optimize with the session in its tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gatesn
marked this pull request as ready for review
November 20, 2026 17:22
Merging this PR will improve performance by 64%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
The sparse codes must share the null fill's nullable dtype. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gatesn
changed the base branch from
develop
to
ngates/sparse-dict-codes-bench
November 20, 2026 18:13
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A dictionary whose codes are null-filled sparse patches,
Dict(Sparse(positions, codes, NULL), values), is rewritten toSparse(positions, Dict(codes, values), NULL). Executing the original shape expands the codes to full length before the dictionary lookup, so its cost scales with the row count. After the rewrite, the lookup touches only the patch codes and the result stays sparse. The rewrite is exact because a null code yields a null output: the null fill passes through unchanged and needs no lookup of its own.The rewrite is registered as a reduce-parent rule, for optimization, and as an execute-parent kernel for
Dict. When the sparse array is the dictionary's values child rather than its codes, execution falls through to the existing take kernel. A second kernel handles a lazySlicebetween the dictionary and the sparse codes. It slices the sparse positions first, so the sparse form survives long enough to be rewritten. If slicing collapses the codes to another encoding, such as a constant for a range with no patches, the kernel does an ordinary lookup.The rewrite applies only when the fill is null and the output dtype can be decoded as sparse all the way down (
supports_sparse). Struct and list dtypes are checked recursively, because decoding a sparse struct distributes sparsity into its fields. A non-null fill would need its own lookup and is left to the existing path.For review: check the
child_idxhandling inrules/dict_codes/mod.rs, where child 0 is the codes and child 1 the values, and thesupports_sparserecursion. The tests cover:CodSpeed, measured against the benchmark added in #11698 (65,536 rows, 1,000-entry dictionary), shows no regressions on the other 1,982 benchmarks:
canonicalize_dict_sparse_codes_utf8[100]canonicalize_dict_sparse_codes_utf8[10]canonicalize_dict_sparse_codes_primitive[100]canonicalize_dict_sparse_codes_primitive[10]Stacked on #11698.
🤖 Generated with Claude Code