Repository navigation
Conversation
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn
changed the base branch from
develop
to
ngates/compress-runend-sequences
November 20, 2026 17:13
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (3)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (0.992x ➖, 2↑ 0↓)
vortex / parquet / ns (0.996x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.992x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Contributor
|
You have pretty bad regressions in the compress benchmarks :/ |
Selecting defaults into a list canonicalizes it to a ListView, which made the sparse-wide list datasets larger and several times slower to compress and decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Zip leaves its validity as a lazy zip array, which the flat writer cannot serialize. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Temporary, to measure the leaf rewrite apart from collapsing struct validity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The struct layout writer normalizes the validity of children beneath nullable structs before splitting a chunk into columns. When an ancestor struct is null at a row, nothing below it at that row is observable. The writer therefore marks inner structs valid at those rows, and replaces hidden leaf values with the type's default, also valid. Visible rows and visible nulls are untouched, and the outermost validity still records which rows are null, so every value read through the struct is identical. On disk, each level no longer repeats its ancestors' null bitmap, and hidden leaf rows no longer carry arbitrary physical bytes into the compressor. Arrow-sourced nulls, for example, can hold any view or offset. In wide, sparse, nested data, that redundant and noisy content dominates a column.
Defaults are substituted for Bool, Primitive, Decimal, Utf8 and Binary leaves. List and FixedSizeList leaves keep their own nulls: selecting into a list canonicalizes it to a
ListView, which compresses and decodes far worse than the incoming list. Union and variant leaves have no cheap default and are also left alone. The leaf rewrite executes the selection withRecursiveCanonical, so the writers receive a materialized validity bitmap rather than a lazy zip.For review: check that the reader always reapplies ancestor validity when a nested field is projected or filtered on its own. A child column read without its ancestors would show defaults where the input had nulls; the
struct_::readernested-validity tests cover projection, filtering andis_nullat each level. Also reviewnormalize_struct_childreninlayouts/struct_/normalize.rs: it recurses onvisible & validityand widens each struct's validity tovalidity | !visible.Compression bench on the sparse-wide datasets, together with #11159 (this PR's base), against develop. Hot times; every other dataset is byte-identical:
#11159 alone accounts for −12.0% on lists-runs; the rest of the size reduction comes from this PR. In CI, lists-shuffled compresses in 562–570 ms across three runs, against 296 ms on develop. That matches lists-runs, which takes 678 ms on develop. Nothing in that dataset's leaves is rewritten; only struct validity changes. The slowdown does not reproduce on an Apple M-series machine with the bench's CI settings (
FLAT_LAYOUT_INLINE_ARRAY_NODE=1), even though the files written are byte-identical to CI's. There, single-threaded compression of the list datasets costs 2–7% more, normalization accounts for 1–2% of CPU, and 16-thread timings are unchanged. The CI slowdown is specific to the benchmark hardware and remains to be profiled there.Stacked on #11159.
🤖 Generated with Claude Code