Repository navigation
Conversation
Signed-off-by: Nicholas Gates <nick@nickgates.com>
# Conflicts: # vortex-btrblocks/src/tests.rs
I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: ee8a9b3 I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: f97753e I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: f7eb9e2 Signed-off-by: Nicholas Gates <nick@nickgates.com>
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
vortex / vortex-file-compressed / bytes (0.995x ➖, 1↑ 0↓)
vortex / parquet / ns (0.996x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.992x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a BtrBlocks integer scheme,
vortex.int.runend_sequence, for the run-ends child of a run-end array. It encodes the ends asChunked[Sequence, Constant(1)]: an arithmetic prefix followed by a single terminal value. This is the shape run-end encoding produces for the offsets of a sparse list column whose present rows are clustered. Rows before the cluster form one long run, each present row is its own run, and the rows after it form one long final run. The ends are thereforestart, start+1, …, start+k, len. Everything but the last value is a sequence, so the whole child costs a few dozen bytes, whatever the row count. A plainSequencecannot represent it because of the terminal value, while bit-packing or delta pays per run. The run values (0, 3, 6, …) are already caught by the existing sequence scheme.The scheme is gated to stay cheap on every other integer column. It only runs on the ends child of a run-end cascade, never on samples, and only on non-null arrays of at least four values. Its estimate is deferred and exact: it encodes the prefix with
sequence_encodeand is skipped up front when the best scheme so far already beats the ratio this one could reach. It is excluded from the CUDA compression mode, like the other schemes without a GPU decode path.For review:
(RunEndScheme.id(), 1), which must select the ends child and not the values;nbytes / 24) used to skip the attempt;builder.rs.The new test compresses clamped, list-offset-shaped data and checks that the ends become
Chunked.Compression bench on the sparse-wide datasets, against develop. Hot times; every other dataset is byte-identical:
The timing differences on datasets whose files are unchanged are run-to-run noise. Shuffled lists don't benefit, because their present rows aren't contiguous, so their offsets don't form a single arithmetic run. #11160 builds on this for nested and shuffled data.
🤖 Generated with Claude Code