Repository navigation
Write patch chunk offsets for sparse arrays as vortex.sparse.v2 - #11650
Closed
robert3005 wants to merge 3 commits into
Closed
robert3005 wants to merge 3 commits into
robert3005 wants to merge 3 commits into
Conversation
The sparse encoder never wrote patch chunk offsets and the deserializer rejected a third child, so readers had to scan all patches to find one chunk's. Sparse::encode now writes them, the btrblocks sparse schemes keep them as a compressed child, and deserialization reads them when present. Files without them read as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtyRD8x899QARbaLjm Signed-off-by: Robert Kruszewski <github@robertk.io>
vortex.sparse is in the frozen core2025.05.0 edition and its readers require exactly two children, so adding the chunk offsets child under that ID would let writers targeting old editions produce files old readers reject. Arrays with chunk offsets now serialize as vortex.sparse.v2, a member of the unstable edition, and arrays without them still write v1. Sparse::encode no longer writes chunk offsets; encode_with_chunk_offsets opts in. Each btrblocks sparse scheme refines to a variant that writes them only when the v2 ID is allowed. The v2 reader does not check the child count, so later revisions can add children without breaking it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtyRD8x899QARbaLjm Signed-off-by: Robert Kruszewski <github@robertk.io>
Merging this PR will improve performance by 12.75%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
The fixture covers sparse arrays with patch chunk offsets across several chunks. Like the decimal_byte_parts v2 fixture, it stays disabled until vortex.sparse.v2 joins an edition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtyRD8x899QARbaLjm Signed-off-by: Robert Kruszewski <github@robertk.io>
Contributor
Author
|
We are going to do a Patched array instead of revving this. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sparse arrays never wrote patch chunk offsets, so a reader had to scan every patch to find one chunk's, and the deserializer dropped any offsets it saw. This mirrors #11644 for ALP-RD.
vortex.sparsebelongs to the frozencore2025.05.0edition, and its readers require exactly two children. So arrays with chunk offsets serialize under a newvortex.sparse.v2ID in theunstableedition, following thedecimal_byte_parts.v2pattern. Arrays without offsets still write v1. The v2 reader doesn't check the child count, so later revisions can add children without breaking existing readers.Sparse::encode_with_chunk_offsetsopts in, andSparse::encodeis unchanged. Each btrblocks sparse scheme refines to a variant that writes and compresses the offsets only when the v2 ID is allowed. Default writers on frozen editions produce the same files as before, apart from the footer's list of permitted IDs in sessions that enable every registered encoding. That is why the expected sizes intest_stock_ticker_file_sizegrow by 40 bytes.Related: #11644
Generated by Claude Code