Skip to content

Write patch chunk offsets for sparse arrays as vortex.sparse.v2 - #11650

Closed
robert3005 wants to merge 3 commits into
developfrom
claude/stoic-shannon-pnu86e
Closed

robert3005 wants to merge 3 commits into
developfrom
claude/stoic-shannon-pnu86e

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Sparse arrays never wrote patch chunk offsets, so a reader had to scan every patch to find one chunk's, and the deserializer dropped any offsets it saw. This mirrors #11644 for ALP-RD. vortex.sparse belongs to the frozen core2025.05.0 edition, and its readers require exactly two children. So arrays with chunk offsets serialize under a new vortex.sparse.v2 ID in the unstable edition, following the decimal_byte_parts.v2 pattern. Arrays without offsets still write v1. The v2 reader doesn't check the child count, so later revisions can add children without breaking existing readers.

Sparse::encode_with_chunk_offsets opts in, and Sparse::encode is unchanged. Each btrblocks sparse scheme refines to a variant that writes and compresses the offsets only when the v2 ID is allowed. Default writers on frozen editions produce the same files as before, apart from the footer's list of permitted IDs in sessions that enable every registered encoding. That is why the expected sizes in test_stock_ticker_file_size grow by 40 bytes.

Related: #11644


Generated by Claude Code

robert3005 and others added 2 commits November 19, 2026 18:11
The sparse encoder never wrote patch chunk offsets and the deserializer
rejected a third child, so readers had to scan all patches to find one
chunk's. Sparse::encode now writes them, the btrblocks sparse schemes keep
them as a compressed child, and deserialization reads them when present.
Files without them read as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEtyRD8x899QARbaLjm
Signed-off-by: Robert Kruszewski <github@robertk.io>
vortex.sparse is in the frozen core2025.05.0 edition and its readers
require exactly two children, so adding the chunk offsets child under that
ID would let writers targeting old editions produce files old readers
reject. Arrays with chunk offsets now serialize as vortex.sparse.v2, a
member of the unstable edition, and arrays without them still write v1.

Sparse::encode no longer writes chunk offsets; encode_with_chunk_offsets
opts in. Each btrblocks sparse scheme refines to a variant that writes
them only when the v2 ID is allowed. The v2 reader does not check the
child count, so later revisions can add children without breaking it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEtyRD8x899QARbaLjm
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the changelog/feature A new feature label Nov 19, 2026 — with Claude
@codspeed

codspeed Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 12.75%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 2 improved benchmarks
✅ 1980 untouched benchmarks
⏩ 57 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime random_rotating_avx2[(0.9, 0.9, "self_90pct_mask_90pct")] 9.5 µs 8.3 µs +15.11%
⚡ Walltime decode_blocked_bitpacked_chunked_neon[u32, 524288] 31 µs 28 µs +10.43%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/stoic-shannon-pnu86e (e6b14c8) with develop (d368834)

Open in CodSpeed

Footnotes

  1. 57 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

The fixture covers sparse arrays with patch chunk offsets across several
chunks. Like the decimal_byte_parts v2 fixture, it stays disabled until
vortex.sparse.v2 joins an edition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbEtyRD8x899QARbaLjm
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005

Copy link
Copy Markdown
Contributor Author

We are going to do a Patched array instead of revving this.

@robert3005 robert3005 closed this Nov 20, 2026
@robert3005
robert3005 deleted the claude/stoic-shannon-pnu86e branch November 20, 2026 11:48

This branch was successfully deployed

1 active deployment
vercel-docs — e6b14c89 Deployed Nov 19, 2026 by robert3005 via Publish preview / Deploy #11335
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant