Repository navigation
feat: harden delta serde ahead of format stabilization - #11441
Merged
Merged
Conversation
Record whether a validity child is present in `DeltaMetadata` instead of inferring it from the child count, and accept trailing children so a later writer can append children that old readers may safely ignore. Children that change the decoded values still need a new encoding ID. Reject corrupt input before it can panic or force large allocations: - non-integer dtypes, which previously panicked in `lane_count` - a deltas length other than `(offset + len)` rounded up to a whole chunk, which let a tiny array claim an unbounded physical length - offsets outside the first chunk, checked before the child lengths Serialize rejects arrays whose deltas extend past the chunk holding their last value, and decompression asserts whole chunks in release builds since `set_len` relies on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
robert3005
approved these changes
Nov 13, 2026
joseph-isaacs
marked this pull request as ready for review
November 13, 2026 13:22
joseph-isaacs
force-pushed
the
ji/vibrant-babbage-3s1djx
branch
from
November 13, 2026 13:23
ad566a4 to
c605f2c
Compare
delta serde ahead of format stabilization
`delta_serde_sliced_signed` sliced the source validity lazily, which left a `vortex.slice` child that cannot be serialized. Build the sliced validity directly instead. Also drop redundant clones that clippy rejects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added a commit
that referenced
this pull request
Nov 20, 2026
Delta's serialized form was settled in #11441, and with the fused decode and kernels (#11428) and the sampled estimator (#11453) it is cheap enough to offer for broad opt-in use. This moves `fastlanes.delta` from `unstable` into the draft `preview2026.08.0` edition, so Delta's wire format becomes fixed when that edition freezes. Writers that enable only `core` still never write Delta. Benchmark sessions now enable `preview` as well as `unstable` through `enable_unstable_encodings`, so every benchmark keeps writing Delta. This also enables `preview`'s other member, the `vortex.uncompressed_size_in_bytes` aggregate, in benchmark files. PR benchmarks against `develop`, which already writes Delta through `unstable`, are flat. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fastlanes.deltais still only in the unstable edition. This PR fixes its serialized contract before it is frozen. A review of Delta's serde found two problems:lane_countbeforevalidate_partsruns.deltas_lenis not tied tolen, so a tiny array with constant children can claim an unbounded physical length and force a huge allocation on decode.