Skip to content

feat: harden delta serde ahead of format stabilization - #11441

Merged
joseph-isaacs merged 2 commits into
developfrom
ji/vibrant-babbage-3s1djx
Nov 13, 2026
Merged

joseph-isaacs merged 2 commits into
developfrom
ji/vibrant-babbage-3s1djx

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

fastlanes.delta is still only in the unstable edition. This PR fixes its serialized contract before it is frozen. A review of Delta's serde found two problems:

  • Validity is inferred from the child count. A third child is always read as validity, so the format can't add a child later without it being mistaken for validity.
  • The reader is not robust to corrupt input.
    • A float dtype panics in lane_count before validate_parts runs.
    • deltas_len is not tied to len, so a tiny array with constant children can claim an unbounded physical length and force a huge allocation on decode.

Record whether a validity child is present in `DeltaMetadata` instead of
inferring it from the child count, and accept trailing children so a later
writer can append children that old readers may safely ignore. Children
that change the decoded values still need a new encoding ID.

Reject corrupt input before it can panic or force large allocations:
- non-integer dtypes, which previously panicked in `lane_count`
- a deltas length other than `(offset + len)` rounded up to a whole chunk,
  which let a tiny array claim an unbounded physical length
- offsets outside the first chunk, checked before the child lengths

Serialize rejects arrays whose deltas extend past the chunk holding their
last value, and decompression asserts whole chunks in release builds since
`set_len` relies on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review November 13, 2026 13:22
@joseph-isaacs
joseph-isaacs force-pushed the ji/vibrant-babbage-3s1djx branch from ad566a4 to c605f2c Compare November 13, 2026 13:23
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Nov 13, 2026
@joseph-isaacs joseph-isaacs changed the title Harden Delta serde ahead of format stabilization feat: harden delta serde ahead of format stabilization Nov 13, 2026
`delta_serde_sliced_signed` sliced the source validity lazily, which left
a `vortex.slice` child that cannot be serialized. Build the sliced
validity directly instead. Also drop redundant clones that clippy rejects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs merged commit e780b53 into develop Nov 13, 2026
69 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/vibrant-babbage-3s1djx branch November 13, 2026 14:04
joseph-isaacs added a commit that referenced this pull request Nov 20, 2026
Delta's serialized form was settled in #11441, and with the fused decode
and kernels (#11428) and the sampled estimator (#11453) it is cheap
enough to offer for broad opt-in use. This moves `fastlanes.delta` from
`unstable` into the draft `preview2026.08.0` edition, so Delta's wire
format becomes fixed when that edition freezes. Writers that enable only
`core` still never write Delta.

Benchmark sessions now enable `preview` as well as `unstable` through
`enable_unstable_encodings`, so every benchmark keeps writing Delta.
This also enables `preview`'s other member, the
`vortex.uncompressed_size_in_bytes` aggregate, in benchmark files. PR
benchmarks against `develop`, which already writes Delta through
`unstable`, are flat.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants