Repository navigation
Harden serde of unstable encodings and the list layout, and document wire-contract design - #11442
Draft
joseph-isaacs wants to merge 8 commits into
Draft
joseph-isaacs wants to merge 8 commits into
joseph-isaacs wants to merge 8 commits into
Conversation
Accept trailing children for `fastlanes.for.v2` so a later writer can append children that old readers may safely ignore, and document the rule on `FoRV2Metadata`. Children that change the decoded values still need a new encoding ID. Both v2 children are required, so no metadata field is needed to locate them and the wire bytes are unchanged. Reject corrupt input before fetching children: - non-integer dtypes - an `offset + len` that overflows usize, which would panic when counting the reference chunks `fastlanes.for` (v1) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`vortex.union` already locates every child from the dtype alone: the type IDs, then one child per variant, with outer nulls carried by the type IDs. Document that wire contract on `Union` and accept trailing children so a later writer can append children that old readers may safely ignore. Children that change the decoded values still need a new encoding ID. Add tests for the round trip, an ignored trailing child, and each deserialize guard: too few children, a non-union dtype, non-empty metadata or buffers, and decoding against the wrong length or variant dtypes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Readers previously rejected any non-empty metadata, which would stop a later writer from adding a field that describes an appended child old readers may ignore. Decode it as a field-less protobuf message instead, so unknown fields are ignored and malformed bytes are still rejected. The serialized form is unchanged: the message encodes to zero bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Accept trailing children under `vortex.decimal_byte_parts.v2` so a later writer can append children that old readers may safely ignore, and document the rule on `DecimalBytePartsV2Metadata`. Children that change the decoded values still need a new encoding ID. Every child position already follows from the metadata: the MSP carries validity, and `lower_part_ptypes` declares one child per lower part. Validate the MSP and every lower-part ptype before reading any child, since the in-memory children panic on a dtype mismatch, and name the MSP in the error for an out-of-range ptype. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Document the `vortex.list` layout wire contract on `ListLayoutMetadata`: the children are `[elements, offsets]`, followed by a validity child exactly when the list dtype is nullable, so each child's position is a function of the dtype alone. Readers now require at least that many children and ignore trailing ones, so a later writer can append children that old readers may safely skip, such as an index. Children or metadata that change the decoded values still need a new layout ID. Reject corrupt input with errors: - offsets ptypes that are not integers, such as f32 - offsets whose row count is not exactly the list row count plus one; the old check saturated and accepted an empty offsets child - list row counts that overflow when adding the extra offset The writer already emits exactly the contracted children, and keeps its exact-count check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Document the `vortex.patched` wire contract on `PatchedMetadata` and accept trailing children so a later writer can append children that old readers may safely ignore. Children that change the decoded values still need a new encoding ID. Patched has no optional children, so the metadata fields and their encoding are unchanged. Read the patch values child with the non-nullable outer dtype. Patching ignores the validity of the patch values, and every writer already stores them as non-nullable, so a nullable values child could only drop nulls silently. Reject corrupt input before reading children: - any buffers, or fewer than four children - non-primitive dtypes - lane counts that are not a power of two between 1 and 128, which included zero lanes that panicked on `index % n_lanes` in `scalar_at` - offsets outside the first chunk - `offset + len` and the lane offsets length overflowing usize `PatchedData::validate` now also checks the dtypes of the lane offsets, patch indices and patch values and the length of the lane offsets. Serialize rejects arrays whose lane offsets length a reader could not derive from the metadata. Move the vtable tests into `vtable/tests.rs` and add a nullable sliced round-trip and negative tests for each guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Record whether a validity child is present in `BitPackedV2Metadata` instead of inferring it from the child count, and write the block offsets first so no child is located relative to the end. The children are now the block offsets, then the patch indices, values and optional chunk offsets when the metadata has patches, then the validity when `has_validity` is set. Readers accept trailing children so a later writer can append children that old readers may safely ignore. Children that change the decoded values still need a new encoding ID. `fastlanes.bitpacked.v2` is only in the unstable edition, so the contract changes in place; `fastlanes.bitpacked` is unchanged. Reject corrupt input before reading any child, so it can neither panic on the in-memory child assertions nor force large allocations: - non-integer dtypes, which previously panicked in `as_ptype` - a validity child declared for a non-nullable dtype - non-unsigned block offset types - `offset + len` overflowing, and packed buffers that are not whole 128-byte rows or exceed the blocks at the type's full bit width - patch counts outside `1..=len`, signed chunk offset types, and chunk offset counts outside `1..=len.div_ceil(1024) + 1` Host-resident block offsets are also validated after they are read, in all builds, where debug builds previously panicked on invalid boundaries. Pin the metadata encoding in a new `bitpacked_v2.metadata` golden file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Add a versioning page that sets out the rules the serde hardening applies: child positions follow from metadata and dtype, readers accept trailing children that are safe to ignore, metadata is always decoded as a protobuf message, value-changing extensions take a new wire ID, and readers reject malformed input with errors. Link it from the versioning index, the design page's section on new formats, and AGENTS.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This was referenced Nov 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This applies the serialized-contract pattern from the Delta PR (#11441) to every other UNSTABLE component that has children, and writes the pattern down as general guidance. The two PRs are independent.
The pattern:
children.len()or located relative to the end of the child list.children.get, because the in-memoryArrayChildrenasserts.These IDs are all in the UNSTABLE edition, so the contracts change in place under the same IDs. Frozen v1 IDs (
fastlanes.bitpacked,fastlanes.for,vortex.decimal_byte_parts) and thevortex.listarray are untouched.Changes
Docs
docs/specs/versioning/wire-contracts.md("Designing wire contracts") covers how to make a format forward compatible:design.md.AGENTS.mdgains one line pointing to the page for anyone adding or changing a serialized format.Encodings
fastlanes.bitpacked.v2[patches…, validity?, block_offsets]to[block_offsets, patches…, validity?]; metadata gainshas_validity(tag 4); serialize builds the child list explicitlyhas_validityneeds a nullable dtype; unsigned block-offset ptype;offset + lenoverflow; packed buffer size bound; patch metadata bounds (count, indices/chunk-offset ptypes, chunk-offset count); host block offsets validated in release buildsfastlanes.for.v2offset + lenoverflowvortex.decimal_byte_parts.v2vortex.patchedn_lanesa power of two ≤ 128 (0 used to divide by zero);offset < 1024; checked lane-offsets length; child dtypes and lengths invalidatevortex.unionvortex.listlayoutrow_count + 1, with overflow checkEach component gets tests for:
bitpacked.v2also gets a new golden file,goldenfiles/bitpacked_v2.metadata. Its bytes were computed by hand with every field set to a non-default value.Patched's inline tests moved to
patched/vtable/tests.rs.Left for follow-up
These need a data scan, or kernel bounds checks, to detect:
Patches::newon corrupt indices.Checks
None have been run yet, so this is a draft:
rustfmt, not the pinned nightly;check_metadata.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBBYkMiCtjPPKVBhjRj