Repository navigation
fix(arrow): import fixed-size lists whose null rows hold null items - #11497
Merged
robert3005 merged 1 commit intoNov 16, 2026
Merged
robert3005 merged 1 commit into
robert3005 merged 1 commit into
Conversation
Arrow leaves the items of a null list undefined, so writers such as pyarrow mark them null even when the item field is non-nullable. Importing such a column, for example an embedding column with a missing vector, failed with "Cannot convert an Arrow array containing nulls into a non-nullable Vortex array". Strip those nulls as the struct import already does. A null item in a valid list is still an error. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
robert3005
approved these changes
Nov 16, 2026
This was referenced Nov 16, 2026
connortsui20
added a commit
that referenced
this pull request
Nov 19, 2026
- Tracking Issue: #11114 Vortex declares the nullability of every nested level, and a non-nullable Vortex array has no validity at all. Arrow is looser: a level whose field is non-nullable can still carry null bits, for example under a null parent row, and the declared storage of an extension type can be stricter than the Arrow fields it imports from. So the import has to put a real value in each of those slots: ```text Arrow Vortex struct (nullable) [ valid, null ] struct (nullable) [ valid, null ] └── x (not null) [ 1, null ] └── x (not null) [ 1, 0 ] ``` #11223 and #11497 handled this by passing a mask of hidden slots down the tree, dropping a null that a null ancestor hides and failing on any other. The mask only went through structs, lists, and fixed-size lists, so the same `x` below a list view, a map, a run-end array, or a dictionary failed to import, and the `vortex_arrow::convert` functions never used the mask at all. #11602 extended it to the other levels, which showed how much the approach costs: list views can overlap, a run-end value covers many rows, and many dictionary keys share one value, so each level needs its own rule for mapping rows to children. This drops the mask, and each level now only looks at its own declared nullability. A non-nullable leaf fills its nulls with the zero of its storage type, and a non-nullable container drops its validity and keeps its contents, so its children follow their own fields. Both import paths share this rule. Importers still get the raw Arrow array. Most go through `from_arrow_storage`, which applies the rule with the nullability of the storage dtype, and the UUID and Parquet Variant importers, which convert validity themselves, now use the new public `level_validity`. A non-nullable dictionary points its null keys at entry zero, and a dictionary with keys but no values (as `new_null_array` builds) gets one entry by importing a single null row of the value type under the same rule. Behavior changes: - A null in a valid row of a non-nullable level now imports as zero instead of failing, so a null element in a valid `Vector` row becomes `0.0`. - Inside extension storage, the fill uses the storage zero without asking the extension, so a non-nullable JSON field under a null row holds `""`. - `vortex_arrow::convert` now imports run-end arrays, since it shares the run-end import with the session. Replaces #11602. AI assisted with this change. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An Arrow fixed-size list column whose item field is non-nullable cannot be imported once it has a null row, which is what an embedding column with a missing vector looks like. Arrow leaves the items of a null list undefined, and pyarrow marks them null:
vx.io.writeand the Ray datasink fail the same way.from_arrow_structalready strips such nulls.Changes
Both fixed-size list importers (
from_arrow_fixed_size_listandArrowSession) now strip the nulls under null lists. Arrays that fail Arrow's own validation, such as a null item in a valid list, are left alone and still error.Tests:
vortex-arrow/tests/fixed_size_list.rsruns both importers (flat, sliced, nested, and the rejected cases), plus a Python round trip throughvx.io.write.