Repository navigation
Drop masked nulls below every nested level in Arrow imports - #11602
Closed
connortsui20 wants to merge 1 commit into
Closed
connortsui20 wants to merge 1 commit into
connortsui20 wants to merge 1 commit into
Conversation
19 of 57 tasks
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/arrow-import-masked-nulls
branch
from
November 18, 2026 20:59
ee9f0e3 to
5a8bada
Compare
connortsui20
marked this pull request as ready for review
November 18, 2026 21:08
Member
Author
|
Replaced by #11617, which fills nulls in non-nullable levels with zero values instead of tracking which nulls a parent hides. |
connortsui20
added a commit
that referenced
this pull request
Nov 19, 2026
- Tracking Issue: #11114 Vortex declares the nullability of every nested level, and a non-nullable Vortex array has no validity at all. Arrow is looser: a level whose field is non-nullable can still carry null bits, for example under a null parent row, and the declared storage of an extension type can be stricter than the Arrow fields it imports from. So the import has to put a real value in each of those slots: ```text Arrow Vortex struct (nullable) [ valid, null ] struct (nullable) [ valid, null ] └── x (not null) [ 1, null ] └── x (not null) [ 1, 0 ] ``` #11223 and #11497 handled this by passing a mask of hidden slots down the tree, dropping a null that a null ancestor hides and failing on any other. The mask only went through structs, lists, and fixed-size lists, so the same `x` below a list view, a map, a run-end array, or a dictionary failed to import, and the `vortex_arrow::convert` functions never used the mask at all. #11602 extended it to the other levels, which showed how much the approach costs: list views can overlap, a run-end value covers many rows, and many dictionary keys share one value, so each level needs its own rule for mapping rows to children. This drops the mask, and each level now only looks at its own declared nullability. A non-nullable leaf fills its nulls with the zero of its storage type, and a non-nullable container drops its validity and keeps its contents, so its children follow their own fields. Both import paths share this rule. Importers still get the raw Arrow array. Most go through `from_arrow_storage`, which applies the rule with the nullability of the storage dtype, and the UUID and Parquet Variant importers, which convert validity themselves, now use the new public `level_validity`. A non-nullable dictionary points its null keys at entry zero, and a dictionary with keys but no values (as `new_null_array` builds) gets one entry by importing a single null row of the value type under the same rule. Behavior changes: - A null in a valid row of a non-nullable level now imports as zero instead of failing, so a null element in a valid `Vector` row becomes `0.0`. - Inside extension storage, the fill uses the storage zero without asking the extension, so a non-nullable JSON field under a null row holds `""`. - `vortex_arrow::convert` now imports run-end arrays, since it shares the run-end import with the session. Replaces #11602. AI assisted with this change. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An Arrow import fails when a non-nullable level holds a null that sits only under a null parent row, if that null is below a list view, a map, a run-end array, or a dictionary. No row reads such a null, so the import should drop it, as #11223 and #11497 already do below structs, lists, and fixed-size lists. These four levels instead treated every child slot as visible. A dictionary built by
new_null_arrayfailed too, because its null keys point at values that don't exist.A dropped null key that points outside the values reads the first value instead, since Arrow ignores what's under a null key. When the dictionary has no values at all, the import adds one default value for those keys to read. Failing the import there would keep rejecting the arrays that
new_null_arraybuilds.This only covers the canonical import of
ArrowSession. The Arrow rebuild that runs before an extension importer and the public functions invortex_arrow::convertstill treat every slot below these levels as visible, so the same input can import through one path and fail through the other. Both are left for follow-ups, and the module docs ofdeclared_nullabilitydescribe them.AI assisted with this change.