Skip to content

Drop masked nulls below every nested level in Arrow imports - #11602

Closed
connortsui20 wants to merge 1 commit into
developfrom
ct/arrow-import-masked-nulls
Closed

connortsui20 wants to merge 1 commit into
developfrom
ct/arrow-import-masked-nulls

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

An Arrow import fails when a non-nullable level holds a null that sits only under a null parent row, if that null is below a list view, a map, a run-end array, or a dictionary. No row reads such a null, so the import should drop it, as #11223 and #11497 already do below structs, lists, and fixed-size lists. These four levels instead treated every child slot as visible. A dictionary built by new_null_array failed too, because its null keys point at values that don't exist.

A dropped null key that points outside the values reads the first value instead, since Arrow ignores what's under a null key. When the dictionary has no values at all, the import adds one default value for those keys to read. Failing the import there would keep rejecting the arrays that new_null_array builds.

This only covers the canonical import of ArrowSession. The Arrow rebuild that runs before an extension importer and the public functions in vortex_arrow::convert still treat every slot below these levels as visible, so the same input can import through one path and fail through the other. Both are left for follow-ups, and the module docs of declared_nullability describe them.

AI assisted with this change.

@connortsui20 connortsui20 mentioned this pull request Nov 18, 2026
19 of 57 tasks
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/arrow-import-masked-nulls branch from ee9f0e3 to 5a8bada Compare November 18, 2026 20:59
@connortsui20
connortsui20 marked this pull request as ready for review November 18, 2026 21:08
@connortsui20

Copy link
Copy Markdown
Member Author

Replaced by #11617, which fills nulls in non-nullable levels with zero values instead of tracking which nulls a parent hides.

@connortsui20
connortsui20 deleted the ct/arrow-import-masked-nulls branch November 18, 2026 21:55
connortsui20 added a commit that referenced this pull request Nov 19, 2026
- Tracking Issue: #11114

Vortex declares the nullability of every nested level, and a
non-nullable Vortex array has no validity at all. Arrow is looser: a
level whose field is non-nullable can still carry null bits, for example
under a null parent row, and the declared storage of an extension type
can be stricter than the Arrow fields it imports from. So the import has
to put a real value in each of those slots:

```text
Arrow                                 Vortex
struct (nullable)  [ valid, null ]    struct (nullable)  [ valid, null ]
└── x (not null)   [ 1,     null ]    └── x (not null)   [ 1,     0    ]
```

#11223 and #11497 handled this by passing a mask of hidden slots down
the tree, dropping a null that a null ancestor hides and failing on any
other. The mask only went through structs, lists, and fixed-size lists,
so the same `x` below a list view, a map, a run-end array, or a
dictionary failed to import, and the `vortex_arrow::convert` functions
never used the mask at all. #11602 extended it to the other levels,
which showed how much the approach costs: list views can overlap, a
run-end value covers many rows, and many dictionary keys share one
value, so each level needs its own rule for mapping rows to children.

This drops the mask, and each level now only looks at its own declared
nullability. A non-nullable leaf fills its nulls with the zero of its
storage type, and a non-nullable container drops its validity and keeps
its contents, so its children follow their own fields. Both import paths
share this rule.

Importers still get the raw Arrow array. Most go through
`from_arrow_storage`, which applies the rule with the nullability of the
storage dtype, and the UUID and Parquet Variant importers, which convert
validity themselves, now use the new public `level_validity`. A
non-nullable dictionary points its null keys at entry zero, and a
dictionary with keys but no values (as `new_null_array` builds) gets one
entry by importing a single null row of the value type under the same
rule.

Behavior changes:

- A null in a valid row of a non-nullable level now imports as zero
instead of failing, so a null element in a valid `Vector` row becomes
`0.0`.
- Inside extension storage, the fill uses the storage zero without
asking the extension, so a non-nullable JSON field under a null row
holds `""`.
- `vortex_arrow::convert` now imports run-end arrays, since it shares
the run-end import with the session.

Replaces #11602.

AI assisted with this change.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant