Skip to content

fix(arrow): import fixed-size lists whose null rows hold null items - #11497

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:fix/arrow-fixed-size-list-null-rows
Nov 16, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:fix/arrow-fixed-size-list-null-rows

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

An Arrow fixed-size list column whose item field is non-nullable cannot be imported once it has a null row, which is what an embedding column with a missing vector looks like. Arrow leaves the items of a null list undefined, and pyarrow marks them null:

item = pa.field("item", pa.float32(), nullable=False)
vx.array(pa.array([[1.0, 2.0], None, [5.0, 6.0]], pa.list_(item, 2)))
# RuntimeError: Cannot convert an Arrow array containing nulls into a non-nullable Vortex array

vx.io.write and the Ray datasink fail the same way. from_arrow_struct already strips such nulls.

Changes

Both fixed-size list importers (from_arrow_fixed_size_list and ArrowSession) now strip the nulls under null lists. Arrays that fail Arrow's own validation, such as a null item in a valid list, are left alone and still error.

Tests: vortex-arrow/tests/fixed_size_list.rs runs both importers (flat, sliced, nested, and the rejected cases), plus a Python round trip through vx.io.write.

Arrow leaves the items of a null list undefined, so writers such as pyarrow
mark them null even when the item field is non-nullable. Importing such a
column, for example an embedding column with a missing vector, failed with
"Cannot convert an Arrow array containing nulls into a non-nullable Vortex
array". Strip those nulls as the struct import already does. A null item in
a valid list is still an error.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 16, 2026
@robert3005
robert3005 merged commit 139294b into vortex-data:develop Nov 16, 2026
103 checks passed
connortsui20 added a commit that referenced this pull request Nov 19, 2026
- Tracking Issue: #11114

Vortex declares the nullability of every nested level, and a
non-nullable Vortex array has no validity at all. Arrow is looser: a
level whose field is non-nullable can still carry null bits, for example
under a null parent row, and the declared storage of an extension type
can be stricter than the Arrow fields it imports from. So the import has
to put a real value in each of those slots:

```text
Arrow                                 Vortex
struct (nullable)  [ valid, null ]    struct (nullable)  [ valid, null ]
└── x (not null)   [ 1,     null ]    └── x (not null)   [ 1,     0    ]
```

#11223 and #11497 handled this by passing a mask of hidden slots down
the tree, dropping a null that a null ancestor hides and failing on any
other. The mask only went through structs, lists, and fixed-size lists,
so the same `x` below a list view, a map, a run-end array, or a
dictionary failed to import, and the `vortex_arrow::convert` functions
never used the mask at all. #11602 extended it to the other levels,
which showed how much the approach costs: list views can overlap, a
run-end value covers many rows, and many dictionary keys share one
value, so each level needs its own rule for mapping rows to children.

This drops the mask, and each level now only looks at its own declared
nullability. A non-nullable leaf fills its nulls with the zero of its
storage type, and a non-nullable container drops its validity and keeps
its contents, so its children follow their own fields. Both import paths
share this rule.

Importers still get the raw Arrow array. Most go through
`from_arrow_storage`, which applies the rule with the nullability of the
storage dtype, and the UUID and Parquet Variant importers, which convert
validity themselves, now use the new public `level_validity`. A
non-nullable dictionary points its null keys at entry zero, and a
dictionary with keys but no values (as `new_null_array` builds) gets one
entry by importing a single null row of the value type under the same
rule.

Behavior changes:

- A null in a valid row of a non-nullable level now imports as zero
instead of failing, so a null element in a valid `Vector` row becomes
`0.0`.
- Inside extension storage, the fill uses the storage zero without
asking the extension, so a non-nullable JSON field under a null row
holds `""`.
- `vortex_arrow::convert` now imports run-end arrays, since it shares
the run-end import with the session.

Replaces #11602.

AI assisted with this change.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants