Repository navigation
Fill nulls in non-nullable Arrow levels with zero values - #11617
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
b834ab4 to
e7c1547
Compare
Merging this PR will improve performance by 24.13%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
A vector literal with a null element now imports as zero, so it no longer exercises an importer that accepts a field but rejects its value. A test-only importer covers that path instead. Claude-Session: https://claude.ai/code/session_01J3PQtDd2heT7TLNnK71bH5 Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Vortex declares the nullability of every nested level, and a non-nullable Vortex array has no validity at all. Arrow is looser: a level whose field is non-nullable can still carry null bits, for example under a null parent row, and the declared storage of an extension type can be stricter than the Arrow fields it imports from. So the import has to put a real value in each of those slots:
#11223 and #11497 handled this by passing a mask of hidden slots down the tree, dropping a null that a null ancestor hides and failing on any other. The mask only went through structs, lists, and fixed-size lists, so the same
xbelow a list view, a map, a run-end array, or a dictionary failed to import, and thevortex_arrow::convertfunctions never used the mask at all. #11602 extended it to the other levels, which showed how much the approach costs: list views can overlap, a run-end value covers many rows, and many dictionary keys share one value, so each level needs its own rule for mapping rows to children.This drops the mask, and each level now only looks at its own declared nullability. A non-nullable leaf fills its nulls with the zero of its storage type, and a non-nullable container drops its validity and keeps its contents, so its children follow their own fields. Both import paths share this rule.
Importers still get the raw Arrow array. Most go through
from_arrow_storage, which applies the rule with the nullability of the storage dtype, and the UUID and Parquet Variant importers, which convert validity themselves, now use the new publiclevel_validity. A non-nullable dictionary points its null keys at entry zero, and a dictionary with keys but no values (asnew_null_arraybuilds) gets one entry by importing a single null row of the value type under the same rule.Behavior changes:
Vectorrow becomes0.0."".vortex_arrow::convertnow imports run-end arrays, since it shares the run-end import with the session.Replaces #11602.
AI assisted with this change.