Skip to content

Fill nulls in non-nullable Arrow levels with zero values - #11617

Merged
connortsui20 merged 2 commits into
developfrom
ct/arrow-import-fill-nulls
Nov 19, 2026
Merged

connortsui20 merged 2 commits into
developfrom
ct/arrow-import-fill-nulls

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Vortex declares the nullability of every nested level, and a non-nullable Vortex array has no validity at all. Arrow is looser: a level whose field is non-nullable can still carry null bits, for example under a null parent row, and the declared storage of an extension type can be stricter than the Arrow fields it imports from. So the import has to put a real value in each of those slots:

Arrow                                 Vortex
struct (nullable)  [ valid, null ]    struct (nullable)  [ valid, null ]
└── x (not null)   [ 1,     null ]    └── x (not null)   [ 1,     0    ]

#11223 and #11497 handled this by passing a mask of hidden slots down the tree, dropping a null that a null ancestor hides and failing on any other. The mask only went through structs, lists, and fixed-size lists, so the same x below a list view, a map, a run-end array, or a dictionary failed to import, and the vortex_arrow::convert functions never used the mask at all. #11602 extended it to the other levels, which showed how much the approach costs: list views can overlap, a run-end value covers many rows, and many dictionary keys share one value, so each level needs its own rule for mapping rows to children.

This drops the mask, and each level now only looks at its own declared nullability. A non-nullable leaf fills its nulls with the zero of its storage type, and a non-nullable container drops its validity and keeps its contents, so its children follow their own fields. Both import paths share this rule.

Importers still get the raw Arrow array. Most go through from_arrow_storage, which applies the rule with the nullability of the storage dtype, and the UUID and Parquet Variant importers, which convert validity themselves, now use the new public level_validity. A non-nullable dictionary points its null keys at entry zero, and a dictionary with keys but no values (as new_null_array builds) gets one entry by importing a single null row of the value type under the same rule.

Behavior changes:

  • A null in a valid row of a non-nullable level now imports as zero instead of failing, so a null element in a valid Vector row becomes 0.0.
  • Inside extension storage, the fill uses the storage zero without asking the extension, so a non-nullable JSON field under a null row holds "".
  • vortex_arrow::convert now imports run-end arrays, since it shares the run-end import with the session.

Replaces #11602.

AI assisted with this change.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/arrow-import-fill-nulls branch from b834ab4 to e7c1547 Compare November 18, 2026 22:10
@connortsui20
connortsui20 marked this pull request as ready for review November 18, 2026 22:12
@codspeed

codspeed Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 24.13%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 2 improved benchmarks
✅ 1973 untouched benchmarks
⏩ 53 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime compare_u8_neon 2.9 µs 2.2 µs +34.53%
⚡ Walltime compare_int_constant_neon 4.3 µs 3.8 µs +14.53%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/arrow-import-fill-nulls (fe6dedc) with develop (22b7d97)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

A vector literal with a null element now imports as zero, so it no
longer exercises an importer that accepts a field but rejects its value.
A test-only importer covers that path instead.

Claude-Session: https://claude.ai/code/session_01J3PQtDd2heT7TLNnK71bH5
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 enabled auto-merge (squash) November 19, 2026 02:09
@connortsui20
connortsui20 merged commit dc2ac56 into develop Nov 19, 2026
74 checks passed
@connortsui20
connortsui20 deleted the ct/arrow-import-fill-nulls branch November 19, 2026 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants