Repository navigation
Make Arrow imports of known extension types strict - #11223
Merged
Merged
Conversation
19 of 57 tasks
connortsui20
added a commit
that referenced
this pull request
Nov 4, 2026
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
from
November 9, 2026 18:40
d09769e to
cf64b5c
Compare
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
2 times, most recently
from
November 16, 2026 19:09
1221e39 to
22629a2
Compare
connortsui20
changed the base branch from
develop
to
ct/arrow-roundtrip-properties
November 16, 2026 19:10
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
from
November 16, 2026 21:24
22629a2 to
64e9094
Compare
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
2 times, most recently
from
November 16, 2026 22:02
0931e95 to
9e73dbd
Compare
connortsui20
marked this pull request as ready for review
November 16, 2026 22:12
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
from
November 17, 2026 12:43
9e73dbd to
20f50c8
Compare
robert3005
approved these changes
Nov 17, 2026
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
from
November 17, 2026 15:00
20f50c8 to
fb6a729
Compare
connortsui20
enabled stacked auto-merge (squash)
November 17, 2026 15:12
connortsui20
disabled stacked auto-merge
November 17, 2026 15:27
ArrowSession::from_arrow_storage imports an array whose Arrow layout is the storage layout of an extension type. Producers such as pyarrow mark nested levels nullable even when they hold no nulls, or hold nulls only under null rows. The helper drops the nulls that no row reads, fails on a null in a valid row, and checks field names, order, and types first. Its errors name the extension type. The canonical import applies the same rule while it converts. Each struct and list level passes the visibility of its slots down to its children, and each level drops the nulls that a null row above it hides. An importer never sees the rows above its column, so the import rebuilds a known-type column under a null row in the form that its dtype declares before the importer runs. So a known type imports under a null row of a struct, list, or fixed-size list that hides nulls in the levels that the type declares non-nullable. This replaces the strip of non-nullable struct children, which dropped every null there without checking that a null row hid it. So a null in a valid row of a non-nullable struct field, or of a non-nullable level below one, now fails where it used to import as a placeholder value. arrow-rs checks an array only against its own type, so arrays that arrow-rs validated reach this failure too, when the field or schema of the import marks a level non-nullable that the type of the array marks nullable. A non-nullable dictionary of fixed-size list elements now drops the key nulls under a null list, as a dictionary struct field already did, so it fails in the same way when a key under them points outside its values, as in a dictionary of only nulls. The rule also replaces the strip of fixed-size list elements in the session import, which validated the whole array before it dropped the nulls under null lists. A null element of a valid list still fails, and the error now names the level and the index of the null. from_arrow_fixed_size_list keeps the strip. The ArrowImportVTable and ArrowExtVTable docs now state when an importer may defer: only for an input that another importer under the same name owns. Importer errors for a field now carry the field name. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The Vector importer used to decline a field that was not a FixedSizeList of floats with non-nullable elements, so the column imported as plain storage. It now fails on a field that cannot store vectors, and accepts nullable elements, such as the ones pyarrow writes. The array import drops the nulls under null vectors and fails on a null element of a valid vector. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The arrow.uuid and arrow.json importers declined a field that broke the canonical spec of its type, so the column imported as plain storage. They now fail on such a field and build their arrays with try_new instead of panicking on a storage mismatch. The arrow.uuid importer uses the check that arrow-rs implements, which matches the spec. The arrow.json check of arrow-rs is stricter than the spec, which reserves the fields of the metadata object for later additions, and Arrow C++ reads a missing metadata key as empty metadata. So the arrow.json importer accepts a missing key, an empty string, and any JSON object, and it still checks the storage type. The arrow.json exporter now checks its target field by the same rule, so a JSON column exports back to the field that it imports from. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
When GeoArrow rejected a field, for example a list view layout or metadata that is not JSON, the spatial importers fell back to the extension name alone. The fallback guessed the dimension from the field names and dropped the CRS, and the array import then declined the same field, so the column became plain storage under an extension dtype. A field that names a GeoArrow type now has to parse as that type, and the WKB importer fails instead of declining. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The spatial importers declined the LargeList layout, which GeoArrow accepts, and kept the nullability that Arrow gives coordinates, so a column with nullable-marked coordinates failed to match its own extension dtype. They now import through ArrowSession::from_arrow_storage. Nullable-marked levels import as the non-nullable levels that GeoArrow declares, nulls under null geometries are dropped, and a null coordinate of a valid geometry fails with an error that names the type. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A literal whose field no importer accepts already stays in DataFusion. Now that importers fail instead of declining, an importer can accept a field and still reject its value, for example a vector literal with a null element. That failure used to fail the whole conversion. It now leaves the literal to DataFusion as well. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/arrow-import-strict
branch
from
November 17, 2026 15:27
fb6a729 to
3495750
Compare
connortsui20
enabled stacked auto-merge (squash)
November 17, 2026 15:28
connortsui20
added a commit
that referenced
this pull request
Nov 17, 2026
## Summary Builds on #11223. LibFuzzer can explore storage encodings and input combinations beyond the bounded property suites, helping catch extension metadata or value loss across Arrow transport. ## Changes The shared assertions can be used without Hegel's native backend because that backend conflicts with libFuzzer coverage flags. Valid JSON and WKB payloads keep malformed logical values from obscuring transport failures, and seed cases ensure every extension family is reachable. Defining a Cargo binary alone leaves it idle in CI, so the target also joins scheduled fuzzing, crash reporting, and corpus maintenance. Variant remains excluded pending #11359. Local validation used source checks, YAML lint, and ShellCheck. No builds or runtime tests were run locally. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This was referenced Nov 17, 2026
connortsui20
added a commit
that referenced
this pull request
Nov 19, 2026
- Tracking Issue: #11114 Vortex declares the nullability of every nested level, and a non-nullable Vortex array has no validity at all. Arrow is looser: a level whose field is non-nullable can still carry null bits, for example under a null parent row, and the declared storage of an extension type can be stricter than the Arrow fields it imports from. So the import has to put a real value in each of those slots: ```text Arrow Vortex struct (nullable) [ valid, null ] struct (nullable) [ valid, null ] └── x (not null) [ 1, null ] └── x (not null) [ 1, 0 ] ``` #11223 and #11497 handled this by passing a mask of hidden slots down the tree, dropping a null that a null ancestor hides and failing on any other. The mask only went through structs, lists, and fixed-size lists, so the same `x` below a list view, a map, a run-end array, or a dictionary failed to import, and the `vortex_arrow::convert` functions never used the mask at all. #11602 extended it to the other levels, which showed how much the approach costs: list views can overlap, a run-end value covers many rows, and many dictionary keys share one value, so each level needs its own rule for mapping rows to children. This drops the mask, and each level now only looks at its own declared nullability. A non-nullable leaf fills its nulls with the zero of its storage type, and a non-nullable container drops its validity and keeps its contents, so its children follow their own fields. Both import paths share this rule. Importers still get the raw Arrow array. Most go through `from_arrow_storage`, which applies the rule with the nullability of the storage dtype, and the UUID and Parquet Variant importers, which convert validity themselves, now use the new public `level_validity`. A non-nullable dictionary points its null keys at entry zero, and a dictionary with keys but no values (as `new_null_array` builds) gets one entry by importing a single null row of the value type under the same rule. Behavior changes: - A null in a valid row of a non-nullable level now imports as zero instead of failing, so a null element in a valid `Vector` row becomes `0.0`. - Inside extension storage, the fill uses the storage zero without asking the extension, so a non-nullable JSON field under a null row holds `""`. - `vortex_arrow::convert` now imports run-end arrays, since it shares the run-end import with the session. Replaces #11602. AI assisted with this change. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tracking Issue: #11114. Builds on #11361.
Known extension importers can silently fall back to plain storage when Arrow uses a different layout. For example, PyArrow marks list elements nullable, which can make a vector column lose its
Vectortype. Valid input needs normalization into the extension's declared storage, while invalid input needs an error.Changes
Arrow permits arbitrary child values under a null parent, so masked nulls must not cause a false rejection. Visible nulls still fail at non-nullable levels. The name-only spatial fallback cannot preserve dimensions and CRS, so spatial metadata must be read fully.
Visibility propagation currently covers structs and fixed-size lists. List views, maps, dictionaries, and run-end arrays remain follow-ups in #11114.
Source checks passed. No builds or runtime tests were run for the split.
API Changes
LargeListinput, import as their extension when the visible values are valid.ArrowSessionrejects visible nulls in non-nullable children, including when the supplied schema is stricter than the Arrow array.ArrowSession::from_arrow_storage.