Skip to content

Make Arrow imports of known extension types strict - #11223

Merged
connortsui20 merged 7 commits into
developfrom
ct/arrow-import-strict
Nov 17, 2026
Merged

connortsui20 merged 7 commits into
developfrom
ct/arrow-import-strict

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Tracking Issue: #11114. Builds on #11361.

Known extension importers can silently fall back to plain storage when Arrow uses a different layout. For example, PyArrow marks list elements nullable, which can make a vector column lose its Vector type. Valid input needs normalization into the extension's declared storage, while invalid input needs an error.

Changes

Arrow permits arbitrary child values under a null parent, so masked nulls must not cause a false rejection. Visible nulls still fail at non-nullable levels. The name-only spatial fallback cannot preserve dimensions and CRS, so spatial metadata must be read fully.

Visibility propagation currently covers structs and fixed-size lists. List views, maps, dictionaries, and run-end arrays remain follow-ups in #11114.

Source checks passed. No builds or runtime tests were run for the split.

API Changes

  • Known extension fields now fail on unsupported or malformed input instead of silently becoming storage arrays.
  • Nullable-marked vector elements and spatial coordinates, plus GeoArrow LargeList input, import as their extension when the visible values are valid.
  • ArrowSession rejects visible nulls in non-nullable children, including when the supplied schema is stricter than the Arrow array.
  • Adds ArrowSession::from_arrow_storage.

@connortsui20 connortsui20 added the changelog/break A breaking API change label Nov 4, 2026
@connortsui20 connortsui20 mentioned this pull request Nov 4, 2026
19 of 57 tasks
connortsui20 added a commit that referenced this pull request Nov 4, 2026
DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.
@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch from d09769e to cf64b5c Compare November 9, 2026 18:40
@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch 2 times, most recently from 1221e39 to 22629a2 Compare November 16, 2026 19:09
@connortsui20
connortsui20 changed the base branch from develop to ct/arrow-roundtrip-properties November 16, 2026 19:10
@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch from 22629a2 to 64e9094 Compare November 16, 2026 21:24
@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch 2 times, most recently from 0931e95 to 9e73dbd Compare November 16, 2026 22:02
@connortsui20
connortsui20 removed this pull request from stack #11376 November 16, 2026 22:02
@connortsui20
connortsui20 marked this pull request as ready for review November 16, 2026 22:12
@codspeed

codspeed Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1965 untouched benchmarks
⏩ 53 skipped benchmarks1


Comparing ct/arrow-import-strict (3495750) with develop (fdc113e)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch from 20f50c8 to fb6a729 Compare November 17, 2026 15:00
@connortsui20
connortsui20 enabled stacked auto-merge (squash) November 17, 2026 15:12
@connortsui20
connortsui20 disabled stacked auto-merge November 17, 2026 15:27
Base automatically changed from ct/arrow-roundtrip-properties to develop November 17, 2026 15:27
ArrowSession::from_arrow_storage imports an array whose Arrow layout is the
storage layout of an extension type. Producers such as pyarrow mark nested
levels nullable even when they hold no nulls, or hold nulls only under null
rows. The helper drops the nulls that no row reads, fails on a null in a
valid row, and checks field names, order, and types first. Its errors name
the extension type.

The canonical import applies the same rule while it converts. Each struct
and list level passes the visibility of its slots down to its children, and
each level drops the nulls that a null row above it hides. An importer
never sees the rows above its column, so the import rebuilds a known-type
column under a null row in the form that its dtype declares before the
importer runs. So a known type imports under a null row of a struct, list,
or fixed-size list that hides nulls in the levels that the type declares
non-nullable.

This replaces the strip of non-nullable struct children, which dropped
every null there without checking that a null row hid it. So a null in a
valid row of a non-nullable struct field, or of a non-nullable level below
one, now fails where it used to import as a placeholder value. arrow-rs
checks an array only against its own type, so arrays that arrow-rs
validated reach this failure too, when the field or schema of the import
marks a level non-nullable that the type of the array marks nullable. A
non-nullable dictionary of fixed-size list elements now drops the key nulls
under a null list, as a dictionary struct field already did, so it fails in
the same way when a key under them points outside its values, as in a
dictionary of only nulls.

The rule also replaces the strip of fixed-size list elements in the session
import, which validated the whole array before it dropped the nulls under
null lists. A null element of a valid list still fails, and the error now
names the level and the index of the null. from_arrow_fixed_size_list keeps
the strip.

The ArrowImportVTable and ArrowExtVTable docs now state when an importer may
defer: only for an input that another importer under the same name owns.
Importer errors for a field now carry the field name.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The Vector importer used to decline a field that was not a FixedSizeList of
floats with non-nullable elements, so the column imported as plain storage.
It now fails on a field that cannot store vectors, and accepts nullable
elements, such as the ones pyarrow writes. The array import drops the nulls
under null vectors and fails on a null element of a valid vector.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The arrow.uuid and arrow.json importers declined a field that broke the
canonical spec of its type, so the column imported as plain storage. They
now fail on such a field and build their arrays with try_new instead of
panicking on a storage mismatch.

The arrow.uuid importer uses the check that arrow-rs implements, which
matches the spec. The arrow.json check of arrow-rs is stricter than the
spec, which reserves the fields of the metadata object for later additions,
and Arrow C++ reads a missing metadata key as empty metadata. So the
arrow.json importer accepts a missing key, an empty string, and any JSON
object, and it still checks the storage type. The arrow.json exporter now
checks its target field by the same rule, so a JSON column exports back to
the field that it imports from.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
When GeoArrow rejected a field, for example a list view layout or metadata
that is not JSON, the spatial importers fell back to the extension name
alone. The fallback guessed the dimension from the field names and dropped
the CRS, and the array import then declined the same field, so the column
became plain storage under an extension dtype. A field that names a GeoArrow
type now has to parse as that type, and the WKB importer fails instead of
declining.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The spatial importers declined the LargeList layout, which GeoArrow
accepts, and kept the nullability that Arrow gives coordinates, so a column
with nullable-marked coordinates failed to match its own extension dtype.
They now import through ArrowSession::from_arrow_storage. Nullable-marked
levels import as the non-nullable levels that GeoArrow declares, nulls under
null geometries are dropped, and a null coordinate of a valid geometry fails
with an error that names the type.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A literal whose field no importer accepts already stays in DataFusion. Now
that importers fail instead of declining, an importer can accept a field
and still reject its value, for example a vector literal with a null
element. That failure used to fail the whole conversion. It now leaves the
literal to DataFusion as well.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/arrow-import-strict branch from fb6a729 to 3495750 Compare November 17, 2026 15:27
@connortsui20
connortsui20 enabled stacked auto-merge (squash) November 17, 2026 15:28
@connortsui20
connortsui20 merged commit 732d748 into develop Nov 17, 2026
100 of 101 checks passed
@connortsui20
connortsui20 deleted the ct/arrow-import-strict branch November 17, 2026 15:44
connortsui20 added a commit that referenced this pull request Nov 17, 2026
## Summary

Builds on #11223.

LibFuzzer can explore storage encodings and input combinations beyond
the bounded property suites, helping catch extension metadata or value
loss across Arrow transport.

## Changes

The shared assertions can be used without Hegel's native backend because
that backend conflicts with libFuzzer coverage flags. Valid JSON and WKB
payloads keep malformed logical values from obscuring transport
failures, and seed cases ensure every extension family is reachable.

Defining a Cargo binary alone leaves it idle in CI, so the target also
joins scheduled fuzzing, crash reporting, and corpus maintenance.

Variant remains excluded pending #11359.

Local validation used source checks, YAML lint, and ShellCheck. No
builds or runtime tests were run locally.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20 added a commit that referenced this pull request Nov 19, 2026
- Tracking Issue: #11114

Vortex declares the nullability of every nested level, and a
non-nullable Vortex array has no validity at all. Arrow is looser: a
level whose field is non-nullable can still carry null bits, for example
under a null parent row, and the declared storage of an extension type
can be stricter than the Arrow fields it imports from. So the import has
to put a real value in each of those slots:

```text
Arrow                                 Vortex
struct (nullable)  [ valid, null ]    struct (nullable)  [ valid, null ]
└── x (not null)   [ 1,     null ]    └── x (not null)   [ 1,     0    ]
```

#11223 and #11497 handled this by passing a mask of hidden slots down
the tree, dropping a null that a null ancestor hides and failing on any
other. The mask only went through structs, lists, and fixed-size lists,
so the same `x` below a list view, a map, a run-end array, or a
dictionary failed to import, and the `vortex_arrow::convert` functions
never used the mask at all. #11602 extended it to the other levels,
which showed how much the approach costs: list views can overlap, a
run-end value covers many rows, and many dictionary keys share one
value, so each level needs its own rule for mapping rows to children.

This drops the mask, and each level now only looks at its own declared
nullability. A non-nullable leaf fills its nulls with the zero of its
storage type, and a non-nullable container drops its validity and keeps
its contents, so its children follow their own fields. Both import paths
share this rule.

Importers still get the raw Arrow array. Most go through
`from_arrow_storage`, which applies the rule with the nullability of the
storage dtype, and the UUID and Parquet Variant importers, which convert
validity themselves, now use the new public `level_validity`. A
non-nullable dictionary points its null keys at entry zero, and a
dictionary with keys but no values (as `new_null_array` builds) gets one
entry by importing a single null row of the value type under the same
rule.

Behavior changes:

- A null in a valid row of a non-nullable level now imports as zero
instead of failing, so a null element in a valid `Vector` row becomes
`0.0`.
- Inside extension storage, the fill uses the storage zero without
asking the extension, so a non-nullable JSON field under a null row
holds `""`.
- `vortex_arrow::convert` now imports run-end arrays, since it shares
the run-end import with the session.

Replaces #11602.

AI assisted with this change.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants