Repository navigation
Require matching extension types in the DataFusion physical schema - #11268
Merged
Merged
Conversation
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
from
November 5, 2026 16:43
49c0209 to
9d81df5
Compare
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
from
November 5, 2026 16:43
99ac17a to
70a18ed
Compare
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
from
November 5, 2026 17:33
70a18ed to
bb850b3
Compare
This was referenced Nov 5, 2026
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
2 times, most recently
from
November 5, 2026 18:47
1ed554d to
f65ff0d
Compare
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
from
November 5, 2026 20:45
bb850b3 to
1266632
Compare
connortsui20
added a commit
that referenced
this pull request
Nov 5, 2026
) ## Summary - Tracking Issue: #11114 Right now, an extension type without an Arrow plugin can't leave Vortex through Arrow at all. That covers `FixedShapeTensor`, types from other crates, and unknown types read from a file, so DataFusion can't even infer the schema of a file that holds one, and the Python, JNI, and FFI schema exports fail the same way. A plugin for every type doesn't scale, and an unknown type can't have one. The import has the same gap, since a field whose extension name is a Vortex ID imports as plain storage. So a type without a plugin now exports its storage plus Arrow extension metadata, with its Vortex ID in `ARROW:extension:name` and its metadata in `ARROW:extension:metadata`. When the session knows the Vortex ID, the import uses that metadata to rebuild the type, so the type survives a round trip through Arrow, as X24 requires of every boundary. ## Changes The metadata is base64, since Arrow field metadata must be UTF-8 and Vortex metadata is arbitrary bytes. An Arrow extension name that Vortex doesn't know still imports as plain storage (X29), unlike the closed #10651, which imported it as a foreign type. Under the opacity plan, a foreign type only supports structural operations, so a pyarrow column with an unknown extension would lose filters and comparisons, and a file that holds it would need `allow_unknown` to read. This has to land before U1, which makes a storage-like target an error. A target whose extension name is the array's own Vortex ID still gets the storage, as long as it has the same metadata. So a caller keeps a lossless way to export, e.g., a timestamp column as `Int64`. An export without a target field (`execute_arrow` with no target, `Datum`, `ArrowSession::to_arrow_datatype`, and the Python array exports) has no Arrow field to hold the extension metadata, so it fails for a top-level column of these types instead of returning storage that has lost its type. All of them build their field with `ArrowSession::default_arrow_export_field`, so the check lives in one place and adds no work to an export. A nested column works, because its child field keeps the metadata. The types with a plugin still lose their name at the top level, which #11286 tracks. Note that until #11268, a table that mixes one of these columns with plain storage labels the plain rows with the extension type, as it already does for `Json`. Only the `ArrowSession` that `vortex_arrow::initialize` registers knows the extension types of its session, so a session created lazily imports these types as plain storage. The tests in `vortex-arrow/tests/vortex_id.rs` pin the field format and the round trip at the top level, in a struct, in a list, and over a timestamp, and a DataFusion test copies a `FixedShapeTensor` column through `COPY`. ## API Changes - `ArrowSession::to_arrow_field` and `to_arrow_schema` export these types instead of failing. - The new `ArrowSession::default_arrow_export_field` returns the field that an export without a target uses, and fails for a top-level type without a plugin. - When no importer accepts a field, but the session knows its Vortex ID, the import rebuilds the type. It fails if the metadata or the storage doesn't fit the type. For example, an `Int64` field named `vortex.timestamp` now imports as a `Timestamp`, and a `vortex.tensor.vector` field that the `Vector` importer declines now fails instead of importing as plain storage. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
from
November 9, 2026 18:45
1266632 to
c6697d2
Compare
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
3 times, most recently
from
November 16, 2026 20:03
700b289 to
bc792e7
Compare
connortsui20
marked this pull request as ready for review
November 16, 2026 20:04
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
from
November 16, 2026 20:11
bc792e7 to
b4e6901
Compare
Merging this PR will degrade performance by 13.03%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
The physical schema copied each table field's extension metadata onto the
file's column without checking the file's dtype. So a plain table field read a
`Json` column as plain strings, and a table inferred from `Json` and plain files
labeled the plain strings as `Json`.
Now every table field, including nested ones, must name the extension type
that `ArrowSession` exports for the file's dtype, or the scan fails:
- A missing metadata entry equals an empty one.
- When both fields name an extension type, the table field also matches if
`ArrowSession` imports it as the file's dtype. So `arrow.json` with `{}`
matches a `vortex.json` column, and so does a field that names the type by
its Vortex ID.
- A dictionary field names the extension type of its values on itself, and its
value type gets the checks of any field.
- A run-end field names the extension type of its values on its values field,
where the Arrow import and export read it. So the values field gets the
checks of any field, and the run-end field must name none.
- The entries field of a map has no Vortex dtype, so it must name none.
- A `Null` column has no values to mislabel, so the extension type of its
table field is not checked. The values of a run-end or dictionary field over
it are not checked.
Errors name the path of the field, such as `col.entries.key`, and the file
opener adds the path of the file.
Other behavior changes:
- Run-end and dictionary values now get layout checks, also in plain columns.
For example, `REE(values: Utf8)` over `i32` now fails at the schema step.
- A dtype that cannot be exported fails at every level. For example, an
extension type stored in another one fails when both have an Arrow extension
name, since one Arrow field holds one name.
- `VortexDataSourceBuilder::build` now checks a caller-supplied schema against
the data source's dtype.
- A mismatch in any column of a file fails every query on that file, as a
layout mismatch already does. The opener still computes the physical schema
over every column of the file, and a separate issue will decide that policy.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/datafusion-ext-type-match
branch
from
November 16, 2026 20:17
b4e6901 to
e79e93a
Compare
joseph-isaacs
approved these changes
Nov 17, 2026
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A table inferred over
Jsonand plainUtf8files can label plain strings as JSON because schema construction copies table metadata without checking the file's dtype. Since #11225, the same code can also read aJsonorVectorcolumn as plain storage. Scans now check extension identity before copying that metadata.Changes
I would actually read this if you are reviewing because a lot of the existing logic is not super intuitive.
Schema calculation combines two inputs:
Utf8or dictionary encoding.Vortex dtypes don't retain every Arrow representation, so both inputs are needed to produce the reader's physical Arrow schema. DataFusion's expression adapter then handles differences between file and table schemas, such as casts and missing columns. Both schemas use Arrow fields.
DFSchemaadds logical planning information.The extension check fits before the existing layout reconciliation:
Field, because extension identity lives in its metadata, whichDataTypealone doesn't carry.ArrowSessionwhat the file's dtype exports as, so the check follows the session's conversion rules instead of hardcoding each extension type.The same check applies recursively inside structs, lists, and maps. Dictionary values carry their extension metadata on the dictionary field because they have no field of their own. Run-end values have a separate values field, so their metadata belongs there. Those values also receive layout checks, so an encoding cannot hide an incompatible type.
Nullis exempt because it has no non-null values to mislabel. This lets an all-null file participate in a JSON table. Tests cover these rules, scans over JSON mixed with plain or null files, and caller-supplied schemas.The opener still checks every file column, so an unused column's mismatch can fail the query. #11270 tracks that policy. Preserving extension identity during scans does not define which DataFusion operations are valid for the extension.
API Changes
col.entries.key.VortexDataSourceBuilder::buildvalidates caller-supplied schemas before projection can hide an incompatible column.