Repository navigation
Scan extension columns over list, map, and string storage in DataFusion - #11225
Merged
Merged
Conversation
19 of 57 tasks
connortsui20
added a commit
that referenced
this pull request
Nov 4, 2026
connortsui20
marked this pull request as ready for review
November 5, 2026 15:47
connortsui20
commented
Nov 5, 2026
mhk197
approved these changes
Nov 5, 2026
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
from
November 5, 2026 16:24
4060fe0 to
055c280
Compare
Member
Author
|
whoops claude pushed a bunch of unnecessary changes |
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
from
November 5, 2026 16:37
055c280 to
49c0209
Compare
This was referenced Nov 5, 2026
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
from
November 5, 2026 16:43
49c0209 to
9d81df5
Compare
connortsui20
enabled auto-merge (squash)
November 5, 2026 16:44
robert3005
approved these changes
Nov 5, 2026
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
from
November 5, 2026 17:49
9d81df5 to
1ed554d
Compare
`to_arrow_map` executed every input to a `MapArray`, and `Canonical::into_map` panics when the dtype is not a map, for example for an extension type over map storage. The list, fixed-size list, and list view exporters already check the dtype first, so the map exporter now does the same. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The layout arms of `calculate_physical_field_type` matched the extension dtype itself instead of its storage, so they rejected every extension column whose logical type was a `List`, `FixedSizeList`, `Map`, `ListView`, `Utf8`, or `Binary` layout. A `Vector` column failed the scan before it read a row. The struct arm already looked through extension types, so that walk now runs before the match and every layout arm uses it. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/datafusion-ext-physical-schema
branch
from
November 5, 2026 18:47
1ed554d to
f65ff0d
Compare
connortsui20
added a commit
that referenced
this pull request
Nov 17, 2026
…11268) ## Summary - Tracking Issue: #11114 A table inferred over `Json` and plain `Utf8` files can label plain strings as JSON because schema construction copies table metadata without checking the file's dtype. Since #11225, the same code can also read a `Json` or `Vector` column as plain storage. Scans now check extension identity before copying that metadata. ## Changes _I would actually read this if you are reviewing because a lot of the existing logic is not super intuitive._ Schema calculation combines two inputs: - The file's Vortex dtype says what values it contains. - The table's Arrow schema says what representation DataFusion expects, such as `Utf8` or dictionary encoding. Vortex dtypes don't retain every Arrow representation, so both inputs are needed to produce the reader's physical Arrow schema. DataFusion's [expression adapter](https://docs.rs/datafusion-physical-expr-adapter/55.2.0/datafusion_physical_expr_adapter/schema_rewriter/trait.PhysicalExprAdapter.html) then handles differences between file and table schemas, such as casts and missing columns. Both schemas use Arrow fields. [`DFSchema`](https://datafusion.apache.org/library-user-guide/working-with-exprs.html#arrow-schema-and-datafusion-dfschema) adds logical planning information. The extension check fits before the existing layout reconciliation: 1. Read the complete table `Field`, because [extension identity lives in its metadata](https://arrow.apache.org/docs/format/Columnar.html#extension-types), which `DataType` alone doesn't carry. 2. Ask [`ArrowSession`](https://github.com/vortex-data/vortex/blob/e79e93a5f2e93b409f87f922bbab1e9b53873754/vortex-arrow/src/session.rs#L267-L306) what the file's dtype exports as, so the check follows the session's conversion rules instead of hardcoding each extension type. 3. Compare extension names and metadata. If both fields name extensions, different spellings also pass when importing the table field reconstructs the same dtype. Otherwise, fail the scan. 4. Once identity agrees, inspect extension storage to choose a supported Arrow layout and copy the table metadata. The same check applies recursively inside structs, lists, and maps. Dictionary values carry their extension metadata on the dictionary field because they have no field of their own. Run-end values have a separate values field, so their metadata belongs there. Those values also receive layout checks, so an encoding cannot hide an incompatible type. `Null` is exempt because it has no non-null values to mislabel. This lets an all-null file participate in a JSON table. Tests cover these rules, scans over JSON mixed with plain or null files, and caller-supplied schemas. The opener still checks every file column, so an unused column's mismatch can fail the query. #11270 tracks that policy. Preserving extension identity during scans does not define which DataFusion operations are valid for the extension. ## API Changes - Scans reject extension identity mismatches and unexportable dtypes, including nested fields and encoded values. - File scans report schema errors with the file and nested field path, such as `col.entries.key`. - `VortexDataSourceBuilder::build` validates caller-supplied schemas before projection can hide an incompatible column. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DataFusion can't scan a
Vectorcolumn today. Arrow carries an extension column as its storage plus field metadata, but the physical schema compares the table's Arrow layout with the extension dtype instead of its storage, so any extension column over list, map, or string storage fails before the scan reads a row. The same bug breaks aJsoncolumn that the table schema declares asUtf8.Changes
The layout checks now look through extension types, as the struct check already did. An extension column keeps the table's layout (e.g.,
Utf8rather thanUtf8ViewforJson) instead of the type thatArrowSessionwould export for it, because the V2 source has no schema adapter, so its batches must match the plan schema exactly.The Arrow map export returns an error instead of panicking on a dtype other than a map, as the list exporters already do, since an extension column over map storage can reach it now.
Note that this only helps types with an Arrow export plugin, such as
VectorandJson, until #11224 exports the other types as their storage. A table field without the extension name now reads aJsonorVectorcolumn as plain storage (against X24), and #11268 makes that fail, so the two PRs should merge together.Tests cover a SQL scan of a
Vectorcolumn, a V2 projection over aJsoncolumn, each layout check of the physical schema, and the map export of a non-map array.