Skip to content

Scan extension columns over list, map, and string storage in DataFusion - #11225

Merged
connortsui20 merged 2 commits into
developfrom
ct/datafusion-ext-physical-schema
Nov 5, 2026
Merged

connortsui20 merged 2 commits into
developfrom
ct/datafusion-ext-physical-schema

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

DataFusion can't scan a Vector column today. Arrow carries an extension column as its storage plus field metadata, but the physical schema compares the table's Arrow layout with the extension dtype instead of its storage, so any extension column over list, map, or string storage fails before the scan reads a row. The same bug breaks a Json column that the table schema declares as Utf8.

Changes

The layout checks now look through extension types, as the struct check already did. An extension column keeps the table's layout (e.g., Utf8 rather than Utf8View for Json) instead of the type that ArrowSession would export for it, because the V2 source has no schema adapter, so its batches must match the plan schema exactly.

The Arrow map export returns an error instead of panicking on a dtype other than a map, as the list exporters already do, since an extension column over map storage can reach it now.

Note that this only helps types with an Arrow export plugin, such as Vector and Json, until #11224 exports the other types as their storage. A table field without the extension name now reads a Json or Vector column as plain storage (against X24), and #11268 makes that fail, so the two PRs should merge together.

Tests cover a SQL scan of a Vector column, a V2 projection over a Json column, each layout check of the physical schema, and the map export of a non-map array.

@connortsui20 connortsui20 added the changelog/fix A bug fix label Nov 4, 2026
@connortsui20 connortsui20 mentioned this pull request Nov 4, 2026
19 of 57 tasks
connortsui20 added a commit that referenced this pull request Nov 4, 2026
DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.
@connortsui20
connortsui20 marked this pull request as ready for review November 5, 2026 15:47
Comment thread vortex-datafusion/src/convert/schema.rs Resolved
Comment thread vortex-datafusion/src/convert/schema.rs Resolved
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 4060fe0 to 055c280 Compare November 5, 2026 16:24
@connortsui20

Copy link
Copy Markdown
Member Author

whoops claude pushed a bunch of unnecessary changes

@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 9d81df5 to 1ed554d Compare November 5, 2026 17:49
`to_arrow_map` executed every input to a `MapArray`, and
`Canonical::into_map` panics when the dtype is not a map, for example
for an extension type over map storage. The list, fixed-size list, and
list view exporters already check the dtype first, so the map exporter
now does the same.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The layout arms of `calculate_physical_field_type` matched the extension
dtype itself instead of its storage, so they rejected every extension
column whose logical type was a `List`, `FixedSizeList`, `Map`,
`ListView`, `Utf8`, or `Binary` layout. A `Vector` column failed the scan
before it read a row. The struct arm already looked through extension
types, so that walk now runs before the match and every layout arm uses
it.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 1ed554d to f65ff0d Compare November 5, 2026 18:47
@connortsui20
connortsui20 merged commit 9748b74 into develop Nov 5, 2026
70 of 71 checks passed
@connortsui20
connortsui20 deleted the ct/datafusion-ext-physical-schema branch November 5, 2026 18:57
connortsui20 added a commit that referenced this pull request Nov 17, 2026
…11268)

## Summary

- Tracking Issue: #11114

A table inferred over `Json` and plain `Utf8` files can label plain
strings as JSON because schema construction copies table metadata
without checking the file's dtype. Since #11225, the same code can also
read a `Json` or `Vector` column as plain storage. Scans now check
extension identity before copying that metadata.

## Changes

_I would actually read this if you are reviewing because a lot of the
existing logic is not super intuitive._

Schema calculation combines two inputs:

- The file's Vortex dtype says what values it contains.
- The table's Arrow schema says what representation DataFusion expects,
such as `Utf8` or dictionary encoding.

Vortex dtypes don't retain every Arrow representation, so both inputs
are needed to produce the reader's physical Arrow schema. DataFusion's
[expression
adapter](https://docs.rs/datafusion-physical-expr-adapter/55.2.0/datafusion_physical_expr_adapter/schema_rewriter/trait.PhysicalExprAdapter.html)
then handles differences between file and table schemas, such as casts
and missing columns. Both schemas use Arrow fields.
[`DFSchema`](https://datafusion.apache.org/library-user-guide/working-with-exprs.html#arrow-schema-and-datafusion-dfschema)
adds logical planning information.

The extension check fits before the existing layout reconciliation:

1. Read the complete table `Field`, because [extension identity lives in
its
metadata](https://arrow.apache.org/docs/format/Columnar.html#extension-types),
which `DataType` alone doesn't carry.
2. Ask
[`ArrowSession`](https://github.com/vortex-data/vortex/blob/e79e93a5f2e93b409f87f922bbab1e9b53873754/vortex-arrow/src/session.rs#L267-L306)
what the file's dtype exports as, so the check follows the session's
conversion rules instead of hardcoding each extension type.
3. Compare extension names and metadata. If both fields name extensions,
different spellings also pass when importing the table field
reconstructs the same dtype. Otherwise, fail the scan.
4. Once identity agrees, inspect extension storage to choose a supported
Arrow layout and copy the table metadata.

The same check applies recursively inside structs, lists, and maps.
Dictionary values carry their extension metadata on the dictionary field
because they have no field of their own. Run-end values have a separate
values field, so their metadata belongs there. Those values also receive
layout checks, so an encoding cannot hide an incompatible type.

`Null` is exempt because it has no non-null values to mislabel. This
lets an all-null file participate in a JSON table. Tests cover these
rules, scans over JSON mixed with plain or null files, and
caller-supplied schemas.

The opener still checks every file column, so an unused column's
mismatch can fail the query. #11270 tracks that policy. Preserving
extension identity during scans does not define which DataFusion
operations are valid for the extension.

## API Changes

- Scans reject extension identity mismatches and unexportable dtypes,
including nested fields and encoded values.
- File scans report schema errors with the file and nested field path,
such as `col.entries.key`.
- `VortexDataSourceBuilder::build` validates caller-supplied schemas
before projection can hide an incompatible column.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants