Skip to content

Require matching extension types in the DataFusion physical schema - #11268

Merged
connortsui20 merged 1 commit into
developfrom
ct/datafusion-ext-type-match
Nov 17, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/datafusion-ext-type-match

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Summary

A table inferred over Json and plain Utf8 files can label plain strings as JSON because schema construction copies table metadata without checking the file's dtype. Since #11225, the same code can also read a Json or Vector column as plain storage. Scans now check extension identity before copying that metadata.

Changes

I would actually read this if you are reviewing because a lot of the existing logic is not super intuitive.

Schema calculation combines two inputs:

  • The file's Vortex dtype says what values it contains.
  • The table's Arrow schema says what representation DataFusion expects, such as Utf8 or dictionary encoding.

Vortex dtypes don't retain every Arrow representation, so both inputs are needed to produce the reader's physical Arrow schema. DataFusion's expression adapter then handles differences between file and table schemas, such as casts and missing columns. Both schemas use Arrow fields. DFSchema adds logical planning information.

The extension check fits before the existing layout reconciliation:

  1. Read the complete table Field, because extension identity lives in its metadata, which DataType alone doesn't carry.
  2. Ask ArrowSession what the file's dtype exports as, so the check follows the session's conversion rules instead of hardcoding each extension type.
  3. Compare extension names and metadata. If both fields name extensions, different spellings also pass when importing the table field reconstructs the same dtype. Otherwise, fail the scan.
  4. Once identity agrees, inspect extension storage to choose a supported Arrow layout and copy the table metadata.

The same check applies recursively inside structs, lists, and maps. Dictionary values carry their extension metadata on the dictionary field because they have no field of their own. Run-end values have a separate values field, so their metadata belongs there. Those values also receive layout checks, so an encoding cannot hide an incompatible type.

Null is exempt because it has no non-null values to mislabel. This lets an all-null file participate in a JSON table. Tests cover these rules, scans over JSON mixed with plain or null files, and caller-supplied schemas.

The opener still checks every file column, so an unused column's mismatch can fail the query. #11270 tracks that policy. Preserving extension identity during scans does not define which DataFusion operations are valid for the extension.

API Changes

  • Scans reject extension identity mismatches and unexportable dtypes, including nested fields and encoded values.
  • File scans report schema errors with the file and nested field path, such as col.entries.key.
  • VortexDataSourceBuilder::build validates caller-supplied schemas before projection can hide an incompatible column.

@connortsui20 connortsui20 added the changelog/fix A bug fix label Nov 5, 2026
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch from 49c0209 to 9d81df5 Compare November 5, 2026 16:43
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-type-match branch from 99ac17a to 70a18ed Compare November 5, 2026 16:43
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-type-match branch from 70a18ed to bb850b3 Compare November 5, 2026 17:33
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-physical-schema branch 2 times, most recently from 1ed554d to f65ff0d Compare November 5, 2026 18:47
Base automatically changed from ct/datafusion-ext-physical-schema to develop November 5, 2026 18:57
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-type-match branch from bb850b3 to 1266632 Compare November 5, 2026 20:45
connortsui20 added a commit that referenced this pull request Nov 5, 2026
)

## Summary

- Tracking Issue: #11114

Right now, an extension type without an Arrow plugin can't leave Vortex
through Arrow at all. That covers `FixedShapeTensor`, types from other
crates, and unknown types read from a file, so DataFusion can't even
infer the schema of a file that holds one, and the Python, JNI, and FFI
schema exports fail the same way. A plugin for every type doesn't scale,
and an unknown type can't have one. The import has the same gap, since a
field whose extension name is a Vortex ID imports as plain storage.

So a type without a plugin now exports its storage plus Arrow extension
metadata, with its Vortex ID in `ARROW:extension:name` and its metadata
in `ARROW:extension:metadata`. When the session knows the Vortex ID, the
import uses that metadata to rebuild the type, so the type survives a
round trip through Arrow, as X24 requires of every boundary.

## Changes

The metadata is base64, since Arrow field metadata must be UTF-8 and
Vortex metadata is arbitrary bytes.

An Arrow extension name that Vortex doesn't know still imports as plain
storage (X29), unlike the closed #10651, which imported it as a foreign
type. Under the opacity plan, a foreign type only supports structural
operations, so a pyarrow column with an unknown extension would lose
filters and comparisons, and a file that holds it would need
`allow_unknown` to read.

This has to land before U1, which makes a storage-like target an error.
A target whose extension name is the array's own Vortex ID still gets
the storage, as long as it has the same metadata. So a caller keeps a
lossless way to export, e.g., a timestamp column as `Int64`.

An export without a target field (`execute_arrow` with no target,
`Datum`, `ArrowSession::to_arrow_datatype`, and the Python array
exports) has no Arrow field to hold the extension metadata, so it fails
for a top-level column of these types instead of returning storage that
has lost its type. All of them build their field with
`ArrowSession::default_arrow_export_field`, so the check lives in one
place and adds no work to an export. A nested column works, because its
child field keeps the metadata. The types with a plugin still lose their
name at the top level, which #11286 tracks.

Note that until #11268, a table that mixes one of these columns with
plain storage labels the plain rows with the extension type, as it
already does for `Json`. Only the `ArrowSession` that
`vortex_arrow::initialize` registers knows the extension types of its
session, so a session created lazily imports these types as plain
storage. The tests in `vortex-arrow/tests/vortex_id.rs` pin the field
format and the round trip at the top level, in a struct, in a list, and
over a timestamp, and a DataFusion test copies a `FixedShapeTensor`
column through `COPY`.

## API Changes

- `ArrowSession::to_arrow_field` and `to_arrow_schema` export these
types instead of failing.
- The new `ArrowSession::default_arrow_export_field` returns the field
that an export without a target uses, and fails for a top-level type
without a plugin.
- When no importer accepts a field, but the session knows its Vortex ID,
the import rebuilds the type. It fails if the metadata or the storage
doesn't fit the type. For example, an `Int64` field named
`vortex.timestamp` now imports as a `Timestamp`, and a
`vortex.tensor.vector` field that the `Vector` importer declines now
fails instead of importing as plain storage.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-type-match branch from 1266632 to c6697d2 Compare November 9, 2026 18:45
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-type-match branch 3 times, most recently from 700b289 to bc792e7 Compare November 16, 2026 20:03
@connortsui20
connortsui20 marked this pull request as ready for review November 16, 2026 20:04
@connortsui20
connortsui20 force-pushed the ct/datafusion-ext-type-match branch from bc792e7 to b4e6901 Compare November 16, 2026 20:11
@codspeed

codspeed Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 13.03%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

❌ 1 regressed benchmark
✅ 1957 untouched benchmarks
⏩ 53 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime decode_chunked_neon[i64, 524288] 13.7 µs 15.7 µs -13.03%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/datafusion-ext-type-match (e79e93a) with develop (c58d623)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

The physical schema copied each table field's extension metadata onto the
file's column without checking the file's dtype. So a plain table field read a
`Json` column as plain strings, and a table inferred from `Json` and plain files
labeled the plain strings as `Json`.

Now every table field, including nested ones, must name the extension type
that `ArrowSession` exports for the file's dtype, or the scan fails:

- A missing metadata entry equals an empty one.
- When both fields name an extension type, the table field also matches if
  `ArrowSession` imports it as the file's dtype. So `arrow.json` with `{}`
  matches a `vortex.json` column, and so does a field that names the type by
  its Vortex ID.
- A dictionary field names the extension type of its values on itself, and its
  value type gets the checks of any field.
- A run-end field names the extension type of its values on its values field,
  where the Arrow import and export read it. So the values field gets the
  checks of any field, and the run-end field must name none.
- The entries field of a map has no Vortex dtype, so it must name none.
- A `Null` column has no values to mislabel, so the extension type of its
  table field is not checked. The values of a run-end or dictionary field over
  it are not checked.

Errors name the path of the field, such as `col.entries.key`, and the file
opener adds the path of the file.

Other behavior changes:

- Run-end and dictionary values now get layout checks, also in plain columns.
  For example, `REE(values: Utf8)` over `i32` now fails at the schema step.
- A dtype that cannot be exported fails at every level. For example, an
  extension type stored in another one fails when both have an Arrow extension
  name, since one Arrow field holds one name.
- `VortexDataSourceBuilder::build` now checks a caller-supplied schema against
  the data source's dtype.
- A mismatch in any column of a file fails every query on that file, as a
  layout mismatch already does. The opener still computes the physical schema
  over every column of the file, and a separate issue will decide that policy.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 merged commit 1283742 into develop Nov 17, 2026
87 of 88 checks passed
@connortsui20
connortsui20 deleted the ct/datafusion-ext-type-match branch November 17, 2026 12:44
@connortsui20 connortsui20 mentioned this pull request Nov 17, 2026
9 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants