Skip to content

Push down DataFusion field access on extension types - #11639

Open
connortsui20 wants to merge 1 commit into
developfrom
ct/datafusion-extension-field-access
Open

connortsui20 wants to merge 1 commit into
developfrom
ct/datafusion-extension-field-access

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

On develop, DataFusion extracts fields from an extension column after Vortex exports the whole value to Arrow. This PR lets Vortex evaluate the field read in the scan when Arrow exposes the extension's storage directly.

Suppose measurements.point is a Point extension with stored coordinates {x: 10, y: 11}. Extension types are opaque at the SQL level, so the query would use an accessor such as ST_X.

SELECT ST_X(point) FROM measurements;

The accessor could lower to this DataFusion expression. A SQL frontend could offer point['x'] as syntax sugar for the same accessor.

get_field(point, 'x')

When Arrow exposes the storage directly, Vortex pushes down get_item("x", ext_storage(get_item("point", root()))) and returns 10. If a plugin exporter is registered for the extension or for any part of the selected field, the field read stays in DataFusion, because the plugin decides which Arrow data DataFusion sees. The Arrow plugin contract does not yet require exports to keep values, and this check can go once it does.

When point is null, the pushed-down read returns null. On develop, DataFusion can return the child value stored under that null.

This change was implemented and reviewed with OpenAI Codex.

@connortsui20 connortsui20 added the changelog/performance A performance improvement label Nov 19, 2026 — with ChatGPT Codex Connector
@connortsui20 connortsui20 self-assigned this Nov 19, 2026
@connortsui20 connortsui20 changed the title Push DataFusion field reads through extension storage Push down DataFusion reads of extension storage fields Nov 19, 2026
@connortsui20 connortsui20 changed the title Push down DataFusion reads of extension storage fields Push down DataFusion field access on extension types Nov 19, 2026
@connortsui20
connortsui20 marked this pull request as ready for review November 20, 2026 17:16
@connortsui20
connortsui20 force-pushed the ct/datafusion-extension-field-access branch from 4f25e2f to c722cd8 Compare November 20, 2026 21:24
@connortsui20
connortsui20 force-pushed the ct/datafusion-extension-field-access branch from c722cd8 to 7230254 Compare November 20, 2026 21:30
Comment on lines +815 to +835
// Each step of a flattened path needs its own Arrow field metadata to identify the
// representation that DataFusion sees.
let source_field = source.return_field(schema)?;
let mut field = source_field.as_ref();
let mut steps = Vec::with_capacity(paths.len());
for path in paths {
let name = path
.downcast_ref::<df_expr::Literal>()
.and_then(|literal| literal.value().try_as_str().flatten())
.ok_or_else(|| {
exec_datafusion_err!("get_field path must be a non-null string literal")
})?;
let DataType::Struct(fields) = field.data_type() else {
return Err(Unconverted::Unsupported);
};
let (_, child_field) = fields
.find(name)
.ok_or_else(|| exec_datafusion_err!("get_field references missing field {name}"))?;
steps.push((field, name));
field = child_field.as_ref();
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change here is that we walk and get Field instead of DataType

@connortsui20
connortsui20 force-pushed the ct/datafusion-extension-field-access branch from 7230254 to 6d7148b Compare November 20, 2026 21:38
Comment on lines +844 to +862
let mut read = self.convert_expr(source, schema, input_dtype)?;
let mut reads_extension_storage = false;
for (parent_field, name) in steps {
// Check the Vortex dtype instead of the Arrow extension name, because the session
// imports an unknown extension name as a plain struct that `get_item` reads directly.
if let DType::Extension(ext_dtype) = converted_dtype(&read, input_dtype)? {
if !self
.session
.arrow()
.exports_extension_storage(&ext_dtype, parent_field)
{
return Err(Unconverted::Unsupported);
}

read = ext_storage(read);
reads_extension_storage = true;
}
read = get_item(name, read);
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the extension check and ext_storage insertion here is new.

DataFusion can see a struct where Vortex sees an opaque extension value. Expose the storage explicitly when the Arrow field uses storage export, so the scan can project the requested field without bypassing exporters that transform values.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/datafusion-extension-field-access branch from 6d7148b to 115318b Compare November 20, 2026 21:48
@connortsui20
connortsui20 enabled auto-merge (squash) November 20, 2026 21:49
connortsui20 added a commit that referenced this pull request Nov 20, 2026
The branch carried the prototype versions of the changes that the open
PRs make, and those PRs have moved on in review. For example, #11640
now names the layout `vortex.extension`. This merge takes the tree of a
stack on develop at 967bcb6, which holds the PR heads and ports of
the two changes that have no PR yet:

- #11640 at 24fd9da.
- #11710 at add4115.
- #11639 at 4f25e2f. Its newer head, 115318b, fails three of its
  own tests.
- #11642 at d16038c.
- Zone maps of bounds beneath nullable structs.
- Single fields of list elements on both scans.

A merge instead of a force-push keeps every commit that the branch
published reachable.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant