Repository navigation
Push down DataFusion field access on extension types - #11639
Open
connortsui20 wants to merge 1 commit into
Open
connortsui20 wants to merge 1 commit into
connortsui20 wants to merge 1 commit into
Conversation
connortsui20
marked this pull request as ready for review
November 20, 2026 17:16
connortsui20
force-pushed
the
ct/datafusion-extension-field-access
branch
from
November 20, 2026 21:24
4f25e2f to
c722cd8
Compare
connortsui20
force-pushed
the
ct/datafusion-extension-field-access
branch
from
November 20, 2026 21:30
c722cd8 to
7230254
Compare
connortsui20
commented
Nov 20, 2026
Comment on lines
+815
to
+835
| // Each step of a flattened path needs its own Arrow field metadata to identify the | ||
| // representation that DataFusion sees. | ||
| let source_field = source.return_field(schema)?; | ||
| let mut field = source_field.as_ref(); | ||
| let mut steps = Vec::with_capacity(paths.len()); | ||
| for path in paths { | ||
| let name = path | ||
| .downcast_ref::<df_expr::Literal>() | ||
| .and_then(|literal| literal.value().try_as_str().flatten()) | ||
| .ok_or_else(|| { | ||
| exec_datafusion_err!("get_field path must be a non-null string literal") | ||
| })?; | ||
| let DataType::Struct(fields) = field.data_type() else { | ||
| return Err(Unconverted::Unsupported); | ||
| }; | ||
| let (_, child_field) = fields | ||
| .find(name) | ||
| .ok_or_else(|| exec_datafusion_err!("get_field references missing field {name}"))?; | ||
| steps.push((field, name)); | ||
| field = child_field.as_ref(); | ||
| } |
Member
Author
There was a problem hiding this comment.
The change here is that we walk and get Field instead of DataType
connortsui20
force-pushed
the
ct/datafusion-extension-field-access
branch
from
November 20, 2026 21:38
7230254 to
6d7148b
Compare
connortsui20
commented
Nov 20, 2026
Comment on lines
+844
to
+862
| let mut read = self.convert_expr(source, schema, input_dtype)?; | ||
| let mut reads_extension_storage = false; | ||
| for (parent_field, name) in steps { | ||
| // Check the Vortex dtype instead of the Arrow extension name, because the session | ||
| // imports an unknown extension name as a plain struct that `get_item` reads directly. | ||
| if let DType::Extension(ext_dtype) = converted_dtype(&read, input_dtype)? { | ||
| if !self | ||
| .session | ||
| .arrow() | ||
| .exports_extension_storage(&ext_dtype, parent_field) | ||
| { | ||
| return Err(Unconverted::Unsupported); | ||
| } | ||
|
|
||
| read = ext_storage(read); | ||
| reads_extension_storage = true; | ||
| } | ||
| read = get_item(name, read); | ||
| } |
Member
Author
There was a problem hiding this comment.
And the extension check and ext_storage insertion here is new.
DataFusion can see a struct where Vortex sees an opaque extension value. Expose the storage explicitly when the Arrow field uses storage export, so the scan can project the requested field without bypassing exporters that transform values. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/datafusion-extension-field-access
branch
from
November 20, 2026 21:48
6d7148b to
115318b
Compare
connortsui20
enabled auto-merge (squash)
November 20, 2026 21:49
connortsui20
added a commit
that referenced
this pull request
Nov 20, 2026
The branch carried the prototype versions of the changes that the open PRs make, and those PRs have moved on in review. For example, #11640 now names the layout `vortex.extension`. This merge takes the tree of a stack on develop at 967bcb6, which holds the PR heads and ports of the two changes that have no PR yet: - #11640 at 24fd9da. - #11710 at add4115. - #11639 at 4f25e2f. Its newer head, 115318b, fails three of its own tests. - #11642 at d16038c. - Zone maps of bounds beneath nullable structs. - Single fields of list elements on both scans. A merge instead of a force-push keeps every commit that the branch published reachable. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
develop, DataFusion extracts fields from an extension column after Vortex exports the whole value to Arrow. This PR lets Vortex evaluate the field read in the scan when Arrow exposes the extension's storage directly.Suppose
measurements.pointis aPointextension with stored coordinates{x: 10, y: 11}. Extension types are opaque at the SQL level, so the query would use an accessor such asST_X.The accessor could lower to this DataFusion expression. A SQL frontend could offer
point['x']as syntax sugar for the same accessor.When Arrow exposes the storage directly, Vortex pushes down
get_item("x", ext_storage(get_item("point", root())))and returns10. If a plugin exporter is registered for the extension or for any part of the selected field, the field read stays in DataFusion, because the plugin decides which Arrow data DataFusion sees. The Arrow plugin contract does not yet require exports to keep values, and this check can go once it does.When
pointis null, the pushed-down read returns null. Ondevelop, DataFusion can return the child value stored under that null.This change was implemented and reviewed with OpenAI Codex.