Skip to content

Push down array_transform in DataFusion - #11031

Merged
robert3005 merged 5 commits into
developfrom
mk/datafusion-array-transform-pr
Nov 6, 2026
Merged

robert3005 merged 5 commits into
developfrom
mk/datafusion-array-transform-pr

Conversation

@mhk197

@mhk197 mhk197 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #10854. DataFusion evaluates array_transform above the Vortex scan, and a projection containing a lambda skips projection pushdown entirely (#10215).

This converts array_transform to Vortex's list_transform.

Changes

  • array_transform(list, (x) -> body) converts to list_transform when the lambda has one parameter.
  • The body is converted in the lambda's own scope, where $ is the struct of its parameters, as Vortex runs a body: x reads $.x, which becomes @x once the body is converted. In that scope a column or an enclosing lambda's parameter is unsupported, and outside it so is any lambda parameter.
  • Filters push down too: a conjunct with array_transform converts like any other, so DataFusion drops its FilterExec and the scan evaluates it.
  • The fix(datafusion): skip projection pushdown for projections with lambdas #10215 guard that turned off projection pushdown for any lambda is removed: the split is all-or-nothing, so a lambda left for DataFusion always runs over the raw columns.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants