Skip to content

fix(datafusion): skip projection pushdown for projections with lambdas - #10215

Merged
joseph-isaacs merged 3 commits into
developfrom
ji/datafusion-lambda-projection
Oct 2, 2026
Merged

joseph-isaacs merged 3 commits into
developfrom
ji/datafusion-lambda-projection

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With projection pushdown enabled, a DataFusion projection that contains a lambda (array_transform / list_transform) either fails or reads the wrong column on Vortex files. Parquet is unaffected. Found while getting the statpopgen benchmark running on DataFusion (stacked on top: #10212).

DataFusion indexes lambda parameters past the end of the projection's
input columns, but split_projection reshapes the scan output (pushed
expressions, one column per unpushable projection) and
reassign_expr_columns only remaps Column indices. The lambda variable
then pointed at a duplicated list column, so statpopgen Q6 tried to
allocate ~137 GB at 1,000 rows, and a bare array_transform in the select
list failed to convert. Fall back to reading the referenced columns
unchanged when the projection contains a lambda.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ
Signed-off-by: Claude <noreply@anthropic.com>
claude added 2 commits October 2, 2026 15:29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ
Signed-off-by: Claude <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the changelog/fix A bug fix label Oct 2, 2026
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) October 2, 2026 15:38

@myrrc myrrc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joseph-isaacs
joseph-isaacs merged commit 4187c18 into develop Oct 2, 2026
110 of 111 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/datafusion-lambda-projection branch October 2, 2026 16:25
robert3005 added a commit that referenced this pull request Oct 11, 2026
Develop changed `opener.rs` after this branch removed it. Apply the same
changes to the morsel planner:

- Let `FilePruner` decide if pruning is useful (#9841).
- Share the physical Arrow schema of each file across partitions.
- Disable projection pushdown for projections with lambdas (#10215).
- Bind expressions before optimizing them (#10002).
- Explain why the limit applies after the filter (#8766).
- Use `ScanBuilder<ArrayRef>` and `RepeatedScan<ArrayRef>`, because
  develop keeps `ScanBuilder::map`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005 pushed a commit that referenced this pull request Nov 6, 2026
## Summary

Stacked on #10854. DataFusion evaluates `array_transform` above the
Vortex scan, and a projection containing a lambda skips projection
pushdown entirely (#10215).

This converts `array_transform` to Vortex's `list_transform`.

## Changes

- `array_transform(list, (x) -> body)` converts to `list_transform` when
the lambda has one parameter.
- The body is converted in the lambda's own scope, where `$` is the
struct of its parameters, as Vortex runs a body: `x` reads `$.x`, which
becomes `@x` once the body is converted. In that scope a column or an
enclosing lambda's parameter is unsupported, and outside it so is any
lambda parameter.
- Filters push down too: a conjunct with `array_transform` converts like
any other, so DataFusion drops its `FilterExec` and the scan evaluates
it.
- The #10215 guard that turned off projection pushdown for any lambda is
removed: the split is all-or-nothing, so a lambda left for DataFusion
always runs over the raw columns.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants