Repository navigation
fix(datafusion): skip projection pushdown for projections with lambdas - #10215
Merged
Merged
Conversation
DataFusion indexes lambda parameters past the end of the projection's input columns, but split_projection reshapes the scan output (pushed expressions, one column per unpushable projection) and reassign_expr_columns only remaps Column indices. The lambda variable then pointed at a duplicated list column, so statpopgen Q6 tried to allocate ~137 GB at 1,000 rows, and a bare array_transform in the select list failed to convert. Fall back to reading the referenced columns unchanged when the projection contains a lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
enabled auto-merge (squash)
October 2, 2026 15:38
robert3005
added a commit
that referenced
this pull request
Oct 11, 2026
Develop changed `opener.rs` after this branch removed it. Apply the same changes to the morsel planner: - Let `FilePruner` decide if pruning is useful (#9841). - Share the physical Arrow schema of each file across partitions. - Disable projection pushdown for projections with lambdas (#10215). - Bind expressions before optimizing them (#10002). - Explain why the limit applies after the filter (#8766). - Use `ScanBuilder<ArrayRef>` and `RepeatedScan<ArrayRef>`, because develop keeps `ScanBuilder::map`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
pushed a commit
that referenced
this pull request
Nov 6, 2026
## Summary Stacked on #10854. DataFusion evaluates `array_transform` above the Vortex scan, and a projection containing a lambda skips projection pushdown entirely (#10215). This converts `array_transform` to Vortex's `list_transform`. ## Changes - `array_transform(list, (x) -> body)` converts to `list_transform` when the lambda has one parameter. - The body is converted in the lambda's own scope, where `$` is the struct of its parameters, as Vortex runs a body: `x` reads `$.x`, which becomes `@x` once the body is converted. In that scope a column or an enclosing lambda's parameter is unsupported, and outside it so is any lambda parameter. - Filters push down too: a conjunct with `array_transform` converts like any other, so DataFusion drops its `FilterExec` and the scan evaluates it. - The #10215 guard that turned off projection pushdown for any lambda is removed: the split is all-or-nothing, so a lambda left for DataFusion always runs over the raw columns. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With projection pushdown enabled, a DataFusion projection that contains a lambda (
array_transform/list_transform) either fails or reads the wrong column on Vortex files. Parquet is unaffected. Found while getting the statpopgen benchmark running on DataFusion (stacked on top: #10212).