Repository navigation
Push down array_transform in DataFusion - #11031
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #10854. DataFusion evaluates
array_transformabove the Vortex scan, and a projection containing a lambda skips projection pushdown entirely (#10215).This converts
array_transformto Vortex'slist_transform.Changes
array_transform(list, (x) -> body)converts tolist_transformwhen the lambda has one parameter.$is the struct of its parameters, as Vortex runs a body:xreads$.x, which becomes@xonce the body is converted. In that scope a column or an enclosing lambda's parameter is unsupported, and outside it so is any lambda parameter.array_transformconverts like any other, so DataFusion drops itsFilterExecand the scan evaluates it.