Repository navigation
Conversation
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VortexSourceaccepted every projection DataFusion offered, soEXPLAINlisted expressions on theDataSourceExecline even when DataFusion evaluated them inside the scan. For example, the cast indecimal_to_integer.slthasn't been pushed down since #9785. The source now accepts a projection only when the scan evaluates all of it, and DataFusion keeps anything else in its ownProjectionExec.Changes
try_pushdown_projectionsplits the projection against the table schema and declines it unless only plain columns are left for DataFusion.process_projection), including the lambda guard.decimal_to_integer, TPC-H Q1, TPC-DS Q5/Q23/Q62/Q99, ClickBench Q29. The slt runner keeps projection pushdown off.Testing
try_pushdown_projection_accepts_only_what_the_scan_evaluatescovers columns, convertible, unconvertible and mixed projections, with pushdown on and off.lineitem_0001.vortex(since Split TPC-H and TPC-DS benchmark tables into ~200MB files #10519), so the Q1 plan was checked locally.Follow-up
When the declined projection also prunes columns, the scan now returns a filter-only column (TPC-H Q1 returns
l_shipdate). DataFusion could push just the referenced columns when a source declines a projection.