Skip to content

Decline DataFusion projections the Vortex scan does not evaluate - #11113

Closed
mhk197 wants to merge 2 commits into
developfrom
mk/datafusion-projection-pushdown
Closed

mhk197 wants to merge 2 commits into
developfrom
mk/datafusion-projection-pushdown

Conversation

@mhk197

@mhk197 mhk197 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Summary

VortexSource accepted every projection DataFusion offered, so EXPLAIN listed expressions on the DataSourceExec line even when DataFusion evaluated them inside the scan. For example, the cast in decimal_to_integer.slt hasn't been pushed down since #9785. The source now accepts a projection only when the scan evaluates all of it, and DataFusion keeps anything else in its own ProjectionExec.

Changes

  • try_pushdown_projection splits the projection against the table schema and declines it unless only plain columns are left for DataFusion.
  • The opener and the source share that split (process_projection), including the lambda guard.
  • Regenerated slt plans: decimal_to_integer, TPC-H Q1, TPC-DS Q5/Q23/Q62/Q99, ClickBench Q29. The slt runner keeps projection pushdown off.
before: DataSourceExec: ..., projection=[CAST(value@1 AS Int64) as integer_value]
after:  ProjectionExec: expr=[CAST(value@0 AS Int64) as integer_value]
          DataSourceExec: ..., projection=[value]

Testing

  • try_pushdown_projection_accepts_only_what_the_scan_evaluates covers columns, convertible, unconvertible and mixed projections, with pushdown on and off.
  • The DataFusion slt suite passes on regenerated TPC-H, TPC-DS and ClickBench data. CI currently skips the TPC-H slt because the generated files are now named lineitem_0001.vortex (since Split TPC-H and TPC-DS benchmark tables into ~200MB files #10519), so the Q1 plan was checked locally.

Follow-up

When the declined projection also prunes columns, the scan now returns a filter-only column (TPC-H Q1 returns l_shipdate). DataFusion could push just the referenced columns when a source declines a projection.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 closed this Nov 3, 2026
@robert3005
robert3005 deleted the mk/datafusion-projection-pushdown branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant