Skip to content

Spark: drop dynamic pruning filters that nest pruning filters - #11603

Open
robert3005 wants to merge 1 commit into
developfrom
rk/spark-nested-dynamic-pruning
Open

robert3005 wants to merge 1 commit into
developfrom
rk/spark-nested-dynamic-pruning

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Spark adds one dynamic pruning subquery per join that can prune a scan, and the subquery holds the plan of the other side of the join. When a common table expression joins with itself, each subquery copies the subqueries of that side, so the plan grows by a factor of three per level of joins. The new NestedDynamicPruning rule drops each pruning filter whose filtering plan holds pruning filters of its own, and the new VortexSparkSessionExtensions installs it. A pruning filter only skips rows that the join drops, so the result does not change.

This is the rule from #11096, taken out so that it can merge and release first. The rule is public so that the Iceberg integration can install it in its own session extensions after the release.

  • Manual testing: on Spark 4.0, the plan of the test query (five levels of self-joins on a partition column) holds 106 pruning filters without the session extensions and 16 with them.

🤖 Generated with Claude Code

https://claude.ai/code/session_018SWbEtEQLKKebxSCarAZrU

Spark adds one pruning subquery per join that can prune a scan, holding
the plan of the join's other side. When a common table expression joins
with itself, each subquery copies the subqueries of that side, and the
plan grows by a factor of three per level of joins.

Add the NestedDynamicPruning optimizer rule, which drops each pruning
filter whose filtering plan holds pruning filters of its own, and
VortexSparkSessionExtensions, which installs it. The rule is public so
that a different Spark integration that reads Vortex files can install
it in its own session extensions.

Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch was successfully deployed

1 active deployment
vercel-docs — 249bc0de Deployed Nov 18, 2026 by robert3005 via Publish preview / Deploy #10816
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant