Repository navigation
Push partitioned hash join filters into V2 scans - #10415
Draft
joseph-isaacs wants to merge 29 commits into
Draft
joseph-isaacs wants to merge 29 commits into
joseph-isaacs wants to merge 29 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#11215 pushes DataFusion's dynamic filters into V2 scans, but a hash join with partitioned build sides still pushes nothing down. Its filter routes each probe key to its partition's filter by hash:
Vortex can't evaluate
hash_repartition, so the converter dropped the whole filter and the scan read every probe row.Every row the join keeps satisfies one partition's filter, so a bound or key list that every partition has still holds for that row once it is loosened to cover all of them. The scan now converts that hull:
The example above becomes
key >= 1 AND key <= 20. Partitions without rows (THEN false) don't constrain the hull. A partition with no bound means nothing is pushed down.The join still enforces its keys exactly, so a looser filter only means the scan returns more rows than it needs to.
This PR previously added dynamic filter support to the V1 scan.
developnow scans with V2 by default and #11215 covers that path, so this branch mergesdevelopkeeping its tree, and carries only the partitioned join change.Testing
vortex-datafusion/src/v2/dynamic_filters.rscover:cargo test -p vortex-datafusion --lib v2::dynamic_filterspasses.cargo clippy -p vortex-datafusion --all-targets --all-features -- -D warningspasses.cargo +nightly-2026-09-10 fmt -p vortex-datafusionis clean.https://claude.ai/code/session_01So4kxsqCY8WFG1NTMNJxKX
Generated by Claude Code