Skip to content

Push partitioned hash join filters into V2 scans - #10415

Draft
joseph-isaacs wants to merge 29 commits into
developfrom
ji/nifty-wright-ahdfl0
Draft

joseph-isaacs wants to merge 29 commits into
developfrom
ji/nifty-wright-ahdfl0

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

#11215 pushes DataFusion's dynamic filters into V2 scans, but a hash join with partitioned build sides still pushes nothing down. Its filter routes each probe key to its partition's filter by hash:

CASE hash_repartition(keys) % n
  WHEN 0 THEN key >= 1 AND key <= 5 AND key IN (1, 3, 5)
  WHEN 1 THEN key >= 10 AND key <= 20 AND hash_lookup(...)
  ...
  ELSE false
END

Vortex can't evaluate hash_repartition, so the converter dropped the whole filter and the scan read every probe row.

Every row the join keeps satisfies one partition's filter, so a bound or key list that every partition has still holds for that row once it is loosened to cover all of them. The scan now converts that hull:

  • the loosest lower and upper bound of each key;
  • the union of the partitions' key lists, while it has at most 256 distinct keys.

The example above becomes key >= 1 AND key <= 20. Partitions without rows (THEN false) don't constrain the hull. A partition with no bound means nothing is pushed down.

The join still enforces its keys exactly, so a looser filter only means the scan returns more rows than it needs to.

This PR previously added dynamic filter support to the V1 scan. develop now scans with V2 by default and #11215 covers that path, so this branch merges develop keeping its tree, and carries only the partitioned join change.

Testing

  • Rstest cases in vortex-datafusion/src/v2/dynamic_filters.rs cover:
    • bounds and key lists;
    • a hash-table partition, which keeps only its bounds;
    • empty partitions;
    • an unbounded partition;
    • the 256-key limit;
    • every partition empty.
  • cargo test -p vortex-datafusion --lib v2::dynamic_filters passes.
  • cargo clippy -p vortex-datafusion --all-targets --all-features -- -D warnings passes.
  • cargo +nightly-2026-09-10 fmt -p vortex-datafusion is clean.
  • Not yet run: benchmarks.

https://claude.ai/code/session_01So4kxsqCY8WFG1NTMNJxKX


Generated by Claude Code

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants