Repository navigation
Plan static and dynamic conjuncts together and prune chunks again on new dynamic values - #11234
joseph-isaacs wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
…new dynamic values The filter planner takes both kinds of conjunct and orders them together, as a join's key bounds are often the most selective. Each conjunct now carries its own plans, and a dynamic one also records how selective it was at its current values. Every scan of the file records it; a scan that may skip unselective dynamic conjuncts, a flag of the prepared scan rather than of the cached plans, acts on it. FilterExpr goes back to ordering alone. The chunk planner prunes a chunk a second time, with a DynamicPrunePlanner, when a dynamic conjunct's values changed after the scan was pruned, such as a top-K bound tightening while earlier chunks ran. It shares its zone pruning loop with the PrunePlanner and is only made when a value changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
29999d8 to
724ba40
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.014x ➖, 0↑ 1↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.025x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 1↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.988x ➖, 1↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.030x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.035x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
|
Summary
Restructures how V2 scans handle dynamic conjuncts, so the logic for each kind lives in the planners rather than in a flag threaded through
ScanBuilder,FilterPlansandFilterExpr. It also adds a second, per-chunk prune for dynamic values that change during a scan.Changes
FilterPlansholds oneConjunctper conjunct. Each has its predicate plan and pruning plan; a dynamic one also has aDynamicConjunctwith its unselective streak.FilterExprgoes back to ordering only.PreparedScan::skip_unselective_dynamic, so the cached filter plans no longer need the "reuse only with the same setting" special case.FilterPlannertakes static and dynamic conjuncts together, in one order from their observed selectivity, since a join's key bounds are often the most selective conjunct.start_nextandreportdispatch on the kind:DynamicPrunePlanner, emitted byChunkPlanner, prunes a chunk again with any dynamic conjunct whose values changed since the chunk planner was made, such as a top-K bound tightening while earlier chunks ran.ZonePruningloop withPrunePlanner.FilterPlans:scan_pruningfor the once-per-scan prune.moved_dynamic_pruningfor the per-chunk prune.SelectedRows::narrowedtrims a chunk's rows after pruning. Chunk selection uses it too.Benchmarks
Local, DataFusion with the V2 scan, interleaved per query. Ratios are against dynamic filters off; lower is faster. The baseline is
developwith #11215, measured before the rebase.Performance is at parity. The outliers in the first pass (TPC-DS q3, ClickBench q26 and q36) were within noise when re-run with more rounds. ClickBench TopK queries keep their wins: q23 0.34, q24 0.39, q25 0.77.
Checks
cargo test -p vortex-layout -p vortex-datafusion: all pass exceptzone_maps_hold_no_variant_state::case_2_variant_in_struct, which fails the same way ondevelop.cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warnings: clean.cargo fmt -p vortex-layout -- --check: clean.chunks_prune_again_when_a_dynamic_value_changes. It fails with the per-chunk prune disabled.vortex-duckdbtests, sqllogictest.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBC16P5xug6P3j7sxr2