Repository navigation
perf(duckdb): group small filtered V2 scan parts on layout boundaries - #11343
joseph-isaacs wants to merge 1 commit into
Conversation
Amortize static filter planning over adjacent layout splits. Use exact binary-flag statistics to choose larger parts for selective filters, preserving gaps and leaving dynamic-filter scheduling unchanged. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.981x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.988x ➖, 1↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.024x ➖, 2↑ 3↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.146x ➖, 0↑ 5↓)
datafusion / parquet / ns (1.023x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.026x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.030x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.986x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 1↓)
|
Stack dependency: #11340. This PR targets that branch so its diff contains only this change.
Summary
Static filtered V2 scans currently create a pruning task for each layout part. Group small adjacent parts while preserving layout boundaries, and use exact binary-flag statistics to choose larger groups for selective filters.
Depends on
ji/clickbench-v2-20261109-04-duck-file-pruningfor access to the opened file’s footer. It does not require the prepared-proof or pre-open-pruning branches.Changes
Benchmark evidence
Warm ClickBench timings against this branch's exact dependency parent. Query IDs are zero-based; these are individual query gains, not whole-suite or additive gains.
24 fresh adjacent process pairs in balanced order; 100 executions/query, first 20 discarded; 8 interspersed identical-binary A/A pairs. Effects use mean paired log ratios of process warm medians. Displayed times are medians across processes, so their ratio need not exactly equal the paired estimate. Intervals are pointwise 99% bootstrap intervals. Each listed gain passes both the >1% interval rule and the exact paired sign test with Holm 1% correction across all 76 planned latency directions. Both execution orders and chronological sensitivity were reviewed; no post-warmup observations were excluded or controls subtracted. The sign test concerns how often a paired improvement exceeds 1%; the displayed interval estimates the mean log effect.
DuckDB Q38 has an A/A drift warning: identical binaries show -0.630% apparent reduction. The main effect remains above 1% in both orders and all six chronological groups. This qualification is retained.
DuckDB Q39 has an A/A drift warning: identical binaries show -0.223% apparent reduction. The main effect remains above 1% in both orders and all six chronological groups. This qualification is retained.
Whole-process user+system CPU reductions on the same focused query set: DuckDB 5.32% (99% interval 5.10–5.53%). These pass the separately corrected CPU family and include setup and warmups; they are not measurements of isolated Vortex CPU.
32-core aarch64 host, full 100-file partitioned ClickBench fixture, Vortex V2, ordinary local reads with no mmap, 32 workers/partitions/threads, DuckDB
--reuse. DataFusion 55.2 retainsenable_topk_aggregation=true. Compiler, Cargo feature union, profile and runtime environment are fixed across parent/candidate. Executable and library hashes, canonical SQL and fixture identities are recorded.Full43 screening
The complete series screen finished 168/168 processes, with no failures or overlaps and successful source/SQL/binary/fixture checks. Each source/engine comparison used four balanced exact-parent process pairs and two matching identical-binary A/A pairs, with 10 executions/query and the first 3 discarded. This screen is descriptive: no new gain, equivalence, or regression-free claim is inferred, and no observations or controls were subtracted.
The focused Q36/Q38/Q39 gains remain valid with their disclosed control warnings. All43 DuckDB Q42 medians are 9.65% slower in every pair, but means rise only 1.49%, and the candidate has fewer extreme events. Describe a median/distribution concern, not worse extreme tails; matching mean controls are very noisy. Q4/Q7 are secondary observations, and the s04 dependency concern remains. Favorable all43 latency sums are descriptive and do not create an additional gain claim.
Validation
scan_v2::tests,test_vortex_scan_v2_groups_nullable_binary_flag_parts,test_vortex_scan_v2_matches_v1.--locked, therelease_debugprofile,profile.release_debug.debug=1, andRUSTFLAGS="-C target-cpu=native -C force-frame-pointers=yes". Rust changes were formatted with pinned nightly; patch whitespace checks passed. Workspace-wide clippy was not run; CI status is separate.Measured local source commit:
9390b632a282f0b4facbc8b25944941ccd7c947c; full source tree:bc746c57847ca56c006ca170e4ba068947030b20. Measured parent:0fbf06a77da9e1cfe226913262e601d90829a45a.PR source commit:
1d56e333cd4b1c3bafbe2bab77e6a1c266bb2c6a; mapped parent:2bc5703c632f9bdeca16cde41b222ea48d1dde35. The complete source tree and mapped parent graph match the measured source. GitHub regenerated commit metadata/timestamps, so the measured local commit and executable hashes remain the benchmark identities. The complete remote Git object, original message and author/committer signoffs were verified.Measurements use the develop snapshot
b2f8d0f946e003d3e463d1447ed357ed88990686. Develop advanced during the campaign; these results retain the recorded source and binary identities.Repository SQL benchmark
The SQL workflow completed 8/8 suites on this exact PR head. 3 suite(s) required one failed-jobs-only retry after EC2 capacity failures before checkout; the head stayed unchanged. The ClickBench bot verdict is No clear signal (low confidence).
The preset covers V2 ClickBench, TPC-H, TPC-DS and FineWeb, plus a legacy PolarSignals control. These cloud runs use suite-specific historical develop baselines and
FLAT_LAYOUT_INLINE_ARRAY_NODE=1; they are separate from the local exact-parent comparisons and cannot isolate this stack's incremental effect. Ordinary draft CI was skipped; SQL benchmark success is not an overall CI pass.