Repository navigation
perf(file): prepare static pruning proofs for multi-file scans - #11341
joseph-isaacs wants to merge 1 commit into
Conversation
Prepare file-statistics proofs once for DuckDB V2 scans and evaluate supported comparisons directly from footer scalars. Preserve three-valued logic, a general-expression fallback, root dtype validation, and rejection of dynamic predicates. Missing statistics remain unknown; each file supplies its own statistics. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.157x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.035x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.044x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.981x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 1↑ 1↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.063x ➖, 1↑ 2↓)
|
Stack dependency: #11340. This PR targets that branch so its diff contains only this change.
Summary
DuckDB applies the same static filter to multiple files. Prepare its file-statistics proof once, then evaluate that proof against each file’s footer. Simple comparisons and Boolean combinations use scalar statistics; other expressions use the existing evaluator.
Depends on
ji/clickbench-v2-20261109-04-duck-file-pruningfor the footer-pruning integration and bound-predicate fallback.Changes
FilePruningPredicateAPI with preparation, opened-file evaluation, and cached-footer evaluation.Benchmark evidence
Warm ClickBench timings against this branch's exact dependency parent. Query IDs are zero-based; these are individual query gains, not whole-suite or additive gains.
24 fresh adjacent process pairs in balanced order; 100 executions/query, first 20 discarded; 8 interspersed identical-binary A/A pairs. Effects use mean paired log ratios of process warm medians. Displayed times are medians across processes, so their ratio need not exactly equal the paired estimate. Intervals are pointwise 99% bootstrap intervals. Each listed gain passes both the >1% interval rule and the exact paired sign test with Holm 1% correction across all 76 planned latency directions. Both execution orders and chronological sensitivity were reviewed; no post-warmup observations were excluded or controls subtracted. The sign test concerns how often a paired improvement exceeds 1%; the displayed interval estimates the mean log effect.
DuckDB Q38 has an A/A drift warning: identical binaries show +0.436% apparent reduction. The main effect remains above 1% in both orders and all six chronological groups. This qualification is retained.
Whole-process user+system CPU reductions on the same focused query set: DuckDB 10.88% (99% interval 10.52–11.22%). These pass the separately corrected CPU family and include setup and warmups; they are not measurements of isolated Vortex CPU.
32-core aarch64 host, full 100-file partitioned ClickBench fixture, Vortex V2, ordinary local reads with no mmap, 32 workers/partitions/threads, DuckDB
--reuse. DataFusion 55.2 retainsenable_topk_aggregation=true. Compiler, Cargo feature union, profile and runtime environment are fixed across parent/candidate. Executable and library hashes, canonical SQL and fixture identities are recorded.Full43 screening
The complete series screen finished 168/168 processes, with no failures or overlaps and successful source/SQL/binary/fixture checks. Each source/engine comparison used four balanced exact-parent process pairs and two matching identical-binary A/A pairs, with 10 executions/query and the first 3 discarded. This screen is descriptive: no new gain, equivalence, or regression-free claim is inferred, and no observations or controls were subtracted.
No new independent landing hold emerges from the all43 screen, but the s04 dependency's Q7 concern remains. DuckDB Q42 medians/means rise 3.15%/5.41%; both baseline-first pairs are adverse and both candidate-first pairs favorable, with large mixed controls. This order/chronology split does not establish causation or clear history-sensitive behavior. Retain the focused Q38/Q41/Q42 gains and their control caveat; the 10.88% focused CPU reduction is not an all43 CPU claim.
Validation
prepared_file_pruning,scalar_proofs_preserve_three_valued_logic,test_can_prune.--locked, therelease_debugprofile,profile.release_debug.debug=1, andRUSTFLAGS="-C target-cpu=native -C force-frame-pointers=yes". Rust changes were formatted with pinned nightly; patch whitespace checks passed. Workspace-wide clippy was not run; CI status is separate.Measured local source commit:
b9d988cae18a03f88689353b81a3e6b3fd94a879; full source tree:6720a94ed0f9e69f3d03089556c83ac32aeeb81b. Measured parent:0fbf06a77da9e1cfe226913262e601d90829a45a.PR source commit:
7fb2efcfb1ea8f751e9e31d60109de325ecc420e; mapped parent:2bc5703c632f9bdeca16cde41b222ea48d1dde35. The complete source tree and mapped parent graph match the measured source. GitHub regenerated commit metadata/timestamps, so the measured local commit and executable hashes remain the benchmark identities. The complete remote Git object, original message and author/committer signoffs were verified.Measurements use the develop snapshot
b2f8d0f946e003d3e463d1447ed357ed88990686. Develop advanced during the campaign; these results retain the recorded source and binary identities.Repository SQL benchmark
The SQL workflow completed 8/8 suites on this exact PR head. 1 suite(s) required one failed-jobs-only retry after EC2 capacity failures before checkout; the head stayed unchanged. The ClickBench bot verdict is No clear signal (low confidence).
The preset covers V2 ClickBench, TPC-H, TPC-DS and FineWeb, plus a legacy PolarSignals control. These cloud runs use suite-specific historical develop baselines and
FLAT_LAYOUT_INLINE_ARRAY_NODE=1; they are separate from the local exact-parent comparisons and cannot isolate this stack's incremental effect. Ordinary draft CI was skipped; SQL benchmark success is not an overall CI pass.