Repository navigation
perf(duckdb): prune excluded files from cached footers before opening - #11342
Conversation
Pass the prepared static V2 predicate through the reader FFI and use the existing multi-file footer cache to skip excluded files before creating their readers. Keep planning opens and unknown footers on the ordinary open path. Cover excluded, retained, absent, and uncached files with a regression test. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.030x ➖, 2↑ 2↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.055x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.013x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.981x ➖, 6↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.107x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.048x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.056x ➖, 0↑ 2↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Stack dependency: #11341. This PR targets that branch so its diff contains only this change.
Summary
When a cached footer proves that a static V2 filter excludes a file, return a pruned-reader marker before opening its file source. DuckDB consumes that marker during reader initialization.
Depends on
ji/clickbench-v2-20261109-05-prepared-file-proofsfor the prepared predicate and cached-footer evaluation API.Changes
Benchmark evidence
Warm ClickBench timings against this branch's exact dependency parent. Query IDs are zero-based; these are individual query gains, not whole-suite or additive gains.
24 fresh adjacent process pairs in balanced order; 100 executions/query, first 20 discarded; 8 interspersed identical-binary A/A pairs. Effects use mean paired log ratios of process warm medians. Displayed times are medians across processes, so their ratio need not exactly equal the paired estimate. Intervals are pointwise 99% bootstrap intervals. Each listed gain passes both the >1% interval rule and the exact paired sign test with Holm 1% correction across all 76 planned latency directions. Both execution orders and chronological sensitivity were reviewed; no post-warmup observations were excluded or controls subtracted. The sign test concerns how often a paired improvement exceeds 1%; the displayed interval estimates the mean log effect.
Whole-process user+system CPU reductions on the same focused query set: DuckDB 21.73% (99% interval 21.51–21.96%). These pass the separately corrected CPU family and include setup and warmups; they are not measurements of isolated Vortex CPU.
32-core aarch64 host, full 100-file partitioned ClickBench fixture, Vortex V2, ordinary local reads with no mmap, 32 workers/partitions/threads, DuckDB
--reuse. DataFusion 55.2 retainsenable_topk_aggregation=true. Compiler, Cargo feature union, profile and runtime environment are fixed across parent/candidate. Executable and library hashes, canonical SQL and fixture identities are recorded.Full43 screening
The complete series screen finished 168/168 processes, with no failures or overlaps and successful source/SQL/binary/fixture checks. Each source/engine comparison used four balanced exact-parent process pairs and two matching identical-binary A/A pairs, with 10 executions/query and the first 3 discarded. This screen is descriptive: no new gain, equivalence, or regression-free claim is inferred, and no observations or controls were subtracted.
Focused Q37/Q38/Q42 gains do not settle the full-history concerns. DuckDB Q26 is 4.79% slower by median and 2.00% by mean, with three adverse pairs after a favorable first pair. Q42 means rise 14.22%, including a 17.97 ms candidate event; controls themselves vary substantially. These are unresolved landing concerns, not proved regressions. All43 CPU is nearly flat, so the focused 21.73% CPU reduction stays workload-specific. The s04 dependency concern also remains.
Validation
file_reader::tests,test_vortex_scan_prunes_multiple_cached_files,test_vortex_scan_v2_matches_v1.--locked, therelease_debugprofile,profile.release_debug.debug=1, andRUSTFLAGS="-C target-cpu=native -C force-frame-pointers=yes". Rust changes were formatted with pinned nightly; patch whitespace checks passed. Workspace-wide clippy was not run; CI status is separate.Measured local source commit:
baf05cc29b5125d250113332ce8c20d94d9b61d3; full source tree:9bb1f1ed623a657e8836347e15f98e887d3f1d8b. Measured parent:b9d988cae18a03f88689353b81a3e6b3fd94a879.PR source commit:
e6ec8c58732f5fb49666c3fa4a066ff1408f1a5d; mapped parent:7fb2efcfb1ea8f751e9e31d60109de325ecc420e. The complete source tree and mapped parent graph match the measured source. GitHub regenerated commit metadata/timestamps, so the measured local commit and executable hashes remain the benchmark identities. The complete remote Git object, original message and author/committer signoffs were verified.Measurements use the develop snapshot
b2f8d0f946e003d3e463d1447ed357ed88990686. Develop advanced during the campaign; these results retain the recorded source and binary identities.Repository SQL benchmark
The SQL workflow completed 8/8 suites on this exact PR head. The ClickBench bot verdict is No clear signal (low confidence).
The preset covers V2 ClickBench, TPC-H, TPC-DS and FineWeb, plus a legacy PolarSignals control. These cloud runs use suite-specific historical develop baselines and
FLAT_LAYOUT_INLINE_ARRAY_NODE=1; they are separate from the local exact-parent comparisons and cannot isolate this stack's incremental effect. Ordinary draft CI was skipped; SQL benchmark success is not an overall CI pass.