Repository navigation
perf(duckdb): avoid redundant legacy layout pruning in V2 scans - #11340
joseph-isaacs wants to merge 1 commit into
Conversation
Retain the opened VortexFile and evaluate a dtype-checked file-statistics proof before V2 execution. V2 continues to evaluate its own zone and row predicates; V1 keeps its existing layout-reader pruning path. Cover bound predicate schema validation and retained versus excluded files. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.051x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.049x ➖, 0↑ 3↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.049x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.953x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 1↓)
|
Stack dependency: #11334. This PR targets that branch so its diff contains only this change.
Summary
For DuckDB V2 scans that project input fields, evaluate whole-file footer statistics during reader initialization and leave zone and row filtering to V2. Count-only and constant projections keep the existing layout probe, which starts their zone reads during initialization.
Based on
ji/clickbench-v2-20261109-01-io-wakeups, the shared-segment wakeup fix used by this V2 series.Changes
VortexFile::can_prune_boundfor an already bound predicate, with root-schema validation before reading statistics.VortexFilealongside its layout reader and select the pruning path from the scan mode and projected input fields.Benchmark evidence
Warm ClickBench timings against this branch's exact dependency parent. Query IDs are zero-based; these are individual query gains, not whole-suite or additive gains.
24 fresh adjacent process pairs in balanced order; 100 executions/query, first 20 discarded; 8 interspersed identical-binary A/A pairs. Effects use mean paired log ratios of process warm medians. Displayed times are medians across processes, so their ratio need not exactly equal the paired estimate. Intervals are pointwise 99% bootstrap intervals. Each listed gain passes both the >1% interval rule and the exact paired sign test with Holm 1% correction across all 76 planned latency directions. Both execution orders and chronological sensitivity were reviewed; no post-warmup observations were excluded or controls subtracted. The sign test concerns how often a paired improvement exceeds 1%; the displayed interval estimates the mean log effect.
The focused Q1 guard is inconclusive: −0.35% latency reduction, 99% interval −1.07% to +0.37%. This does not prove absence of a small count-query regression.
32-core aarch64 host, full 100-file partitioned ClickBench fixture, Vortex V2, ordinary local reads with no mmap, 32 workers/partitions/threads, DuckDB
--reuse. DataFusion 55.2 retainsenable_topk_aggregation=true. Compiler, Cargo feature union, profile and runtime environment are fixed across parent/candidate. Executable and library hashes, canonical SQL and fixture identities are recorded.Full43 screening
The complete series screen finished 168/168 processes, with no failures or overlaps and successful source/SQL/binary/fixture checks. Each source/engine comparison used four balanced exact-parent process pairs and two matching identical-binary A/A pairs, with 10 executions/query and the first 3 discarded. This screen is descriptive: no new gain, equivalence, or regression-free claim is inferred, and no observations or controls were subtracted.
The focused Q37/Q38/Q42 gains remain scoped to their long-warm protocol. All43 DuckDB Q7 medians are 4.91% slower in every pair; means rise 3.53%, with one favorable pair. Matching controls are variable, so this is a landing concern rather than a proved regression. Q11 and Q29 remain secondary observations. Broad resource changes are small and mixed. Keep Q7 visible in dependent branches s05, s06, and s07; draft publication does not require another campaign.
Validation
scan_v2::tests,test_vortex_scan_v2_matches_v1.--locked, therelease_debugprofile,profile.release_debug.debug=1, andRUSTFLAGS="-C target-cpu=native -C force-frame-pointers=yes". Rust changes were formatted with pinned nightly; patch whitespace checks passed. Workspace-wide clippy was not run; CI status is separate.Measured local source commit:
0fbf06a77da9e1cfe226913262e601d90829a45a; full source tree:be4e97c235b2101c52a466b63bcdf51adf454b94. Measured parent:a2765d42438062db73d431ef6d296930e69f5b28.PR source commit:
2bc5703c632f9bdeca16cde41b222ea48d1dde35; mapped parent:da10665e5b0fad748a1e0e6550cc19880edd0317. The complete source tree and mapped parent graph match the measured source. GitHub regenerated commit metadata/timestamps, so the measured local commit and executable hashes remain the benchmark identities. The complete remote Git object, original message and author/committer signoffs were verified.Measurements use the develop snapshot
b2f8d0f946e003d3e463d1447ed357ed88990686. Develop advanced during the campaign; these results retain the recorded source and binary identities.Repository SQL benchmark
The SQL workflow completed 8/8 suites on this exact PR head. 1 suite(s) required one failed-jobs-only retry after EC2 capacity failures before checkout; the head stayed unchanged. The ClickBench bot verdict is No clear signal (low confidence).
The preset covers V2 ClickBench, TPC-H, TPC-DS and FineWeb, plus a legacy PolarSignals control. These cloud runs use suite-specific historical develop baselines and
FLAT_LAYOUT_INLINE_ARRAY_NODE=1; they are separate from the local exact-parent comparisons and cannot isolate this stack's incremental effect. Ordinary draft CI was skipped; SQL benchmark success is not an overall CI pass.