Repository navigation
Prune static V2 DataFusion files before assigning scan partitions - #11344
joseph-isaacs wants to merge 1 commit into
Conversation
Use available file statistics before byte-range partitioning so workers share the surviving files. Preserve unknown files, leave dynamic and ordered scans on the existing path, and downgrade table statistics after pruning. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.008x ➖, 2↑ 1↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.043x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.027x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.919x ➖, 5↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.111x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.045x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.025x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.034x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.009x ➖, 1↑ 3↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.002x ➖, 1↑ 0↓)
|
Stack dependency: #11334. This PR targets that branch so its diff contains only this change.
Summary
DataFusion currently assigns byte-range partitions before its opener can exclude files. For eligible static V2 scans, prune from available file statistics during repartitioning so the retained files determine the scan groups. Files with unknown statistics remain eligible to scan.
Based on
ji/clickbench-v2-20261109-01-io-wakeups, the shared-segment wakeup fix. This change is independent of the DuckDB pruning branches.Changes
Benchmark evidence and hold
Whole-process user+system CPU increases 13.31% (99% interval 12.83–13.65%). Every one of the 24 primary process pairs increases CPU, and the corrected CPU regression test passes. CPU includes setup and warmups; it is not isolated Vortex CPU. This source-level hold remains despite the latency reductions.
Observed warm latency reductions against the exact dependency parent (zero-based query IDs):
24 fresh adjacent process pairs in balanced order; 100 executions/query, first 20 discarded; 8 interspersed identical-binary A/A pairs. Effects use mean paired log ratios of process warm medians; displayed time medians need not have exactly that ratio. Intervals are pointwise 99% bootstrap intervals. The focused latency family contains all 76 planned directions and the CPU family contains 24 directions, each with Holm 1% correction. No post-warmup observations were excluded or controls subtracted. These latency effects do not override the CPU hold and are not additive with other branches.
32-core aarch64 host, full 100-file partitioned ClickBench fixture, Vortex V2, ordinary local reads with no mmap, 32 workers/partitions/threads. DataFusion 55.2 retains
enable_topk_aggregation=true.Source inspection found an inherited automatic scan-concurrency multiplier as a possible contributor to the extra CPU. That hypothesis is unproven; no proposed tuning patch is included in this PR.
The matching CPU A/A control shows +0.309% apparent reduction (99% interval +0.073% to +0.532%). This small drift warning is retained; it does not explain the 13.31% measured CPU increase, and the control is not subtracted.
Full43 screening
The complete series screen finished 168/168 processes, with no failures or overlaps and successful source/SQL/binary/fixture checks. Each source/engine comparison used four balanced exact-parent process pairs and two matching identical-binary A/A pairs, with 10 executions/query and the first 3 discarded. This screen is descriptive: no new gain, equivalence, or regression-free claim is inferred, and no observations or controls were subtracted.
Keep this as a held latency/CPU trade-off. The corrected focused CPU increase is 13.31% (99% interval 12.83–13.65%), despite substantial focused latency gains. All43 CPU is 0.33% lower and latency sums are favorable, but this different workload composition cannot overturn the focused cost or establish equivalence. The small Q24 screen signal is control-sized and adds no independent hold. Publication is authorized; the existing CPU evidence already explains why landing needs further judgment.
Validation
persistent::source::tests,test_filter_with_schema_evolution,test_filter_schema_evolution_order.--locked,release_debug,profile.release_debug.debug=1, andRUSTFLAGS="-C target-cpu=native -C force-frame-pointers=yes". Rust formatting and patch whitespace checks passed. Workspace-wide clippy was not run; CI status is separate.Measured local commit:
3d81b7b8e2b48752de3332c4283aa0e6089500fc; full source tree:e227923037de048e9e851f6b91283378789533a4; measured parent:a2765d42438062db73d431ef6d296930e69f5b28.PR source commit:
768b49f9fe012ae9ef8e7ca72d6ea2cbff335c15; mapped parent:da10665e5b0fad748a1e0e6550cc19880edd0317. Complete source-tree equality, mapped parent, raw Git object, original message and author/committer signoffs were verified. GitHub regenerated commit metadata, so measured local commits and executable hashes remain the benchmark identities.Measurements use the develop snapshot
b2f8d0f946e003d3e463d1447ed357ed88990686. Develop advanced during the campaign; these results retain the recorded source and binary identities.Repository SQL benchmark
The SQL workflow completed 8/8 suites on this exact PR head. The ClickBench bot verdict is No clear signal (low confidence).
The preset covers V2 ClickBench, TPC-H, TPC-DS and FineWeb, plus a legacy PolarSignals control. These cloud runs use suite-specific historical develop baselines and
FLAT_LAYOUT_INLINE_ARRAY_NODE=1; they are separate from the local exact-parent comparisons and cannot isolate this stack's incremental effect. Ordinary draft CI was skipped; SQL benchmark success is not an overall CI pass.