Repository navigation
Preserve pruned V2 file groups without collapsing unpruned scans - #11345
joseph-isaacs wants to merge 2 commits into
Conversation
Let V2 parallelize chunks within each surviving file instead of repeating setup and dictionary decoding across tiny byte ranges. This includes filtered scans with unknown statistics. Keep the scheduling policy separate from early pruning because lower latency can trade against higher total CPU. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Previous Runs (8)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.055x ➖, 1↑ 2↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.927x ➖, 4↑ 1↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.214x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.059x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.077x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.020x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.044x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 1↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 2↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Restrict the V2 whole-file grouping preference to configurations where static statistics actually excluded input entries. Keep immutable input provenance on the returned source so repeated optimizer passes retain the preference without affecting sibling sources or changed inputs. Use ordinary byte-range partitioning when no inputs were excluded. Cover unknown and inclusive statistics, changed inputs and predicates, repeated target changes, and preservation of existing ranges. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
When eligible static pruning actually excludes an input and fewer file entries remain than the target partition count, keep each retained entry in a separate scan group without further splitting its existing range. Scans that retain every input use DataFusion's ordinary partitioner. This narrows the earlier policy, which also reduced grouping when a static predicate could not exclude anything.
Depends on #11344. Grouping after actual pruning can still limit scan and downstream parallelism; the CPU hold remains.
Changes
Local candidate evidence and hold
Measured local candidate
58c7b519fd4c54a31b3133973108cbeca6a48ff3(tree5c239cd927f012509270c5af3629bfb18df28e59) against exact s08 parent3d81b7b8e2b48752de3332c4283aa0e6089500fc:Query IDs are zero-based. These are paired effects from process warm medians; displayed time medians need not have exactly that ratio. All four pass the prespecified >1% statistical threshold. They do not override the hold.
For the selected ClickBench process workload, user+system CPU increases 22.97% (pointwise 99% interval 22.35–23.57%), passing the corrected CPU regression threshold. Peak RSS increases 12.99%, descriptively. CPU and RSS include process setup and all selected queries/iterations; they are not per-query or Vortex-only measurements.
The focused run completed 96 serial processes across TPC-H SF10 Q3/Q13, FineWeb Q0/Q4 and the four ClickBench queries: 12 balanced adjacent primary pairs plus four identical-candidate A/A pairs per suite, 31 iterations/query, first six discarded. All observations are retained. Paired-log bootstrap intervals are pointwise; exact >1% sign tests use Holm 1% correction across 16 latency directions and a separate six-direction CPU family. Ordinary local V2 reads use 32 workers/partitions and no mmap. DataFusion 55.2 keeps
enable_topk_aggregation=true. The fixtures are the full 100-file partitioned ClickBench dataset, TPC-H SF10, and the existing 1,048,576-row FineWeb sample.Q37's identical-candidate A/A control shows 1.05% apparent reduction (99% interval 0.52–1.59%), indicating drift or chance imbalance. No control is subtracted. Four A/A pairs provide limited calibration and do not establish equivalence. TPC-H/FineWeb's earlier large penalties were not reproduced in these selected local comparisons; the current small differences remain inconclusive.
The separate full43 screen completed 12/12 processes: four balanced parent/candidate pairs and two identical-candidate A/A pairs, 10 iterations/query with the first three discarded. Sum of query warm medians changed −0.04%, whole-process CPU +0.03%, and process wall time −0.22%, descriptively. This workload composition does not overturn the focused CPU or RSS cost, and the screen establishes neither equivalence nor a general no-regression claim.
The earlier Q12/Q25/Q27 median penalties were not reproduced at their prior magnitude: −0.07% / +0.09% / +0.11%. Remaining observations are retained for follow-up before landing:
These are unconfirmed screen signals, not additional confirmed regressions. No extra query gains were promoted from the screen.
Validation and publication status
Published source commit:
aea8f377061e38ef47b58b99726a318f80416fa0. Its complete tree matches the measured local candidate above. The remote parent is the previously published s09 commita112c0edb4125646e68b683a6101db6046c3eb7c; the PR still targets s08 (768b49f9fe012ae9ef8e7ca72d6ea2cbff335c15). Full source-tree equality, ordered parent, raw Git object, unchanged message and author/committer sign-offs were verified. GitHub regenerated metadata, so the local compiled commit and executable hash remain the measurement identity.Repository SQL benchmarks for this revised source are running on the published head. The build completed; eight benchmark suites are queued as of 2026-11-09 14:20:37 UTC. Completed SQL comments for the previous
a112c0ed...head used suite-specific historical develop baselines; those results are historical evidence, not validation of the revised source. Local measurements above compare the exact s08 dependency source. CI status is separate.