Repository navigation
Speed up V2 scans and default datafusion-bench to V2 for TPC-DS and ClickBench - #10799
Conversation
Merging this PR will degrade performance by 10.41%3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes |
Polar Signals Profiling ResultsLatest Run
Previous Runs (70)
Powered by Polar Signals Cloud |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.990x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.885x ✅, 10↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.960x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.949x ➖, 23↑ 3↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 1↑ 4↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.936x ➖, 2↑ 1↓)
datafusion / parquet / ns (0.952x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.018x ➖, 1↑ 2↓)
duckdb / parquet / ns (0.972x ➖, 0↑ 0↓)
|
4960855 to
02d9b08
Compare
02d9b08 to
ba0fb42
Compare
ba0fb42 to
78afd60
Compare
78afd60 to
06c0701
Compare
06c0701 to
bef7c23
Compare
4e32a01 to
75af0e4
Compare
8f49581 to
e503266
Compare
A V2 scan reads each chunk through a session of its own, and the file's segment source shares a read only while it is in flight. A stored segment that spans a chunk boundary was therefore read again by every chunk that started after its neighbour had finished, and by every partition and every concurrent scan of the file. TPC-DS q28 on SF1 read 188 MB where V1 reads 114 MB, every segment V1 reads up to two and a half times as often. The file's ScanCache now retains up to 64 MiB of recently read host segments in least-recently-used order, and every scan of the file reads through it. ScanCache::with_retained_bytes sets the budget; zero disables it. q28 now reads 125 MB. TPC-DS SF1 on DataFusion, V2 vs V1 geomean: 1.052 -> 1.002. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Codex <codex@openai.com>
TPC-H already defaults to the planner-driven V2 scan. Extend the default to TPC-DS and ClickBench so the SQL benchmarks measure V2 on them too. `--scan-v2 false` or `VORTEX_SCAN_V2=0` still selects V1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Codex <codex@openai.com>
The V2 scan cache keyed filter and projection plans by `ExactBoundExpr`, which compares children by pointer. DataFusion opens each partition of a file separately and binds its predicate and projection anew, so no partition ever found the plans another had built: every partition lowered its own filter plans, optimized its own projection and re-evaluated the zone maps. The cost scales with the partition count, so TPC-DS q09 and q28 ran twice as long as V1 on the 32-core benchmark runners. Key the cache by structure instead. Hashing walks the scalar functions and skips dtypes, so a lookup does not walk a wide schema; equality still compares dtypes. Both walk the tree iteratively. Locally (4 cores, 32 partitions), q09/q13/q28/q44/q48 go from a 1.41x geomean of V1 to 1.15x, and q28 from 1.57x to 1.15x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Codex <codex@openai.com>
A V2 scan cuts its rows on a fixed grid in the file's row domain, but DataFusion's partition edges fall off that grid. Every partition of a filtered scan then planned a sliver at its edge: TPC-DS q28 at 32 partitions cut each 90,112-row partition into chunks of 81,920 and 8,192 rows. The sliver costs a whole chunk's pipeline compile and conjunct setup, which dominates a selective filter over few rows. The chunk planner now joins an edge chunk under half a chunk to its neighbour; interior boundaries stay on the grid. Locally at 32 partitions, q28's V2 CPU time falls from 1.32x V1 to 1.10x and its wall time matches V1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYor8ydYMRcjUSh622VTEC Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Codex <codex@openai.com>
Expose encoded values for large string dictionaries when a batch selects fewer than one sixteenth of their entries. Encoding-specific take kernels can then gather the selected values without canonicalizing the whole dictionary. Keep shared decoding for small and denser takes, and reuse any already-cached canonical values. Cover nullable UTF-8 and binary values, nullable codes, and dictionary cache reuse in the pipeline tests. Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Codex <codex@openai.com>
Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
e503266 to
f0752c6
Compare
Rebased onto develop as one commit. Conflicts with #10799 resolved: - The scan cache keeps develop's structural ExprKey and retained segments, with this branch's build-once OnceCell entries. - A filtered chunk is sized by concurrency up to develop's per-shape cap (128Ki rows, or 1Mi for one fixed-width field), never below 64Ki. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEEc3okQ7JZh6AZDur
Reconcile with merged #10799 and leave overlapping scheduling and read ownership policies to #10789, #10857, and #10969. Preserve current develop's structural cache, sparse dictionary threshold, chunk sizing, short-edge merging, retained segments, and benchmark defaults. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Rebased onto develop as one commit. Conflicts with #10799 resolved: - The scan cache keeps develop's structural ExprKey and retained segments, with this branch's build-once OnceCell entries. - A filtered chunk is sized by concurrency up to develop's per-shape cap (128Ki rows, or 1Mi for one fixed-width field), never below 64Ki. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEEc3okQ7JZh6AZDur
Rebased onto develop as one commit. Conflicts with #10799 resolved: - The scan cache keeps develop's structural ExprKey and retained segments, with this branch's build-once OnceCell entries. - A filtered chunk is sized by concurrency up to develop's per-shape cap (128Ki rows, or 1Mi for one fixed-width field), never below 64Ki. Rebased again onto develop. Conflicts resolved: - The filter planner keeps develop's dynamic-conjunct skipping, with the conjunct's version held beside its MaskScan. - Filter plans built once per filter still honour develop's skip_unselective_dynamic: a scan whose setting differs from the cached plans plans its own. - Roots lower through Layout::new_plan, and is_identity_expression keeps develop's bool return. - v2_inline_chunks_read_through_their_own_sessions now sets scan_concurrency 1: 0 means unset here, and 1 is what runs inline. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEEc3okQ7JZh6AZDur Claude-Session: https://claude.ai/code/session_01UeniB9i8ce2ie4BaCQkHSc
V2 ClickBench scans can do less CPU work than V1 yet take longer because 1M-row filtered chunks leave most workers idle behind a few expensive chunks. Use 128K-row chunks for filtered string and wider scans. Keep 1M-row chunks when the filter and projection together read only one primitive or boolean field, where smaller chunks mainly add setup overhead.
This PR also reuses structurally equivalent scan plans, retains recently read segments, joins short edge chunks, and preserves encoded values for sparse string/binary dictionary takes. It enables V2 by default in
datafusion-benchfor ClickBench and TPC-DS.The filter/slice rewrite is already in
developthrough #10959, so it is no longer part of this diff.DataFusion performance
Matched local ARM64 measurements compare V2 before and after the chunk-sizing change, with the filter/slice rewrite present in both binaries. CPU affinity was 2–31; Tokio workers and DataFusion partitions were both 30. Each query ran nine iterations, discarding the first and reporting the median of the remaining eight. No competing benchmark or build overlap was detected.