Repository navigation
Overlap reads across two DuckDB Scan V2 chunks per thread - #11140
joseph-isaacs wants to merge 1 commit into
Conversation
Each DuckDB thread ran one V2 chunk at a time, so it sat idle whenever that chunk waited on storage. Keep two chunks live on each thread's run so one chunk's reads overlap the other's compute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.988x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.042x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.092x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 4↑ 3↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.926x ➖, 6↑ 1↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.058x ➖, 0↑ 16↓)
duckdb / parquet / ns (1.001x ➖, 1↑ 2↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.973x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.528x ❌, 0↑ 15↓)
datafusion / parquet / ns (1.105x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.181x ➖, 0↑ 7↓)
duckdb / parquet / ns (1.173x ➖, 0↑ 5↓)
|
Summary
Split out of #10964 as an independently measurable piece.
Each DuckDB thread ran one Scan V2 chunk at a time, so it sat idle whenever that chunk waited on storage. With two chunks live on each thread's run, one chunk's reads overlap the other's compute.
Changes
CHUNKS_PER_THREADinvortex-duckdb/src/scan_v2.rsgoes from 1 to 2.Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations against
develop(246a397). Base-vs-base noise: +1.3% geomean, up to ±8% per query.cargo nextest run -p vortex-duckdb: 276 passed, 2 skipped.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code