Repository navigation
Reserve DuckDB Scan V2 work under the global lock before joining a file - #11148
joseph-isaacs wants to merge 1 commit into
Conversation
A thread joined a V2 file whenever its queue was not yet exhausted, then took its first chunk after DuckDB released the global lock. Several threads could join for the last chunk; the losers found no work, returned to DuckDB empty-handed, and moved on to the next file only after another round through the scheduler. Take the thread's first chunk while DuckDB still holds the global lock, so a thread joins a file only with work in hand, and run it outside the lock before taking more. Threads with no work left in the file move straight on to the next one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.066x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.977x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 1↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.053x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.071x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.047x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.020x ➖, 0↑ 1↓)
|
Summary
Split out of #10964 as an independently measurable piece.
A thread joined a V2 file whenever the file's queue was not yet exhausted, then took its first chunk after DuckDB released the global lock. Several threads could join for the last chunk. The losers found no work, returned to DuckDB empty-handed, and only moved on to the next file after another round through the scheduler.
Changes
reader_try_initialize_scantakes the thread's first chunk while DuckDB still holds the global lock, so a thread joins a file only with work in hand.Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations against
develop(246a397). Base-vs-base noise: +1.3% geomean, up to ±8% per query.The TPC-H tables are single files at SF10, so there is little cross-file contention. The multi-file ClickBench run is still in progress locally.
cargo nextest run -p vortex-duckdb: 284 passed, 2 skipped.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code