Repository navigation
Conversation
|
DuckDB benchmarks, V1 scan ( Headline: total time is flat: TPC-H 0.99× and ClickBench 1.00×. The geomean is 0.98× for TPC-H and 0.97× for ClickBench. Regressions over 5% that held in both rounds:
These are mostly short queries, 9–50 ms. I haven't tuned anything. Caveat: the machine was not idle. Other sessions were compiling on it, with a load average of about 74 during the follow-up A/B. That A/B ran the flagged queries on develop, this PR as it is (4 splits in flight per thread), and this PR with
The gaps grew with the load. Lowering in-flight splits to 1 closes little of it, except on q22, so the new default of 4 is not the main cause. These numbers need a rerun on a quiet machine before reading anything into single queries. Full first run (ms, median)tpch
geomean ratio: 0.976 clickbench
geomean ratio: 0.966 |
975aa4f to
5c76944
Compare
aafa512 to
0bf4d78
Compare
A host reading an opened scan, such as DuckDB showing a progress bar per file, asks its stream how much of the work its clones have taken. A layout scan's stream reports the fraction of its splits claimed; a stream over partitions reports none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Each file DuckDB's MultiFileReader opens is opened once as a V1 layout scan, and every DuckDB thread that joins the file polls a clone of its stream. A thread joins a file until the first clone ends, which happens only once no split is left to claim, so threads move on to the next file without waiting for the last splits to finish. DuckDB's private per-file split list is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
0bf4d78 to
cdb5fad
Compare
A ScanStream is Send but not Sync, so the file state that every thread scanning the file shares holds it behind a mutex, and each thread clones it there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
DuckDB runs one thread per core, and each polls its own clone of a file's scan stream. A clone that keeps several splits in flight claims splits that other threads could have run, so on a file with few splits, such as TPC-H's smaller tables, most threads find the queue empty. Capping each clone at one split matches DuckDB's own parallelism. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ates/duckdb-stream-3 Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3 Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3 Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3 Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3 Signed-off-by: Nicholas Gates <nick@nickgates.com>
DuckDB iterates files itself: its
MultiFileReaderopens each file, initializes it once under a file lock, and hands threads to it under a global lock until the file reports no more work, then closes it and never returns to it. The Vortex table function maps that onto the Scan API with oneopenper file. Initializing a file builds aScanRequestfrom DuckDB's pushed-down projection, filter, row range and selection, scans it throughLayoutReaderDataSource, and opens it into aScanStream. Each DuckDB thread that joins the file polls its own clone of that stream withblock_on, so DuckDB's threads are the scan's parallelism and the scan starts no threads of its own.One
openper file, rather than one across the glob, keeps DuckDB in charge of what it already owns: file order, globs, thefilenameand hive columns, file-level skipping from footer statistics, and per-file progress. A file's V1 scan opens into a stream over a queue of its splits. DuckDB runs one thread per core, so its threads are the scan's parallelism, and the request setssplit_concurrencyto one: each clone claims the next split only once it has finished its last, as each DuckDB thread runs one split at a time. A clone with room for more would claim splits that other threads could have run, so on a file with few splits, such as TPC-H's smaller tables, most threads would find the queue empty. With one split in flight, a clone polls the split itself rather than spawning it on DuckDB's shared runtime, so the thread that wants the split's array decodes it, with no wakeup from another thread, and exports it from the same core. The request is unordered with no limit: DuckDB's parallel scan does not promise row order across threads, andfile_row_numberis projected per row rather than counted, so it stays correct whichever thread reads a split.A clone ends only once no split is left to claim and the splits it claimed are done, so DuckDB hands threads to a file until the first of its clones ends. From then on threads move on to the next file while the clones holding the last splits finish them: a clone yields only the splits it claimed, so no thread waits on another's work, and splits of neighbouring files overlap. DuckDB's progress bar reads
ScanStream::progress, which this PR adds: the fraction of the scan's work its clones have taken, which a layout scan reports as the fraction of its splits claimed.DuckDB's private per-file list of V1 split tasks is removed; the stream's split queue is that list behind the Scan API. V2 scans keep their own queue until the V2 source is behind
open.This PR is the stack's caller of
open: a morsel-driven host whose threads each pull from a shared stream, the counterpart to DataFusion's fixed partitions throughsplitin #11666. It is part of the Scan API work tracked in #9062.Local DuckDB benchmarks, as head time over develop time (geomean), with each query interleaved with develop over at least three rounds:
Stacked on #11666
🤖 Generated with Claude Code