Skip to content

Read DuckDB's V1 scans through open - #11664

Draft
gatesn wants to merge 10 commits into
ngates/datafusion-scan-poolfrom
ngates/duckdb-scan-pool
Draft

gatesn wants to merge 10 commits into
ngates/datafusion-scan-poolfrom
ngates/duckdb-scan-pool

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

DuckDB iterates files itself: its MultiFileReader opens each file, initializes it once under a file lock, and hands threads to it under a global lock until the file reports no more work, then closes it and never returns to it. The Vortex table function maps that onto the Scan API with one open per file. Initializing a file builds a ScanRequest from DuckDB's pushed-down projection, filter, row range and selection, scans it through LayoutReaderDataSource, and opens it into a ScanStream. Each DuckDB thread that joins the file polls its own clone of that stream with block_on, so DuckDB's threads are the scan's parallelism and the scan starts no threads of its own.

One open per file, rather than one across the glob, keeps DuckDB in charge of what it already owns: file order, globs, the filename and hive columns, file-level skipping from footer statistics, and per-file progress. A file's V1 scan opens into a stream over a queue of its splits. DuckDB runs one thread per core, so its threads are the scan's parallelism, and the request sets split_concurrency to one: each clone claims the next split only once it has finished its last, as each DuckDB thread runs one split at a time. A clone with room for more would claim splits that other threads could have run, so on a file with few splits, such as TPC-H's smaller tables, most threads would find the queue empty. With one split in flight, a clone polls the split itself rather than spawning it on DuckDB's shared runtime, so the thread that wants the split's array decodes it, with no wakeup from another thread, and exports it from the same core. The request is unordered with no limit: DuckDB's parallel scan does not promise row order across threads, and file_row_number is projected per row rather than counted, so it stays correct whichever thread reads a split.

A clone ends only once no split is left to claim and the splits it claimed are done, so DuckDB hands threads to a file until the first of its clones ends. From then on threads move on to the next file while the clones holding the last splits finish them: a clone yields only the splits it claimed, so no thread waits on another's work, and splits of neighbouring files overlap. DuckDB's progress bar reads ScanStream::progress, which this PR adds: the fraction of the scan's work its clones have taken, which a layout scan reports as the fraction of its splits claimed.

DuckDB's private per-file list of V1 split tasks is removed; the stream's split queue is that list behind the Scan API. V2 scans keep their own queue until the V2 source is behind open.

This PR is the stack's caller of open: a morsel-driven host whose threads each pull from a shared stream, the counterpart to DataFusion's fixed partitions through split in #11666. It is part of the Scan API work tracked in #9062.

Local DuckDB benchmarks, as head time over develop time (geomean), with each query interleaved with develop over at least three rounds:

Suite V1 V2
TPC-H 1.011 1.002
ClickBench parity: no query above 1.05 in every rotation (q15 0.98–1.03, q17 0.95–1.08) 0.985

Stacked on #11666

🤖 Generated with Claude Code

@gatesn gatesn added changelog/performance A performance improvement changelog/chore A trivial change and removed changelog/performance A performance improvement labels Nov 19, 2026
@gatesn gatesn changed the title Read DuckDB scans through scan pools Read DuckDB's V1 scans through scan pools Nov 19, 2026
@gatesn gatesn mentioned this pull request Nov 19, 2026
16 of 46 tasks
@gatesn

gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

DuckDB benchmarks, V1 scan (duckdb-bench --scan-v1 true, --formats vortex), develop (d3688347ff) against this PR (d891c22319) on an M-series laptop. TPC-H is SF1 and ClickBench is the 100-file partitioned set, which also covers multi-file globs. Each target ran two interleaved rounds of 5 iterations, and the tables show the median of 8 timings after dropping each round's first iteration.

Headline: total time is flat: TPC-H 0.99× and ClickBench 1.00×. The geomean is 0.98× for TPC-H and 0.97× for ClickBench.

Regressions over 5% that held in both rounds:

  • TPC-H: q22 (1.43× and 1.39×), q10 (1.23× and 1.16×), q16 (1.09× and 1.12×).
  • ClickBench: q23 (1.12× and 1.21×), q39 (1.12× and 1.13×), q15, and q28 (1.07× and 1.05×).

These are mostly short queries, 9–50 ms. I haven't tuned anything.

Caveat: the machine was not idle. Other sessions were compiling on it, with a load average of about 74 during the follow-up A/B. That A/B ran the flagged queries on develop, this PR as it is (4 splits in flight per thread), and this PR with split_concurrency = 1:

query develop 4 in flight 1 in flight
tpch q02 8.6 13.5 12.7
tpch q10 13.2 20.0 18.4
tpch q16 8.9 12.3 12.2
tpch q22 8.3 16.0 11.8
clickbench q09 190 286 263
clickbench q23 37.7 51.4 54.6
clickbench q28 1917 2699 2621
clickbench q39 23.6 22.5 22.4

The gaps grew with the load. Lowering in-flight splits to 1 closes little of it, except on q22, so the new default of 4 is not the main cause. These numbers need a rerun on a quiet machine before reading anything into single queries.

Full first run (ms, median)

tpch

query develop (ms) pool (ms) ratio
q01 10.7 11.2 1.05
q02 8.7 10.5 1.21
q03 10.5 9.8 0.93
q04 7.0 6.7 0.96
q05 12.4 13.0 1.04
q06 4.2 2.9 0.69
q07 12.5 12.8 1.02
q08 13.9 13.9 1.00
q09 21.8 21.0 0.97
q10 13.7 16.4 1.20 ⚠️
q11 4.6 4.6 1.01
q12 10.2 9.5 0.93
q13 15.5 15.3 0.99
q14 7.0 5.9 0.85
q15 5.5 4.3 0.78
q16 9.4 10.3 1.10
q17 9.6 8.3 0.86
q18 17.8 17.0 0.96
q19 7.9 7.3 0.93
q20 10.2 9.5 0.93
q21 26.6 23.6 0.89
q22 9.2 13.1 1.43 ⚠️
total 249 247 0.993

geomean ratio: 0.976

clickbench

query develop (ms) pool (ms) ratio
q00 1.1 1.1 1.04
q01 9.5 6.8 0.71
q02 12.7 9.9 0.78
q03 36.3 34.1 0.94
q04 107.0 112.8 1.05 ⚠️
q05 107.4 108.0 1.01
q06 1.5 1.8 1.22
q07 11.2 10.4 0.93
q08 139.3 148.5 1.07 ⚠️
q09 207.4 228.1 1.10 ⚠️
q10 41.7 33.4 0.80
q11 45.7 39.9 0.87
q12 111.8 116.0 1.04
q13 234.4 237.9 1.01
q14 132.7 139.5 1.05 ⚠️
q15 147.1 158.7 1.08 ⚠️
q16 308.5 324.8 1.05 ⚠️
q17 301.8 237.5 0.79
q18 611.9 559.9 0.92
q19 18.8 11.9 0.64
q20 85.3 74.1 0.87
q21 183.7 177.8 0.97
q22 299.5 231.2 0.77
q23 40.0 48.5 1.21 ⚠️
q24 12.7 13.4 1.05
q25 36.7 31.1 0.85
q26 10.7 11.2 1.04
q27 112.4 100.3 0.89
q28 1974.2 2129.4 1.08 ⚠️
q29 21.1 16.9 0.80
q30 102.5 95.6 0.93
q31 137.7 128.9 0.94
q32 598.8 620.8 1.04
q33 529.0 519.0 0.98
q34 628.8 639.9 1.02
q35 142.7 151.9 1.06 ⚠️
q36 12.4 11.4 0.92
q37 7.2 7.6 1.05
q38 9.4 8.4 0.90
q39 16.8 18.9 1.13 ⚠️
q40 5.7 6.2 1.09
q41 5.9 6.3 1.08
q42 5.3 6.3 1.19
total 7566 7576 1.001

geomean ratio: 0.966

@gatesn
gatesn changed the base branch from ngates/scan-native-pool to ngates/v2-scan-pool November 20, 2026 00:31
@gatesn
gatesn changed the base branch from ngates/v2-scan-pool to ngates/scan-plan-pool November 20, 2026 00:40
gatesn added a commit that referenced this pull request Nov 20, 2026
Replaces #11664's tree with its port onto #11663's split and open, keeping the
earlier history so the branch moves forward without a force push.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gatesn
gatesn changed the base branch from ngates/scan-plan-pool to ngates/scan-native-pool November 20, 2026 18:25
@gatesn gatesn changed the title Read DuckDB's V1 scans through scan pools Read DuckDB's V1 scans through split and open Nov 20, 2026
@gatesn
gatesn changed the base branch from ngates/scan-native-pool to ngates/datafusion-scan-pool November 20, 2026 19:00
@gatesn
gatesn added this pull request to stack #11706 November 20, 2026 19:00
@gatesn
gatesn force-pushed the ngates/duckdb-scan-pool branch 3 times, most recently from 975aa4f to 5c76944 Compare November 20, 2026 19:38
@gatesn gatesn changed the title Read DuckDB's V1 scans through split and open Read DuckDB's V1 scans through open Nov 20, 2026
@gatesn
gatesn force-pushed the ngates/duckdb-scan-pool branch 2 times, most recently from aafa512 to 0bf4d78 Compare November 20, 2026 19:46
gatesn and others added 2 commits November 20, 2026 14:48
A host reading an opened scan, such as DuckDB showing a progress bar per file, asks its stream how much of the work its clones have taken. A layout scan's stream reports the fraction of its splits claimed; a stream over partitions reports none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Each file DuckDB's MultiFileReader opens is opened once as a V1 layout scan,
and every DuckDB thread that joins the file polls a clone of its stream. A
thread joins a file until the first clone ends, which happens only once no
split is left to claim, so threads move on to the next file without waiting
for the last splits to finish. DuckDB's private per-file split list is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/duckdb-scan-pool branch from 0bf4d78 to cdb5fad Compare November 20, 2026 19:48
A ScanStream is Send but not Sync, so the file state that every thread scanning the file shares holds it behind a mutex, and each thread clones it there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 7 commits November 20, 2026 15:05
DuckDB runs one thread per core, and each polls its own clone of a file's scan stream. A clone that keeps several splits in flight claims splits that other threads could have run, so on a file with few splits, such as TPC-H's smaller tables, most threads find the queue empty. Capping each clone at one split matches DuckDB's own parallelism.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ates/duckdb-stream-3

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…o ngates/duckdb-stream-3

Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant