Skip to content

Experiment: read DuckDB V2 scans through open - #11717

Draft
gatesn wants to merge 5 commits into
ngates/scan-exp-a-file-plan-sourcefrom
ngates/scan-exp-a-duckdb-v2
Draft

gatesn wants to merge 5 commits into
ngates/scan-exp-a-file-plan-sourcefrom
ngates/scan-exp-a-duckdb-v2

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Part of #9062

DuckDB's V2 scan reads through the Scan API the way its V1 scan does. For each file it builds the same ScanRequest as the V1 path, calls scan on the file's plan_data_source and opens it once, and each DuckDB thread polls a clone of the stream with RUNTIME.block_on(stream.next()). A thread joins a file until the first clone ends, and DuckDB's progress bar reads ScanStream::progress for both executors. Split::V2, FileScan and the private part queue in scan_v2.rs are gone, so DuckDB has one scan path whatever the executor. Files are opened with_caching so the V2 splitter reuses the layout reader DuckDB already builds. V2 batches that join several layout chunks are still exported chunk by chunk, so DuckDB evaluates expressions over each chunk's dictionary once.

This PR is one layer of experiment A, which tests one decision about the Scan API: whether DataSourceScan::open(), a cloneable async ScanStream that each host thread polls, is enough for the V2 scan engine, or whether a host needs V2's synchronous step. The stack puts the V2 engine behind open() and moves DuckDB's V2 scan onto it. The gate is DuckDB V2 on TPC-H and ClickBench within noise of develop's DuckDB V2, measured on the last layer with rotated runs. If V2 cannot reach parity through the stream, the experiment stops and reports what the engine needs that ScanStream cannot express, with profile evidence, rather than tuning around it. This is the layer the benchmark measures.

What the port shows about the trait, as findings:

  • ScanStream is not Sync, so DuckDB clones a file's stream under a mutex, for V2 as for V1.
  • A stream has no "work left" signal. A thread joins a file until the first clone ends, so a thread that arrives after the last part is taken polls a clone that ends at once: at most one wasted join per thread.
  • A source does the reads it needs to know its units of work in the async scan(), and open() and split() read nothing. For the V2 source that read is the whole-file prune of a dynamic filter, which therefore runs when DuckDB initializes a file. Breaking the rule is measurable: with that prune on the first poll of a clone, ClickBench q25, a top-N with a dynamic filter, runs 9.5% slower than develop, against 3% with it in scan().
  • ScanRequest::split_concurrency means nothing to V2: a clone always runs one part at a time.

Stacked on #11716

🤖 Generated with Claude Code

DuckDB's V2 scan builds the same ScanRequest as its V1 scan, scans the
file's plan data source and opens it, and each thread polls a clone of
the stream, as for V1. The private V2 queue in scan_v2.rs goes away, and
progress comes from ScanStream::progress for both executors.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@gatesn gatesn added the changelog/chore A trivial change label Nov 20, 2026
gatesn and others added 4 commits November 20, 2026 17:20
…b-v2

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…b-v2

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both executors now bind the projection against each file through
ScanRequest, so the bound copy in GlobalState and its helper go.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…b-v2

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant