Skip to content

Run DuckDB's V1 file scan through the Scan API - #11645

Closed
gatesn wants to merge 1 commit into
developfrom
ngates/duckdb-v1-scan-api
Closed

gatesn wants to merge 1 commit into
developfrom
ngates/duckdb-v1-scan-api

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

DuckDB's V1 scan now plans each file through the Scan API (LayoutReaderDataSource) instead of building a V1 ScanBuilder itself. This is the first step towards DuckDB reading any DataSource without a bespoke integration. DuckDB left the Scan API in #9383 to get MultiFileReader's features, and this keeps MultiFileReader.

LayoutReaderDataSource::with_layout_split_partitions() prepares the scan once in scan() and makes each layout split a partition. A partition's stream polls its split future on the task that polls the stream. These are the same split futures DuckDB's threads claimed and polled before, so threading and I/O are unchanged. The Partition::execute doc now allows a single-unit partition to do its work on the polling task. A scan with a limit keeps one whole-range partition, because split futures cannot share a limit.

Differences from the old path: projection and filter are bound per file rather than once per query; each split costs one extra block_on to see its stream end; a scan with an empty projection and no filter takes the source's length-only path (one whole-file constant array on one thread); V1 skips a file when all its partitions have an exact row count of zero, replacing the separate can_skip check (V2 keeps it); and LayoutReaderDataSource now clamps row_range to the file for every caller. Nothing was compiled, tested or benchmarked locally, so CI is the first build. The check that matters is duckdb-bench on V1 before and after.

🤖 Generated with Claude Code

DuckDB's V1 path now plans each file through `LayoutReaderDataSource` and
claims one Scan API partition per thread, instead of building a V1
`ScanBuilder` itself.

`LayoutReaderDataSource::with_layout_split_partitions` makes each layout
split of the scanned fields its own partition. The scan is prepared once
in `scan()`, and each partition polls its split future on the task that
polls its stream, as DuckDB's threads did with the split futures before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #11664, which reads DuckDB's file scans through the Scan API's split/open pools instead of a private queue over partitions.

@gatesn gatesn closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant