Repository navigation
Conversation
DuckDB's V1 path now plans each file through `LayoutReaderDataSource` and claims one Scan API partition per thread, instead of building a V1 `ScanBuilder` itself. `LayoutReaderDataSource::with_layout_split_partitions` makes each layout split of the scanned fields its own partition. The scan is prepared once in `scan()`, and each partition polls its split future on the task that polls its stream, as DuckDB's threads did with the split futures before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Contributor
Author
|
Superseded by #11664, which reads DuckDB's file scans through the Scan API's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DuckDB's V1 scan now plans each file through the Scan API (
LayoutReaderDataSource) instead of building a V1ScanBuilderitself. This is the first step towards DuckDB reading anyDataSourcewithout a bespoke integration. DuckDB left the Scan API in #9383 to getMultiFileReader's features, and this keepsMultiFileReader.LayoutReaderDataSource::with_layout_split_partitions()prepares the scan once inscan()and makes each layout split a partition. A partition's stream polls its split future on the task that polls the stream. These are the same split futures DuckDB's threads claimed and polled before, so threading and I/O are unchanged. ThePartition::executedoc now allows a single-unit partition to do its work on the polling task. A scan with a limit keeps one whole-range partition, because split futures cannot share a limit.Differences from the old path: projection and filter are bound per file rather than once per query; each split costs one extra
block_onto see its stream end; a scan with an empty projection and no filter takes the source's length-only path (one whole-file constant array on one thread); V1 skips a file when all its partitions have an exact row count of zero, replacing the separatecan_skipcheck (V2 keeps it); andLayoutReaderDataSourcenow clampsrow_rangeto the file for every caller. Nothing was compiled, tested or benchmarked locally, so CI is the first build. The check that matters isduckdb-benchon V1 before and after.🤖 Generated with Claude Code