Skip to content

Experiment: open V2 plan sources into a ScanStream - #11715

Draft
gatesn wants to merge 4 commits into
ngates/scan-exp-a-part-rootfrom
ngates/scan-exp-a-plan-source
Draft

gatesn wants to merge 4 commits into
ngates/scan-exp-a-part-rootfrom
ngates/scan-exp-a-plan-source

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Part of #9062

Adds PlanDataSource, a Scan API DataSource over a list of V2 plan children, each a ScanInput. The source is format-agnostic: anything that lowers its data to plans over placed segments can expose itself through it, and every scan of a child shares the child's ScanCache. A request applies to each child: its row range and selection are in the child's own rows, partition_range and partition_selection pick children by ordinal, and a child's file-level statistics can prune it by the filter before any work is planned.

This PR establishes a rule for the Scan API: a source does the reads it needs to know a scan's units of work in the async scan, which the host awaits, and open and split stay synchronous and read nothing. Making an unordered scan prepares each child once and part_root cuts it into parts, at the child's own splits when it has them and at the scan's chunk boundaries otherwise. With a dynamic filter, such as a top-N's bound, the root first prunes the child as a whole against the bound as it stands, which reads zone maps, so that happens in scan too. open returns a ScanStream built with ScanStream::new(dtype, worker_factory) and only queues the parts, so no clone waits on another for its first work. Each part prunes again against the bound as it stands when a clone takes it. Each clone of the stream drives its own Run on the task that polls it, one part at a time, and spawns nothing, so a host thread that polls its clone with block_on runs the scan's compute itself, as V2's synchronous step did. Reads are awaited through Run::poll_completion. The stream reports progress as the fraction of the scan's rows whose parts have been taken, and 1.0 once every part has been taken.

An ordered open returns all rows in order through the first clone polled, reading one child after another on ScanDriver, so every clone stays in scan order. Later clones return nothing. A limit is one RowLimit shared by all clones. split returns the scan whole in this stack, and the scan's one partition reads its opened stream. A real V2 split is a later experiment.

Contracts to check: every row of every selected child is read exactly once across clones, and open performs no reads.

This PR is one layer of experiment A, which tests one decision about the Scan API: whether DataSourceScan::open(), a cloneable async ScanStream that each host thread polls, is enough for the V2 scan engine, or whether a host needs V2's synchronous step. The stack puts the V2 engine behind open() and moves DuckDB's V2 scan onto it. The gate is DuckDB V2 on TPC-H and ClickBench within noise of develop's DuckDB V2, measured on the last layer with rotated runs. If V2 cannot reach parity through the stream, the experiment stops and reports what the engine needs that ScanStream cannot express, with profile evidence, rather than tuning around it.

Stacked on #11714

🤖 Generated with Claude Code

PlanDataSource exposes anything that lowers its data to plans over
placed segments through the Scan API. Opening an unordered scan prepares
each child once and cuts it into parts with part_root, queued for every
clone of the returned ScanStream. Each clone drives a Run of its own on
the task that polls it, one part at a time, and spawns nothing. An
ordered scan runs in order on ScanDriver through the first clone polled.
split returns the scan whole.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
gatesn and others added 3 commits November 20, 2026 17:20
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dynamic filter's root reads zone maps before it cuts a child into
parts. Running it in the async scan, which the host awaits, means
opening only queues the parts, and no clone of the stream waits on
another for its first work. The scan's work is known before it is
opened, so the queue is a plain list.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant