Skip to content

Run opened V2 plan scans on V2's own driver - #11703

Draft
gatesn wants to merge 2 commits into
ngates/v2-plan-splitfrom
ngates/v2-plan-open
Draft

gatesn wants to merge 2 commits into
ngates/v2-plan-splitfrom
ngates/v2-plan-open

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Gives PlanDataSource a pool of its own: open runs the scan on V2's planner driver, in the shape of vortex-duckdb/src/scan_v2.rs.

open prepares each ready child once and queues the child's parts. Each worker owns a planning Run, driven on the task that polls the worker. A worker takes parts from the shared queue, runs a part's chunks side by side, yields batches as they are produced, and awaits reads with poll_completion. A host thread therefore does its own CPU work, and nothing is spawned per part.

The part planner lives in vortex-scan as part_root so that hosts can share it.

Contracts to check:

  • Parts.
    • A part is the rows between a child's splits; without splits, it is a chunk of the scan.
    • A Vortex file's splits are its layout splits over the fields the scan reads, the same ones a V1 scan cuts on.
    • Each part prunes its own rows before its chunks.
  • Dynamic filters. With a dynamic filter, the root prunes the child as a whole first. That reads zone maps, so it runs as queued work rather than inside open.
  • Parts per worker. A worker runs one part at a time unless the request's split_concurrency raises that.
  • Deferred children. Expanding a deferred child is queued work, and it queues the work of the children it expands into.
  • Limits. All workers share one RowLimit.
  • Ordered scans. An ordered pool gives all the work to its first worker, which runs parts in row order.
  • Progress.
    • Each top-level child holds an equal share.
    • A deferred child divides its share among the children it expands into.
    • A prepared child divides its share among its parts by row count.

About half of the diff is vortex-file/src/plan_tests.rs. Those tests run real files through the session's I/O, with a reader that pends before each read completes.

Stacked on #11702.

🤖 Generated with Claude Code

A PartitionPool runs each partition of a PlanDataSource scan with its
own ScanDriver, which spends a driver, an I/O session and a task per
partition. open instead runs the scan as develop's V2 DuckDB scan does:
it prepares each child once and queues its parts, and each worker
drives a planning run of its own over them, on the task that polls it.
A file's parts follow its layout splits over the fields the scan reads,
as a V1 scan's do, and a dynamic filter prunes the whole child before
its parts are cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant