Skip to content

Run layout and plan scans through one work pool - #11678

Closed
gatesn wants to merge 13 commits into
ngates/v2-scan-poolfrom
ngates/scan-plan-pool
Closed

gatesn wants to merge 13 commits into
ngates/v2-scan-poolfrom
ngates/scan-plan-pool

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

The layout scan engine (V1) and the plan scan engine (V2) each had their own pool behind the Scan API's open, with the same shape. This replaces both with one generic pool, vortex_scan::pool::open_work_pool, so each engine is only a producer of WorkUnits: a unit either runs and returns arrays, or expands into more units in its place. Opening a deferred file, expanding a manifest entry, and a dynamic-filter root that prunes before cutting parts are all expansions, so "work creates work" lives in one place. The units are boxed closures, not a trait, and V1's copies of RowLimit and group_contiguous are gone.

The pool follows #11663's model, in which workers share the output, not the work. When an unordered pool's first worker starts, the pool spawns lanes on the session runtime: four per core, capped by split_concurrency. The lanes take units and expansions from one queue. At most expand_concurrency expansions run at once, using V1's slots that pass on when dropped. Lanes trim their rows to a trimmed row limit and send them into one bounded output channel that every worker reads. A V2 part is a streaming unit: its lane runs it and passes each chunk on as it finishes, as develop's DuckDB V2 threads do, so a top-N tightens its bound after the first chunk. Unordered V2 opens one child per core at once, as V1 opens files. There is no per-worker bound. If a unit fails, every lane stops. Dropping the pool aborts the lanes, and the pool counts as exhausted once the output channel is closed and empty. Ordered pools still give every unit, in row order, to the first worker. V1 and V2 behaviour is otherwise unchanged, with two small differences: trimmed output drops empty arrays, and an ordered V2 pool whose limit is spent reports progress 1.0.

Stacked on #11665.

🤖 Generated with Claude Code

Both scan engines had their own channel pool with the same shape. Move
the channel, expansion slots, ordered first-worker handoff, in-flight
bound, row limit, progress and exhaustion into one pool in
vortex_scan::pool, which V1 layout splits and V2 plan units feed as
work units. Expanding a unit yields more units in its place, so opening
a deferred file, expanding a manifest entry and pruning before cutting
parts are all expansions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the changelog/chore A trivial change label Nov 20, 2026
gatesn and others added 4 commits November 19, 2026 20:19
The generic work pool takes the foundation's lanes model for unordered
pools: once the first worker starts, `in_flight` lanes on the session's
runtime run units, expansions included under the same expand bound, and
send their trimmed rows into one bounded output channel that every
worker reads. The per-worker in-flight bound and worker hoarding are
gone; V1 and V2 both inherit this through the shared pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered pool's lane now runs a V2 part itself and passes each chunk
on as it finishes, as a DuckDB thread of develop's V2 scan does, instead
of spawning the part and holding all its chunks until it is done. A
top-N then tightens its bound after the first chunk, not the first part,
and a part's chunks no longer sit in memory together. Ordered pools keep
collecting parts in row order. An unordered V2 pool now opens one child
per core at once, as V1 opens files, rather than every child at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The generic work pool takes #11663's latest unordered model. An opened
file's splits, or a V2 child's parts, are one turn in the channel: a
lane takes the turn's next unit and queues the rest behind the other
turns, so lanes rotate over open files and children. Lanes start one per
ready unit, up to the cap, and none for an empty scan, and they run
splits and opens inline rather than spawning each. V1 files also recheck
whole-file pruning when their first split is taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn mentioned this pull request Nov 20, 2026
16 of 46 tasks
gatesn and others added 8 commits November 19, 2026 23:38
As in #11663, a lane takes a turn's next run of adjacent units and runs
them together, so their reads coalesce: a run is in_flight divided by
expand_concurrency, and the lanes hold in_flight units in all. V2 parts
are layout splits, as V1 splits are, so V2 takes runs of parts too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A lane decodes its run's units on its own task, so a single-file V1 scan,
whose one turn took every unit in flight, ran on one lane. Cap a run at
in_flight per core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Runs of adjacent units put a run of V2 parts on one lane's task, since a
part streams its chunks from its lane, and capped the lanes at in_flight
over the run: a table with few parts got a quarter of the lanes, each
decoding four parts on one task. V2 parts already coalesce their chunks'
reads within the part, so lanes take one part at a time, as before the
runs. V1 keeps its runs and its single-file fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Units that return their arrays at once, such as V1 splits, are taken in
runs of adjacent units; units that stream their arrays, such as V2
parts, are taken alone, one per lane. Once a pool queues units of the
first kind, its lanes are capped at in_flight over a run, so a pool
mixing both kinds still keeps in_flight units, and lanes above the cap
end. A skipped unit is an empty stream, so it never caps a pool of
streaming units. Pools of either kind alone behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
vortex-layout/src/scan/pool.rs keeps this branch's version. The work pool already carries #11663's 'Decode a single-file pool's splits on every core': a run is at most `in_flight` divided by the cores, and each unit of a run is spawned as its own task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the Scan API stack now puts each source's own tuned execution behind split/open (V1 on its split pipeline in #11663, V2 on its planner driver in #11665) instead of tuning one shared work pool for every engine. An engine-independent pool may come back once both sources behind split/open match develop. The pool-leave rule is dropped with it, since neither engine's own execution has it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant