Repository navigation
Conversation
Both scan engines had their own channel pool with the same shape. Move the channel, expansion slots, ordered first-worker handoff, in-flight bound, row limit, progress and exhaustion into one pool in vortex_scan::pool, which V1 layout splits and V2 plan units feed as work units. Expanding a unit yields more units in its place, so opening a deferred file, expanding a manifest entry and pruning before cutting parts are all expansions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The generic work pool takes the foundation's lanes model for unordered pools: once the first worker starts, `in_flight` lanes on the session's runtime run units, expansions included under the same expand bound, and send their trimmed rows into one bounded output channel that every worker reads. The per-worker in-flight bound and worker hoarding are gone; V1 and V2 both inherit this through the shared pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered pool's lane now runs a V2 part itself and passes each chunk on as it finishes, as a DuckDB thread of develop's V2 scan does, instead of spawning the part and holding all its chunks until it is done. A top-N then tightens its bound after the first chunk, not the first part, and a part's chunks no longer sit in memory together. Ordered pools keep collecting parts in row order. An unordered V2 pool now opens one child per core at once, as V1 opens files, rather than every child at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The generic work pool takes #11663's latest unordered model. An opened file's splits, or a V2 child's parts, are one turn in the channel: a lane takes the turn's next unit and queues the rest behind the other turns, so lanes rotate over open files and children. Lanes start one per ready unit, up to the cap, and none for an empty scan, and they run splits and opens inline rather than spawning each. V1 files also recheck whole-file pruning when their first split is taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
16 of 46 tasks
As in #11663, a lane takes a turn's next run of adjacent units and runs them together, so their reads coalesce: a run is in_flight divided by expand_concurrency, and the lanes hold in_flight units in all. V2 parts are layout splits, as V1 splits are, so V2 takes runs of parts too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A lane decodes its run's units on its own task, so a single-file V1 scan, whose one turn took every unit in flight, ran on one lane. Cap a run at in_flight per core. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Runs of adjacent units put a run of V2 parts on one lane's task, since a part streams its chunks from its lane, and capped the lanes at in_flight over the run: a table with few parts got a quarter of the lanes, each decoding four parts on one task. V2 parts already coalesce their chunks' reads within the part, so lanes take one part at a time, as before the runs. V1 keeps its runs and its single-file fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Units that return their arrays at once, such as V1 splits, are taken in runs of adjacent units; units that stream their arrays, such as V2 parts, are taken alone, one per lane. Once a pool queues units of the first kind, its lanes are capped at in_flight over a run, so a pool mixing both kinds still keeps in_flight units, and lanes above the cap end. A skipped unit is an empty stream, so it never caps a pool of streaming units. Pools of either kind alone behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
vortex-layout/src/scan/pool.rs keeps this branch's version. The work pool already carries #11663's 'Decode a single-file pool's splits on every core': a run is at most `in_flight` divided by the cores, and each unit of a run is spawned as its own task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Contributor
Author
|
Closing: the Scan API stack now puts each source's own tuned execution behind |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The layout scan engine (V1) and the plan scan engine (V2) each had their own pool behind the Scan API's
open, with the same shape. This replaces both with one generic pool,vortex_scan::pool::open_work_pool, so each engine is only a producer ofWorkUnits: a unit either runs and returns arrays, or expands into more units in its place. Opening a deferred file, expanding a manifest entry, and a dynamic-filter root that prunes before cutting parts are all expansions, so "work creates work" lives in one place. The units are boxed closures, not a trait, and V1's copies ofRowLimitandgroup_contiguousare gone.The pool follows #11663's model, in which workers share the output, not the work. When an unordered pool's first worker starts, the pool spawns lanes on the session runtime: four per core, capped by
split_concurrency. The lanes take units and expansions from one queue. At mostexpand_concurrencyexpansions run at once, using V1's slots that pass on when dropped. Lanes trim their rows to a trimmed row limit and send them into one bounded output channel that every worker reads. A V2 part is a streaming unit: its lane runs it and passes each chunk on as it finishes, as develop's DuckDB V2 threads do, so a top-N tightens its bound after the first chunk. Unordered V2 opens one child per core at once, as V1 opens files. There is no per-worker bound. If a unit fails, every lane stops. Dropping the pool aborts the lanes, and the pool counts as exhausted once the output channel is closed and empty. Ordered pools still give every unit, in row order, to the first worker. V1 and V2 behaviour is otherwise unchanged, with two small differences: trimmed output drops empty arrays, and an ordered V2 pool whose limit is spent reports progress 1.0.Stacked on #11665.
🤖 Generated with Claude Code