Repository navigation
Conversation
d862569 to
540b8cc
Compare
Layout scans now split on their layout splits and open into a pool whose workers share those splits through a channel. Multi-file scans queue an open for each deferred file; the worker that opens one sends the file's splits back into the channel for any worker to take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Decoding a split inline kept each worker's splits on its own thread. Spawning them onto the session's runtime, as V1 scans do, lets idle runtime threads decode them and starts their I/O at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Pool splits are spawned onto the session runtime, so the test session needs a runtime that runs them while test workers block on their streams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A benchmark of the pool should measure the pool, not a different way of running splits. Pools now build a file's split tasks all at once when a worker first takes one, so the file's reads register together and coalesce, as execute_stream builds a partition's tasks up front. Scans with a filter and a limit build each split lazily on a CPU task and optimize the trimmed output on a second one, through the same helpers execute_stream now uses. A worker keeps as many splits in flight as one partition stream does by default (four per core, capped by split_concurrency), so a host with a worker per core matches the partitions path's in-flight total. Deferred files open on the session runtime, at most the source's open concurrency at once, in both pool kinds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
An open slot now passes to the next deferred file whether the open finished or its worker gave it up, so a dropped worker strands only the file it was opening. A spent limit counts the unopened files as read, so progress ends at 1.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
540b8cc to
c638db4
Compare
When pool workers share one file's splits, many threads request segments from one SharedSegmentSource. Each request took an exclusive shard lock and held it across the inner request. Requests now look up an in-flight request under a read lock, make the inner request outside any lock, and lock exclusively only to record it, with the shard count scaled to the cores. A requester that loses a race for the same segment drops its own request, which reads the same bytes, so coalescing is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Workers of an unordered pool each kept their own window of splits, so with fewer splits than workers × window the first workers took most of a scan and the rest idled. The pool now runs its splits in lanes on the session runtime, splits_in_flight of them across the whole pool, and every worker takes rows from one bounded output channel. A worker is just one host thread's share of the output, so the number of workers is only the host's parallelism. Trimming to a filtered limit happens once, in the lanes, and dropping a worker gives up nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A lane is already a task on the session's runtime, so spawning each split and each deferred-file open as a second task and awaiting it only added a spawn and a cross-thread wake per unit. The ordered pool, whose units are polled by its worker on the host thread, still spawns them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A pool checks a file's statistics against the filter when the file starts, which for an opened scan is when the pool opens. A filter with dynamic parts, such as a join's, may reject the file by the time its splits run, and execute_stream checks when it starts reading. Check again, once per file, when the first split is taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A pool started splits_in_flight lanes, four per core, when its first worker started, and woke every one of them to end, however few units the scan had. A TPC-H query scans several small single-file tables, so it paid that spawn and teardown once per table. Lanes now start one for each unit that no waiting lane will take, when the first worker starts and whenever a lane takes a unit with more queued behind it, up to splits_in_flight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered pool queued each opened file's splits one after another, so its lanes read about one file at a time. A wide scan then waited on that file's read latency, and every split of the file was started before the first rows could tighten a dynamic filter. Each opened file is now one turn in the channel: a lane takes the file's next split and queues the turn again behind the other files', so the lanes read the open files' splits in turn. On ClickBench q23 through DataFusion this cut wall time from 169 to 114 ms (develop: 139 ms) and scan output rows from 886 to 400 (develop: 2,425); q22 returned to parity. Lanes now grow with the splits and opens pending rather than the units in the channel, since a file is a single unit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The generic work pool takes #11663's latest unordered model. An opened file's splits, or a V2 child's parts, are one turn in the channel: a lane takes the turn's next unit and queues the rest behind the other turns, so lanes rotate over open files and children. Lanes start one per ready unit, up to the cap, and none for an empty scan, and they run splits and opens inline rather than spawning each. V1 files also recheck whole-file pruning when their first split is taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Reading opened files in turn handed each file's read driver one polled split at a time, so adjacent splits no longer coalesced into shared reads, and short filtered scans such as ClickBench q01 and q06 paid for many more object-store reads. A file's turn now gives a lane a run of adjacent splits, its share of the splits in flight (splits_in_flight / open_concurrency, four with the defaults), which the lane polls together. The files still rotate, a run at a time, and the lanes still hold splits_in_flight splits in all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
As in #11663, a lane takes a turn's next run of adjacent units and runs them together, so their reads coalesce: a run is in_flight divided by expand_concurrency, and the lanes hold in_flight units in all. V2 parts are layout splits, as V1 splits are, so V2 takes runs of parts too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A single-file scan opens one file at a time, so its run was every split in flight and the pool started one lane, which polled the run's splits inline and decoded them on one thread. A run is now at most one core's share of the splits in flight, and each split of a run is spawned as `execute_stream` spawns it: the run still starts its splits together, so their reads coalesce, but they decode in parallel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
vortex-layout/src/scan/pool.rs keeps this branch's version. The work pool already carries #11663's 'Decode a single-file pool's splits on every core': a run is at most `in_flight` divided by the cores, and each unit of a run is spawned as its own task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Each host should get develop's execution unchanged, so this replaces the shared layout pool with develop's own paths. A single-file scan opens into a queue of its split futures from RepeatedScan::execute: a worker claims the next split and runs it outside the lock, one at a time, which is how DuckDB's threads read a file's splits on develop. A limited or ordered scan, and a multi-file scan, open into a PartitionPool over their unchanged partition pipeline. split cuts a file at its layout split boundaries, and a multi-file scan into groups of whole files, reading metadata only. The I/O and execute_stream changes that served the old pool are reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
… files An opened single-file scan's workers each keep the stream pipeline's per-thread count of splits in flight, spawned on the session runtime, so a host polling one worker per core keeps as many in flight as the stream does. A worker yields only the splits it claimed, so no worker waits on another's. A multi-file scan split into more pieces than it has files cuts its open files at their layout split boundaries, sharing the pieces by row count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A multi-file scan splits into runs of whole files only, so the path stays small until multi-file reading moves to a composite over child sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Layout scans open into a ScanStream over SplitPool. Unordered scans split into shared clones; ordered multi-file scans no longer cut unopened files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Rename SplitPool to SplitQueue to match the ScanStream API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Implements
splitandopenfor the V1 layout scans by putting the existing V1 execution paths behind them, so a host reading through the Scan API gets the same execution it gets when it drives those paths directly.open()ScanStreamoverSplitQueue, a queue of the split futures fromRepeatedScan::execute.DEFAULT_SPLITS_PER_THREAD, capped bysplit_concurrency. It spawns them on the session runtime and yields each one as it finishes. A host that polls one clone per core therefore keeps as many splits in flight as the stream pipeline does.split_concurrency= 1) has nothing to overlap it with, so it polls that split itself instead of spawning it. The split then runs on the thread that consumes its array, as a DuckDB thread's split does withblock_on.concurrencyfiles ahead. Each clone reads two files at once, each file with its own split concurrency, so that one file's last splits overlap the next file's first. This matches the read-ahead of develop's DataFusionVortexTablepath, whose partitions share one stream of files as these clones do.ScanStream::from_partitions, which reads one partition at a time. Their partitions are unchanged.split(n)split_ranges) into pieces of about equal rows. A split is never cut in two.nfiles is cut into runs of whole files, balanced by byte size when every file's size is known and by file count otherwise.nfiles, each file gets a share of the pieces in proportion to its size, at least one per file. An open file is cut at its split boundaries. A file that isn't open stays one piece, because cutting it would mean opening it, so the result may have fewer thannpieces.Contracts to check:
split_concurrencybounds each partition being read, or each clone that reads a single split queue. A multi-file clone reads two partitions at once.This PR changes no default read path:
partitions()and the directScanBuilderpaths are untouched, andsplitandopenare new entry points.This PR is the V1 engine behind the contract in #11662, on which it is stacked. #11666 (DataFusion) calls
split, and #11664 (DuckDB) callsopen. Part of #9062.🤖 Generated with Claude Code