Skip to content

Read V1 layout scans through split and open - #11663

Draft
gatesn wants to merge 37 commits into
ngates/scan-split-openfrom
ngates/scan-native-pool
Draft

gatesn wants to merge 37 commits into
ngates/scan-split-openfrom
ngates/scan-native-pool

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Implements split and open for the V1 layout scans by putting the existing V1 execution paths behind them, so a host reading through the Scan API gets the same execution it gets when it drives those paths directly.

open()

  • A single-file scan without a limit or ordering opens into a ScanStream over SplitQueue, a queue of the split futures from RepeatedScan::execute.
    • Each clone claims splits from the queue and keeps up to the stream pipeline's per-thread count of them in flight: DEFAULT_SPLITS_PER_THREAD, capped by split_concurrency. It spawns them on the session runtime and yields each one as it finishes. A host that polls one clone per core therefore keeps as many splits in flight as the stream pipeline does.
    • A clone claims a split only when it has room for one, and yields only the splits it claimed, so no clone waits on another's split.
    • A clone with room for only one split (split_concurrency = 1) has nothing to overlap it with, so it polls that split itself instead of spawning it. The split then runs on the thread that consumes its array, as a DuckDB thread's split does with block_on.
  • An unordered multi-file scan without a limit opens into a stream whose clones claim files from the shared partition stream, which opens up to concurrency files ahead. Each clone reads two files at once, each file with its own split concurrency, so that one file's last splits overlap the next file's first. This matches the read-ahead of develop's DataFusion VortexTable path, whose partitions share one stream of files as these clones do.
  • A scan with a limit and an ordered scan open into ScanStream::from_partitions, which reads one partition at a time. Their partitions are unchanged.
    • Split futures cannot share a row limit, and a queue's workers finish splits out of row order. The partition pipeline handles both cases.

split(n)

  • Fixed contiguous pieces in row order, from metadata only, for every request. An unordered piece may return its own rows in any order.
    • A single file is cut at its layout's natural split boundaries (split_ranges) into pieces of about equal rows. A split is never cut in two.
    • A multi-file scan with at least n files is cut into runs of whole files, balanced by byte size when every file's size is known and by file count otherwise.
    • With fewer than n files, each file gets a share of the pieces in proportion to its size, at least one per file. An open file is cut at its split boundaries. A file that isn't open stays one piece, because cutting it would mean opening it, so the result may have fewer than n pieces.
  • With a limit: the scan stays whole, because the limit is one budget across all its rows. A multi-file scan that selects partitions also stays whole, because partition indices count from its first file.

Contracts to check:

  • Pieces together read every row exactly once, and ordered pieces read them in order.
  • Every split of an opened scan goes to exactly one clone.
  • split_concurrency bounds each partition being read, or each clone that reads a single split queue. A multi-file clone reads two partitions at once.

This PR changes no default read path: partitions() and the direct ScanBuilder paths are untouched, and split and open are new entry points.

This PR is the V1 engine behind the contract in #11662, on which it is stacked. #11666 (DataFusion) calls split, and #11664 (DuckDB) calls open. Part of #9062.

🤖 Generated with Claude Code

gatesn and others added 5 commits November 19, 2026 19:15
Layout scans now split on their layout splits and open into a pool
whose workers share those splits through a channel. Multi-file scans
queue an open for each deferred file; the worker that opens one sends
the file's splits back into the channel for any worker to take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Decoding a split inline kept each worker's splits on its own thread.
Spawning them onto the session's runtime, as V1 scans do, lets idle
runtime threads decode them and starts their I/O at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Pool splits are spawned onto the session runtime, so the test session needs
a runtime that runs them while test workers block on their streams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A benchmark of the pool should measure the pool, not a different way of
running splits. Pools now build a file's split tasks all at once when a
worker first takes one, so the file's reads register together and coalesce,
as execute_stream builds a partition's tasks up front. Scans with a filter
and a limit build each split lazily on a CPU task and optimize the trimmed
output on a second one, through the same helpers execute_stream now uses.

A worker keeps as many splits in flight as one partition stream does by
default (four per core, capped by split_concurrency), so a host with a
worker per core matches the partitions path's in-flight total. Deferred
files open on the session runtime, at most the source's open concurrency
at once, in both pool kinds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
An open slot now passes to the next deferred file whether the open finished
or its worker gave it up, so a dropped worker strands only the file it was
opening. A spent limit counts the unopened files as read, so progress ends
at 1.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/scan-native-pool branch from 540b8cc to c638db4 Compare November 20, 2026 00:17
gatesn and others added 4 commits November 19, 2026 19:25
When pool workers share one file's splits, many threads request segments
from one SharedSegmentSource. Each request took an exclusive shard lock and
held it across the inner request. Requests now look up an in-flight
request under a read lock, make the inner request outside any lock, and lock
exclusively only to record it, with the shard count scaled to the cores.
A requester that loses a race for the same segment drops its own request,
which reads the same bytes, so coalescing is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Workers of an unordered pool each kept their own window of splits, so with
fewer splits than workers × window the first workers took most of a scan
and the rest idled. The pool now runs its splits in lanes on the session
runtime, splits_in_flight of them across the whole pool, and every worker
takes rows from one bounded output channel. A worker is just one host
thread's share of the output, so the number of workers is only the host's
parallelism. Trimming to a filtered limit happens once, in the lanes, and
dropping a worker gives up nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 4 commits November 19, 2026 22:06
A lane is already a task on the session's runtime, so spawning each split and
each deferred-file open as a second task and awaiting it only added a spawn and
a cross-thread wake per unit. The ordered pool, whose units are polled by its
worker on the host thread, still spawns them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A pool checks a file's statistics against the filter when the file starts,
which for an opened scan is when the pool opens. A filter with dynamic parts,
such as a join's, may reject the file by the time its splits run, and
execute_stream checks when it starts reading. Check again, once per file, when
the first split is taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A pool started splits_in_flight lanes, four per core, when its first worker
started, and woke every one of them to end, however few units the scan had. A
TPC-H query scans several small single-file tables, so it paid that spawn and
teardown once per table. Lanes now start one for each unit that no waiting lane
will take, when the first worker starts and whenever a lane takes a unit with
more queued behind it, up to splits_in_flight.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered pool queued each opened file's splits one after another, so its
lanes read about one file at a time. A wide scan then waited on that file's read
latency, and every split of the file was started before the first rows could
tighten a dynamic filter. Each opened file is now one turn in the channel: a lane
takes the file's next split and queues the turn again behind the other files',
so the lanes read the open files' splits in turn. On ClickBench q23 through
DataFusion this cut wall time from 169 to 114 ms (develop: 139 ms) and scan
output rows from 886 to 400 (develop: 2,425); q22 returned to parity.

Lanes now grow with the splits and opens pending rather than the units in the
channel, since a file is a single unit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
The generic work pool takes #11663's latest unordered model. An opened
file's splits, or a V2 child's parts, are one turn in the channel: a
lane takes the turn's next unit and queues the rest behind the other
turns, so lanes rotate over open files and children. Lanes start one per
ready unit, up to the cap, and none for an empty scan, and they run
splits and opens inline rather than spawning each. V1 files also recheck
whole-file pruning when their first split is taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Reading opened files in turn handed each file's read driver one polled split
at a time, so adjacent splits no longer coalesced into shared reads, and short
filtered scans such as ClickBench q01 and q06 paid for many more object-store
reads. A file's turn now gives a lane a run of adjacent splits, its share of the
splits in flight (splits_in_flight / open_concurrency, four with the defaults),
which the lane polls together. The files still rotate, a run at a time, and the
lanes still hold splits_in_flight splits in all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
As in #11663, a lane takes a turn's next run of adjacent units and runs
them together, so their reads coalesce: a run is in_flight divided by
expand_concurrency, and the lanes hold in_flight units in all. V2 parts
are layout splits, as V1 splits are, so V2 takes runs of parts too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 2 commits November 19, 2026 23:55
A single-file scan opens one file at a time, so its run was every split in
flight and the pool started one lane, which polled the run's splits inline
and decoded them on one thread. A run is now at most one core's share of the
splits in flight, and each split of a run is spawned as `execute_stream`
spawns it: the run still starts its splits together, so their reads
coalesce, but they decode in parallel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
vortex-layout/src/scan/pool.rs keeps this branch's version. The work pool already carries #11663's 'Decode a single-file pool's splits on every core': a run is at most `in_flight` divided by the cores, and each unit of a run is spawned as its own task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 2 commits November 20, 2026 13:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Each host should get develop's execution unchanged, so this replaces the
shared layout pool with develop's own paths. A single-file scan opens into a
queue of its split futures from RepeatedScan::execute: a worker claims the
next split and runs it outside the lock, one at a time, which is how DuckDB's
threads read a file's splits on develop. A limited or ordered scan, and a
multi-file scan, open into a PartitionPool over their unchanged partition
pipeline. split cuts a file at its layout split boundaries, and a multi-file
scan into groups of whole files, reading metadata only.

The I/O and execute_stream changes that served the old pool are reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn changed the title Give layout scans a native split and open Read V1 layout scans through split and open Nov 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Clickbench on NVME (pr) has entries that did not report: clickbench-nvme. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=10 on NVME (pr) has entries that did not report: tpcds-nvme-10. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=1 on NVME (pr) has entries that did not report: tpcds-nvme. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark PolarSignals Profiling (pr) has entries that did not report: polarsignals. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark FineWeb NVMe (pr) has entries that did not report: fineweb. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on S3 (pr) has entries that did not report: tpch-s3. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=10 on NVME (pr) has entries that did not report: tpch-nvme-10. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on NVME (pr) has entries that did not report: tpch-nvme. Check the workflow run for details.

gatesn and others added 3 commits November 20, 2026 13:41
… files

An opened single-file scan's workers each keep the stream pipeline's
per-thread count of splits in flight, spawned on the session runtime, so a
host polling one worker per core keeps as many in flight as the stream does.
A worker yields only the splits it claimed, so no worker waits on another's.

A multi-file scan split into more pieces than it has files cuts its open
files at their layout split boundaries, sharing the pieces by row count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A multi-file scan splits into runs of whole files only, so the path stays
small until multi-file reading moves to a composite over child sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn added this pull request to stack #11706 November 20, 2026 18:52
gatesn and others added 7 commits November 20, 2026 13:53
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Layout scans open into a ScanStream over SplitPool. Unordered scans split into
shared clones; ordered multi-file scans no longer cut unopened files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 8 commits November 20, 2026 14:37
Rename SplitPool to SplitQueue to match the ScanStream API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant