Skip to content

Make split and open the methods every scan implements - #11662

Draft
gatesn wants to merge 16 commits into
developfrom
ngates/scan-split-open
Draft

gatesn wants to merge 16 commits into
developfrom
ngates/scan-split-open

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

This is the first of four PRs that give every host one way to read a scan, and move two hosts onto it: #11662 (the trait), #11663 (V1 layout sources), #11666 (DataFusion VortexTable) and #11664 (DuckDB). Part of #9062.

What the stack does. Today a scan hands out partitions, which do two jobs: they are the unit a host declares to its engine, and the unit a thread runs. Each host works around that privately: DuckDB with its own V1 and V2 work queues, DataFusion with a shared partition stream. This stack replaces those workarounds with two methods on DataSourceScan. split(n) partitions a scan into at most n fixed pieces from metadata only, each owning its rows, for a host that needs partitions. open() returns a stream that can be cloned, where each clone is another worker on the same remaining work, for a host that owns threads or wants work stealing. A host keeps its own execution model and only states properties on the request (ordered, limit); the source decides the rest. Hosts that want work stealing call open() and give each thread or partition a clone; hosts that need partitions call split. When the request is ordered, the pieces are contiguous and in scan order.

Why it is worth having. It is the seam the V2 scan engine plugs into: V2 implements the same two methods, and no host changes. Each host is written once against split/open and each source once behind them, which is what makes a composite multi-file source or an Iceberg table reachable from every engine. The stack removes DuckDB's private V1 queue and DataFusion's shared partition stream, and it lets ScanBuilder later become private to the V1 source and partitions be removed. Every method added has a caller in the stack: DataFusion calls split(n) for ordered scans and open() for unordered ones, following its own file scan's rule (fixed file groups at planning, a shared work queue at execution unless order must be preserved); DuckDB calls open(). Public language bindings are unchanged.

Performance. The target is parity with develop, measured locally for each host on its existing benchmark. The table shows each host's time relative to develop; the PR that carries each full table is linked.

Host Benchmark Relative to develop
DataFusion VortexTable (#11666) TPC-H 0.998
DataFusion VortexTable (#11666) ClickBench 0.987
DuckDB V1 (#11664) TPC-H 1.011
DuckDB V1 (#11664) ClickBench no suite figure; q9 and q14–q17 at parity
DuckDB V2 (#11664) TPC-H 1.002
DuckDB V2 (#11664) ClickBench 0.985

Each figure is the geometric mean over the suite's queries, from runs that rotate the builds through every position. The same develop build measured against itself in those runs lands between 0.98 and 1.01.

Caveat. This is a start, not the final shape. We expect the API to change as more engines move over and as the V2 scan engine lands behind it. Known gaps:

  • Hosts that cut files themselves by byte range, which are DataFusion's default file scan and Spark, do not call split or open yet. They map a byte range to rows on their own and read through the older builder and partition APIs.
  • A source made of lazily expanded children, such as an Iceberg table, needs a rule for when expansion happens relative to split.
  • split_concurrency should become a per-worker memory limit.
  • A host that runs its own threads gets inline polling today by asking for a split concurrency of 1, which needs a clearer way to say it.

This PR is the contract. It adds split and open to DataSourceScan so that each host reads every scan the same way, whatever the source: a host's execution model is fixed. A host that needs a fixed number of partitions calls split(n) and reads one piece per partition: DataFusion's n is its output partition count, and a host such as Spark would choose n itself from its target split size. A host that runs its own threads (DuckDB), or that wants its partitions to share work, never splits: it calls open() once and has each thread or partition poll a clone of the returned stream. The host states what it needs on the ScanRequest (ordered, limit), and the source decides what the pieces are and how they run.

open() returns a ScanStream (in the new vortex_scan::stream module). This is a Send + Clone stream of arrays that carries the scan's dtype. A clone is another worker over the same remaining work, like a clone of a channel receiver: together the clones return every row once, and the number of clones a host polls sets its parallelism. A source builds one from a worker factory with ScanStream::new. A source with no worker queue of its own uses ScanStream::from_partitions, whose clones take the scan's partitions one at a time. Every in-repo scan uses from_partitions in this PR.

split(n) partitions the scan into at most n fixed pieces, reading metadata only and never returning an empty list. Each piece owns its rows: together the pieces return every row exactly once. When the request is ordered, the pieces are contiguous and concatenate in scan order. A scan with no cut points returns itself as the one piece, which is what every in-repo scan does in this PR. split corresponds to DataFusion's FileGroupPartitioner: static groups of files, with large files cut by range.

The source owns its execution, so split and open are required: each source puts its own tuned pipeline behind them, and a generic scheduler cannot know a source's units of work, read-ahead or ordering. Sharing lives only in open, so a piece of split is a real partition: its row count, statistics and order hold before it runs, and the contract does not depend on the request's order.

Contracts to check:

  • A ScanStream never starts threads. Its work may run as tasks on the session's runtime, which the host owns. A host whose runtime has no threads of its own must drive that runtime while it waits on a clone.
  • Clones together return every row exactly once. Dropping a clone early gives up the work it holds.
  • ScanRequest::split_concurrency bounds each worker (a partition or a clone), not the whole scan.

Breaking: a DataSourceScan outside this repository must implement split and open. Returning Ok(vec![self]) from split and Ok(ScanStream::from_partitions(self)) from open keeps its current behaviour.

🤖 Generated with Claude Code

@gatesn
gatesn force-pushed the ngates/scan-split-open branch from 847dab6 to 43ab343 Compare November 19, 2026 20:39
A scan can now be split into independent pieces, or opened into a pool
that host threads share by each polling a worker. Both have defaults on
top of partitions, so every existing source supports them; sources and
hosts move over in later changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/scan-split-open branch from f225998 to 28163bf Compare November 19, 2026 20:47
@gatesn gatesn changed the title Replace Scan API partitions with split and open Add split and open to the Scan API Nov 19, 2026
gatesn and others added 5 commits November 19, 2026 18:30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Settle the ScanPool trait before hosts build on it: progress() and
is_exhausted() let a host draw a progress bar and stop starting workers on
a drained pool. Both have safe defaults. DataSourceScan gains plan-time
row_count/byte_size estimates, and PartitionPool becomes public so a scan
that cannot split can still open over its partitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A worker is one host thread's share of a pool's output, while the pool's
work runs as tasks on the session's runtime. A host whose runtime has no
threads of its own must drive it while waiting on a worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 2 commits November 20, 2026 12:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A source now puts its own tuned execution behind split and open instead of
tuning one shared pool to fit every engine. partitions() gets a default
derived from them: one partition per split piece, each draining one worker of
the piece's opened pool. PartitionPool stays as the stopgap open for a source
that still implements partitions(), which is what every in-repo scan does for
now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn changed the title Add split and open to the Scan API Make split and open the methods every scan implements Nov 20, 2026
@gatesn gatesn added the changelog/break A breaking API change label Nov 20, 2026
gatesn added a commit that referenced this pull request Nov 20, 2026
Takes the reworked #11663, which reads V1 layout scans through split and open on develop's pipeline and includes #11662's required split and open. The V1 layout files are #11663's as they stand; vortex-scan's pool keeps V2's RowLimit, trim_to_limit and group_contiguous beside them. The resolution is the tree of #11665's commits replayed on the new base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Nov 20, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 794ac0c clickbench-nvme 1 Explore Profiling Data
🟢 Done 794ac0c polarsignals 1 Explore Profiling Data
🟢 Done 794ac0c tpcds-nvme-10 1 Explore Profiling Data
🟢 Done 794ac0c tpch-nvme-10 1 Explore Profiling Data
🟢 Done 794ac0c fineweb 1 Explore Profiling Data
🟢 Done 794ac0c tpch-nvme 1 Explore Profiling Data
🟢 Done 794ac0c tpcds-nvme 1 Explore Profiling Data
🟢 Done 794ac0c tpch-s3 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=1 on NVME (pr) has entries that did not report: tpcds-nvme. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark FineWeb NVMe (pr) has entries that did not report: fineweb. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=10 on NVME (pr) has entries that did not report: tpch-nvme-10. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on S3 (pr) has entries that did not report: tpch-s3. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark PolarSignals Profiling (pr) has entries that did not report: polarsignals. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=10 on NVME (pr) has entries that did not report: tpcds-nvme-10. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on NVME (pr) has entries that did not report: tpch-nvme. Check the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Clickbench on NVME (pr) has entries that did not report: clickbench-nvme. Check the workflow run for details.

A host reading several pools, one per file of a table, moves threads to the
next pool once the current one is exhausted, so is_exhausted must turn true
once every unit is taken, not once the work finishes. PartitionPool reports
it as soon as it has taken an exact partition count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn added this pull request to stack #11706 November 20, 2026 18:52
gatesn and others added 7 commits November 20, 2026 13:53
DataSourceScan::row_count and byte_size and ScanPool::progress and
is_exhausted have no caller yet. Each comes back with the host that calls it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Every source in the stack implements partitions, and no host reads the
derived default, so partitions and partition_count are required as on
develop. A worker's stream already carries its dtype, so the pool does not
repeat it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant