Repository navigation
Conversation
847dab6 to
43ab343
Compare
A scan can now be split into independent pieces, or opened into a pool that host threads share by each polling a worker. Both have defaults on top of partitions, so every existing source supports them; sources and hosts move over in later changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
f225998 to
28163bf
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Settle the ScanPool trait before hosts build on it: progress() and is_exhausted() let a host draw a progress bar and stop starting workers on a drained pool. Both have safe defaults. DataSourceScan gains plan-time row_count/byte_size estimates, and PartitionPool becomes public so a scan that cannot split can still open over its partitions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A worker is one host thread's share of a pool's output, while the pool's work runs as tasks on the session's runtime. A host whose runtime has no threads of its own must drive it while waiting on a worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A source now puts its own tuned execution behind split and open instead of tuning one shared pool to fit every engine. partitions() gets a default derived from them: one partition per split piece, each draining one worker of the piece's opened pool. PartitionPool stays as the stopgap open for a source that still implements partitions(), which is what every in-repo scan does for now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Takes the reworked #11663, which reads V1 layout scans through split and open on develop's pipeline and includes #11662's required split and open. The V1 layout files are #11663's as they stand; vortex-scan's pool keeps V2's RowLimit, trim_to_limit and group_contiguous beside them. The resolution is the tree of #11665's commits replayed on the new base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
A host reading several pools, one per file of a table, moves threads to the next pool once the current one is exhausted, so is_exhausted must turn true once every unit is taken, not once the work finishes. PartitionPool reports it as soon as it has taken an exact partition count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
DataSourceScan::row_count and byte_size and ScanPool::progress and is_exhausted have no caller yet. Each comes back with the host that calls it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Every source in the stack implements partitions, and no host reads the derived default, so partitions and partition_count are required as on develop. A worker's stream already carries its dtype, so the pool does not repeat it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
This is the first of four PRs that give every host one way to read a scan, and move two hosts onto it: #11662 (the trait), #11663 (V1 layout sources), #11666 (DataFusion
VortexTable) and #11664 (DuckDB). Part of #9062.What the stack does. Today a scan hands out partitions, which do two jobs: they are the unit a host declares to its engine, and the unit a thread runs. Each host works around that privately: DuckDB with its own V1 and V2 work queues, DataFusion with a shared partition stream. This stack replaces those workarounds with two methods on
DataSourceScan.split(n)partitions a scan into at mostnfixed pieces from metadata only, each owning its rows, for a host that needs partitions.open()returns a stream that can be cloned, where each clone is another worker on the same remaining work, for a host that owns threads or wants work stealing. A host keeps its own execution model and only states properties on the request (ordered,limit); the source decides the rest. Hosts that want work stealing callopen()and give each thread or partition a clone; hosts that need partitions callsplit. When the request is ordered, the pieces are contiguous and in scan order.Why it is worth having. It is the seam the V2 scan engine plugs into: V2 implements the same two methods, and no host changes. Each host is written once against
split/openand each source once behind them, which is what makes a composite multi-file source or an Iceberg table reachable from every engine. The stack removes DuckDB's private V1 queue and DataFusion's shared partition stream, and it letsScanBuilderlater become private to the V1 source and partitions be removed. Every method added has a caller in the stack: DataFusion callssplit(n)for ordered scans andopen()for unordered ones, following its own file scan's rule (fixed file groups at planning, a shared work queue at execution unless order must be preserved); DuckDB callsopen(). Public language bindings are unchanged.Performance. The target is parity with develop, measured locally for each host on its existing benchmark. The table shows each host's time relative to develop; the PR that carries each full table is linked.
VortexTable(#11666)VortexTable(#11666)Each figure is the geometric mean over the suite's queries, from runs that rotate the builds through every position. The same develop build measured against itself in those runs lands between 0.98 and 1.01.
Caveat. This is a start, not the final shape. We expect the API to change as more engines move over and as the V2 scan engine lands behind it. Known gaps:
splitoropenyet. They map a byte range to rows on their own and read through the older builder and partition APIs.split.split_concurrencyshould become a per-worker memory limit.This PR is the contract. It adds
splitandopentoDataSourceScanso that each host reads every scan the same way, whatever the source: a host's execution model is fixed. A host that needs a fixed number of partitions callssplit(n)and reads one piece per partition: DataFusion'snis its output partition count, and a host such as Spark would choosenitself from its target split size. A host that runs its own threads (DuckDB), or that wants its partitions to share work, never splits: it callsopen()once and has each thread or partition poll a clone of the returned stream. The host states what it needs on theScanRequest(ordered,limit), and the source decides what the pieces are and how they run.open()returns aScanStream(in the newvortex_scan::streammodule). This is aSend + Clonestream of arrays that carries the scan's dtype. A clone is another worker over the same remaining work, like a clone of a channel receiver: together the clones return every row once, and the number of clones a host polls sets its parallelism. A source builds one from a worker factory withScanStream::new. A source with no worker queue of its own usesScanStream::from_partitions, whose clones take the scan's partitions one at a time. Every in-repo scan usesfrom_partitionsin this PR.split(n)partitions the scan into at mostnfixed pieces, reading metadata only and never returning an empty list. Each piece owns its rows: together the pieces return every row exactly once. When the request is ordered, the pieces are contiguous and concatenate in scan order. A scan with no cut points returns itself as the one piece, which is what every in-repo scan does in this PR.splitcorresponds to DataFusion'sFileGroupPartitioner: static groups of files, with large files cut by range.The source owns its execution, so
splitandopenare required: each source puts its own tuned pipeline behind them, and a generic scheduler cannot know a source's units of work, read-ahead or ordering. Sharing lives only inopen, so a piece ofsplitis a real partition: its row count, statistics and order hold before it runs, and the contract does not depend on the request's order.Contracts to check:
ScanStreamnever starts threads. Its work may run as tasks on the session's runtime, which the host owns. A host whose runtime has no threads of its own must drive that runtime while it waits on a clone.ScanRequest::split_concurrencybounds each worker (a partition or a clone), not the whole scan.Breaking: a
DataSourceScanoutside this repository must implementsplitandopen. ReturningOk(vec![self])fromsplitandOk(ScanStream::from_partitions(self))fromopenkeeps its current behaviour.🤖 Generated with Claude Code