Repository navigation
Conversation
Layout scans now split on their layout splits and open into a pool whose workers share those splits through a channel. Multi-file scans queue an open for each deferred file; the worker that opens one sends the file's splits back into the channel for any worker to take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Decoding a split inline kept each worker's splits on its own thread. Spawning them onto the session's runtime, as V1 scans do, lets idle runtime threads decode them and starts their I/O at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
|
Benchmarks for the v2
Headline: TPC-H geomean 0.43× (develop's shared partition stream was badly serialized: q01 went from 160 ms to 19 ms). ClickBench geomean 0.94×, total 1.04×. Regressions over 5%, rechecked. I reran the flagged ClickBench queries over 4 rounds with alternating order, 8 iterations each. q04, q05, q07 and q08 came out at 0.84–0.99×, so those were noise. q21, q22 and q23 are real: 1.11×, 1.20× and 1.32× by median, 1.18–1.76× by min. On the base branch (#11663, same setup, 3 rounds), base matches develop, so this PR is the cause:
Cause (q23, Samply over 20 iterations). This PR uses about 70% more user CPU than base for the same query (
The likely fix sits below this PR: either a read-locked fast path in The scan's Full interleaved results (median ms)tpch
clickbench
|
Each DataSourceExec execution now opens its Vortex scan into a pool whose workers back the output partitions, or splits it into one piece per partition when order matters or work stealing is disabled, replacing the partition stream shared under a mutex. A fetch now bounds each of several output partitions rather than forcing one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The layout file in the split-mode test had a single layout split, so splitting it returned one piece. The test scan now splits natively. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
4bc6703 to
d135991
Compare
Adds ArrowArrayStreamExt::to_arrow, which converts each array of a stream to a RecordBatch as a CPU task on the session's runtime with bounded, order-preserving concurrency. The v2 DataFusion source now calls it on each worker instead of spawning conversions itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Benchmarking VortexTable needed both --scan-v1 true and VORTEX_USE_SCAN_API=1, so it could not be compared with the listing table's V2 scan in one run. --scan-api true registers Vortex tables through the Scan API whatever --scan-v1 says; the environment variable keeps working. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The v2 DataFusion source keeps develop's execution, output partitions sharing the scan's partition stream with its single-partition fetch, selectable at runtime so one binary can compare it with split and open. VORTEX_SCAN_POOL defaults to split and open; the partition stream goes away with the partition API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Takes the base's version of every file outside vortex-datafusion, vortex-arrow and the datafusion-bench flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Replace the `VORTEX_USE_SCAN_API` environment variable in `datafusion-bench` with a `--scan-api` flag, defined as #11666 defines it, and have the orchestrator pass it for Scan API targets. PR benchmarks compare against the benchmarks database rather than a base build, so the flag only needs to exist on develop once this lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Replace the `VORTEX_USE_SCAN_API` environment variable in `datafusion-bench` with a `--scan-api` flag, defined as #11666 defines it, and have the orchestrator pass it for Scan API targets. PR benchmarks compare against the benchmarks database rather than a base build, so the flag only needs to exist on develop once this lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The bench's VORTEX_USE_SCAN_API fallback is dropped: develop's #11685 removed it, so only --scan-api reaches VortexTable now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Split and open are now required on DataSourceScan, so the test scan keeps its own split and gains an open over PartitionPool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
DataFusion runs output partitions independently, so output partition i of n now splits the scan into at most n pieces and reads piece i with one worker. Partitions share no state, which keeps them correct when they run in separate processes. VORTEX_SCAN_POOL=0 keeps the shared partition stream for comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered source may have the pieces of one split share a queue of work, so the pieces must come from a single split call. The first output partition polled scans and splits through the execution's sibling state, and output partition i reads piece i. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ex into ngates/datafusion-scan-pool
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ex into ngates/datafusion-scan-pool
…ex into ngates/datafusion-scan-pool
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ex into ngates/datafusion-scan-pool
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Stacked on #11663.
DataFusion is a partitioned engine: it plans a fixed number of output partitions and drives each as its own stream.
VortexTablemaps a Vortex scan onto those partitions through the Scan API'sDataSourceScan::openandDataSourceScan::split. Each execution of theDataSourceExecscans once, through DataFusion's sibling state (DataSource::create_sibling_state), and the first output partition polled runsscan(request).reset_statestarts a fresh execution.How the partitions divide the work follows DataFusion's own file scan. Its
FileGroupPartitionercuts files into fixed groups at planning, andFileScanConfig::create_sibling_statepools those groups into shared work unless order or partitioning must be preserved, in which case each sorted group declares the ordering and aSortPreservingMergeExecabove merges them.ScanStream. The clones draw from one queue of work, so a partition that finishes early keeps reading, and per-partition row counts are unknown until the scan runs.with_preserve_order), the execution callssplit(N)once for the N output partitions, and output partitionireads fixed pieceiin order. The pieces are contiguous, so every partition keeps the ordering and DataFusion merges them where it needs one global order. If the scan yields fewer pieces than partitions, the remaining partitions return no rows.Each partition's arrays become Arrow record batches through a new
ArrowArrayStreamExt::to_arrowinvortex-arrow. It converts arrays as CPU tasks on the session runtime, with bounded concurrency, and keeps their order.Per-partition statistics are not reported: shared work has no per-partition row count, and
partition_statisticsruns at plan time, before the execution scans.VORTEX_SCAN_POOL=0(orVortexDataSourceBuilder::with_split_scan(false)) instead has the output partitions take partitions from one shared partition stream, which keeps an ordered or limited scan in one output partition, for A/B comparison; the default is1. The switch and that path go away when the Scan API dropspartitions().In every mode an output partition holds work no other partition reads: what its clone has taken, its piece, or the partitions it has taken. Dropping an output partition before it finishes therefore cancels the execution, and the other partitions report an error rather than finish without those rows.
Measured at c55c047 against develop 7591044: DataFusion
VortexTablewith the V1 scan engine, geomean of per-query time ratios (lower is better), from four rotations of five iterations in which every configuration runs twice, with a load gate before each query. "Split on vs off" comparesVORTEX_SCAN_POOL=1with=0; "develop vs develop" is the noise between two runs of develop:No query is more than 5% slower in every rotation. The largest gaps against develop are ClickBench q19 (8.3 → 8.8 ms, 1.08) and q36 (1.06), and TPC-H q14 (1.02).
In the Scan API stack this PR maps a scan onto DataFusion's fixed output partitions, the partitioned counterpart to DuckDB's use of
openin #11664. The roadmap is tracked in #9062.🤖 Generated with Claude Code