Skip to content

Read DataFusion VortexTable scans through split and open - #11666

Draft
gatesn wants to merge 32 commits into
ngates/scan-native-poolfrom
ngates/datafusion-scan-pool
Draft

gatesn wants to merge 32 commits into
ngates/scan-native-poolfrom
ngates/datafusion-scan-pool

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #11663.

DataFusion is a partitioned engine: it plans a fixed number of output partitions and drives each as its own stream. VortexTable maps a Vortex scan onto those partitions through the Scan API's DataSourceScan::open and DataSourceScan::split. Each execution of the DataSourceExec scans once, through DataFusion's sibling state (DataSource::create_sibling_state), and the first output partition polled runs scan(request). reset_state starts a fresh execution.

How the partitions divide the work follows DataFusion's own file scan. Its FileGroupPartitioner cuts files into fixed groups at planning, and FileScanConfig::create_sibling_state pools those groups into shared work unless order or partitioning must be preserved, in which case each sorted group declares the ordering and a SortPreservingMergeExec above merges them.

  • Unordered: the execution opens the scan once, and every output partition polls a clone of its ScanStream. The clones draw from one queue of work, so a partition that finishes early keeps reading, and per-partition row counts are unknown until the scan runs.
  • Ordered: for a declared ordering, or when DataFusion asks each partition to keep its rows in order (with_preserve_order), the execution calls split(N) once for the N output partitions, and output partition i reads fixed piece i in order. The pieces are contiguous, so every partition keeps the ordering and DataFusion merges them where it needs one global order. If the scan yields fewer pieces than partitions, the remaining partitions return no rows.
  • Fetch: a fetch bounds each output partition, and DataFusion's limit (or the merge's fetch) above the scan trims the total. DataFusion pushes a fetch after it has fixed the partitioning, so an ordered fetch reads up to the fetch from each piece.

Each partition's arrays become Arrow record batches through a new ArrowArrayStreamExt::to_arrow in vortex-arrow. It converts arrays as CPU tasks on the session runtime, with bounded concurrency, and keeps their order.

Per-partition statistics are not reported: shared work has no per-partition row count, and partition_statistics runs at plan time, before the execution scans.

VORTEX_SCAN_POOL=0 (or VortexDataSourceBuilder::with_split_scan(false)) instead has the output partitions take partitions from one shared partition stream, which keeps an ordered or limited scan in one output partition, for A/B comparison; the default is 1. The switch and that path go away when the Scan API drops partitions().

In every mode an output partition holds work no other partition reads: what its clone has taken, its piece, or the partitions it has taken. Dropping an output partition before it finishes therefore cancels the execution, and the other partitions report an error rather than finish without those rows.

Measured at c55c047 against develop 7591044: DataFusion VortexTable with the V1 scan engine, geomean of per-query time ratios (lower is better), from four rotations of five iterations in which every configuration runs twice, with a load gate before each query. "Split on vs off" compares VORTEX_SCAN_POOL=1 with =0; "develop vs develop" is the noise between two runs of develop:

suite this PR vs develop split on vs off develop vs develop
TPC-H SF1 0.998 1.002 1.006
ClickBench 0.987 1.000 0.983

No query is more than 5% slower in every rotation. The largest gaps against develop are ClickBench q19 (8.3 → 8.8 ms, 1.08) and q36 (1.06), and TPC-H q14 (1.02).

In the Scan API stack this PR maps a scan onto DataFusion's fixed output partitions, the partitioned counterpart to DuckDB's use of open in #11664. The roadmap is tracked in #9062.

🤖 Generated with Claude Code

@gatesn gatesn added the changelog/performance A performance improvement label Nov 19, 2026
@gatesn gatesn mentioned this pull request Nov 19, 2026
16 of 46 tasks
gatesn and others added 2 commits November 19, 2026 18:30
Layout scans now split on their layout splits and open into a pool
whose workers share those splits through a channel. Multi-file scans
queue an open for each deferred file; the worker that opens one sends
the file's splits back into the channel for any worker to take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Decoding a split inline kept each worker's splits on its own thread.
Spawning them onto the session's runtime, as V1 scans do, lets idle
runtime threads decode them and starts their I/O at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Benchmarks for the v2 VortexTable path: datafusion-bench --formats vortex with VORTEX_USE_SCAN_API=1, release_debug with -C target-cpu=native, TPC-H SF1 and ClickBench (100 partitioned files), on an 18-core Apple Silicon Mac. Develop is d368834 and this PR is 4bc6703. Each suite ran 3 interleaved rounds of develop then this PR, 5 iterations per round. The tables show the median over all rounds, dropping each round's first iteration. Other builds were running on the machine the whole time (load average ~25–30), so treat differences under ~10% as noise.

vx-bench run does not exercise this code. It passes --scan-v1 false, which routes Vortex through the persistent FileFormat path, and the v2 VortexTable only registers with --scan-v1 true plus VORTEX_USE_SCAN_API=1. My first vx-bench run of both builds was therefore an A/A run of identical code. It showed 1.02× (TPC-H) and 1.15× (ClickBench) geomean, with single tiny queries off by up to 3.8×, which gives a measure of the noise.

Headline: TPC-H geomean 0.43× (develop's shared partition stream was badly serialized: q01 went from 160 ms to 19 ms). ClickBench geomean 0.94×, total 1.04×.

Regressions over 5%, rechecked. I reran the flagged ClickBench queries over 4 rounds with alternating order, 8 iterations each. q04, q05, q07 and q08 came out at 0.84–0.99×, so those were noise. q21, q22 and q23 are real: 1.11×, 1.20× and 1.32× by median, 1.18–1.76× by min. On the base branch (#11663, same setup, 3 rounds), base matches develop, so this PR is the cause:

query develop (ms) base #11663 (ms) this PR (ms) PR / base
q01 6.0 5.9 6.5 1.10
q08 185.9 180.7 186.3 1.03
q21 159.0 150.6 168.0 1.12 ⚠️
q22 185.7 173.3 211.4 1.22 ⚠️
q23 176.6 157.3 218.1 1.39 ⚠️

Cause (q23, Samply over 20 iterations). This PR uses about 70% more user CPU than base for the same query (/usr/bin/time: 44 s vs 26 s over 15 iterations), so the cost is extra work, not just lost parallelism. A third of the CPU samples spin in dashmap::lock::RawRwLock::lock_exclusive_slow, called from SharedSegmentSource::request (vortex-io/src/segments/shared.rs) via FlatReader::array_future under split_exec and pool::Shared::admit.

request takes an exclusive entry() lock on every segment request. With the pool, all 18 workers share one file's splits, and so one SharedSegmentSource, and SELECT * requests ~100 segments per split. On develop and base, output partitions spread across the 100 files, so that lock was never contended. The CPU-heavy LIKE '%google%' queries suffer most, and the machine load likely amplifies it (a preempted lock holder leaves the others spinning).

The likely fix sits below this PR: either a read-locked fast path in SharedSegmentSource::request, or handing out splits so that workers tend to stay on different files.

The scan's elapsed_compute now includes decoding, since workers decode inside the output partition's poll rather than on spawned tasks.

Full interleaved results (median ms)

tpch

query develop (ms) this PR (ms) ratio
q01 160.5 18.9 0.12 ✅
q02 19.0 10.1 0.53 ✅
q03 28.6 9.2 0.32 ✅
q04 21.9 7.4 0.34 ✅
q05 58.9 18.1 0.31 ✅
q06 5.0 3.5 0.70 ✅
q07 108.0 52.8 0.49 ✅
q08 51.2 13.6 0.27 ✅
q09 94.0 26.6 0.28 ✅
q10 36.7 15.7 0.43 ✅
q11 11.2 6.2 0.56 ✅
q12 17.1 8.6 0.50 ✅
q13 16.0 12.1 0.76 ✅
q14 8.2 5.2 0.63 ✅
q15 16.8 8.2 0.49 ✅
q16 12.9 8.8 0.68 ✅
q17 106.5 28.7 0.27 ✅
q18 151.8 52.2 0.34 ✅
q19 9.6 6.8 0.71 ✅
q20 20.7 10.5 0.51 ✅
q21 118.5 46.5 0.39 ✅
q22 11.8 6.6 0.56 ✅
geomean 0.427
sum 1085 376 0.347

clickbench

query develop (ms) this PR (ms) ratio
q00 3.3 3.3 1.01
q01 6.4 6.1 0.95 ✅
q02 14.8 13.7 0.93 ✅
q03 14.5 14.0 0.97
q04 136.1 150.2 1.10 ⚠️
q05 157.2 187.6 1.19 ⚠️
q06 3.8 3.8 1.01
q07 8.5 11.0 1.29 ⚠️
q08 201.4 250.3 1.24 ⚠️
q09 277.7 286.5 1.03
q10 32.4 35.3 1.09 ⚠️
q11 41.8 42.2 1.01
q12 145.1 151.8 1.05
q13 251.7 240.4 0.96
q14 174.6 169.7 0.97
q15 207.6 211.4 1.02
q16 419.7 439.3 1.05
q17 432.3 417.5 0.97
q18 771.4 802.9 1.04
q19 12.3 11.8 0.96
q20 115.0 121.0 1.05 ⚠️
q21 187.9 214.7 1.14 ⚠️
q22 224.7 272.4 1.21 ⚠️
q23 186.2 264.3 1.42 ⚠️
q24 12.7 13.8 1.09 ⚠️
q25 36.1 33.5 0.93 ✅
q26 23.0 22.0 0.96
q27 202.9 197.1 0.97
q28 1521.9 1494.6 0.98
q29 21.5 22.9 1.06 ⚠️
q30 119.9 128.7 1.07 ⚠️
q31 119.9 124.6 1.04
q32 581.0 598.2 1.03
q33 864.9 926.9 1.07 ⚠️
q34 875.2 945.9 1.08 ⚠️
q35 181.3 192.0 1.06 ⚠️
q36 43.9 11.8 0.27 ✅
q37 23.9 9.7 0.41 ✅
q38 15.7 6.8 0.43 ✅
q39 69.9 21.1 0.30 ✅
q40 5.7 5.2 0.90 ✅
q41 5.6 5.3 0.94 ✅
q42 6.0 5.2 0.86 ✅
geomean 0.938
sum 8757 9086 1.038

gatesn and others added 2 commits November 19, 2026 18:33
Each DataSourceExec execution now opens its Vortex scan into a pool whose
workers back the output partitions, or splits it into one piece per
partition when order matters or work stealing is disabled, replacing the
partition stream shared under a mutex. A fetch now bounds each of several
output partitions rather than forcing one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The layout file in the split-mode test had a single layout split, so
splitting it returned one piece. The test scan now splits natively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/datafusion-scan-pool branch from 4bc6703 to d135991 Compare November 19, 2026 23:34
gatesn and others added 3 commits November 19, 2026 18:37
Adds ArrowArrayStreamExt::to_arrow, which converts each array of a
stream to a RecordBatch as a CPU task on the session's runtime with
bounded, order-preserving concurrency. The v2 DataFusion source now
calls it on each worker instead of spawning conversions itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Benchmarking VortexTable needed both --scan-v1 true and
VORTEX_USE_SCAN_API=1, so it could not be compared with the listing table's
V2 scan in one run. --scan-api true registers Vortex tables through the Scan
API whatever --scan-v1 says; the environment variable keeps working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The v2 DataFusion source keeps develop's execution, output partitions
sharing the scan's partition stream with its single-partition fetch,
selectable at runtime so one binary can compare it with split and open.
VORTEX_SCAN_POOL defaults to split and open; the partition stream goes
away with the partition API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 7 commits November 19, 2026 19:19
Takes the base's version of every file outside vortex-datafusion,
vortex-arrow and the datafusion-bench flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
Replace the `VORTEX_USE_SCAN_API` environment variable in `datafusion-bench`
with a `--scan-api` flag, defined as #11666 defines it, and have the
orchestrator pass it for Scan API targets. PR benchmarks compare against the
benchmarks database rather than a base build, so the flag only needs to exist
on develop once this lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
Replace the `VORTEX_USE_SCAN_API` environment variable in `datafusion-bench`
with a `--scan-api` flag, defined as #11666 defines it, and have the
orchestrator pass it for Scan API targets. PR benchmarks compare against the
benchmarks database rather than a base build, so the flag only needs to exist
on develop once this lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 4 commits November 20, 2026 12:54
The bench's VORTEX_USE_SCAN_API fallback is dropped: develop's #11685 removed it, so only
--scan-api reaches VortexTable now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Split and open are now required on DataSourceScan, so the test scan keeps its own split and
gains an open over PartitionPool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn changed the title Read DataFusion v2 scans through split and open Read DataFusion VortexTable scans through split and open Nov 20, 2026
@gatesn
gatesn added this pull request to stack #11706 November 20, 2026 18:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 4 commits November 20, 2026 14:02
DataFusion runs output partitions independently, so output partition i of n now splits the
scan into at most n pieces and reads piece i with one worker. Partitions share no state, which
keeps them correct when they run in separate processes. VORTEX_SCAN_POOL=0 keeps the shared
partition stream for comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered source may have the pieces of one split share a queue of work, so the pieces must
come from a single split call. The first output partition polled scans and splits through the
execution's sibling state, and output partition i reads piece i.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 9 commits November 20, 2026 14:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/bench-scan Run scan benchmarks label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-scan Run scan benchmarks label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant