Skip to content

Stream ordered V2 DataFusion scans chunk by chunk - #11448

Draft
joseph-isaacs wants to merge 2 commits into
developfrom
claude/datafusion-scan-v2-m5iwn0
Draft

joseph-isaacs wants to merge 2 commits into
developfrom
claude/datafusion-scan-v2-m5iwn0

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When output ordering is required, the DataFusion V2 scan driver collected every batch of the partition and sorted them by start row before yielding anything (ScanDriver::run). An ordered scan therefore held the whole partition decoded in memory, emitted nothing until the partition was done, and ORDER BY … LIMIT k could not stop early because the LimitStream sits after the collect. This is one of the blockers to making V2 the DataFusion default.

Changes

vortex-datafusion/src/persistent/scan_v2.rs:

  • ChunkPlanner already spawns chunks in row order, so in ordered mode the driver keeps that order instead of re-sorting the partition: try_buffered(in_flight) spawns up to in_flight chunks ahead and yields their streams in order, and try_flatten drains them one at a time. Memory is now bounded by the chunks in flight, not the partition.
  • A chunk's morsels can still finish out of order (a later morsel runs while an earlier one waits for reads), so in ordered mode chunk_stream holds that chunk's batches, at most 16, until the chunk is done and sends them sorted by start row.
  • Ordered scans never run chunks inline (in_flight <= 1), because inline chunks are separate roots of one Run and their output interleaves. With in_flight <= 1 they run one at a time on their own tasks.
  • Unordered scans are unchanged.
  • The module doc now says when chunks run inline. The old doc said "when the engine's partitions already fill the cores", which the code doesn't do.

Test: test_ordered_v2_scan_yields_chunks_in_row_order in opener.rs scans a 300,000-row file (several chunks) with ordering at scan_concurrency 0, 1 and 4, and checks the output equals the input in order. It covers the new ordered path, including concurrency 0, which used to run inline. It checks order only, not memory or early output: the old collect-and-sort passes it too.

vortex-file/src/v2/scan.rs (second commit) had the same collect-and-sort and gets the same change: ordered chunks stream in planning order through try_buffered, each holding only its own batches, and never run inline. Test: ordered_scan_yields_chunks_in_row_order in vortex-file/src/v2/scan/tests.rs, at concurrency 1 and 4. Like the DataFusion test, it checks order only.

Known limits, unchanged here:

  • A chunk yields nothing until all of its morsels are done, so first-batch latency is about one chunk.
  • Lookahead refills only once the head chunk is drained, so a slow head chunk lowers effective parallelism.
  • in_flight is still concurrency * cores; Fix and speed up the V2 scan's planner chain #10789 changes that to concurrency, and whichever merges second must keep both that and && !self.ordered.

Checks

Rebased onto develop (2026-11-16), no conflicts.

  • cargo clippy -p vortex-file -p vortex-datafusion --all-targets --all-features -- -D warnings: clean
  • cargo +nightly-2026-11-13 fmt -p vortex-file -p vortex-datafusion
  • cargo nextest run -p vortex-file -p vortex-datafusion: 1193 passed

Generated by Claude Code

joseph-isaacs and others added 2 commits November 16, 2026 14:54
An ordered V2 scan collected every batch of the partition and sorted them
before yielding anything, so it held the whole partition decoded in memory
and an ORDER BY ... LIMIT could not stop early.

The chunk planner already spawns chunks in row order, so the driver now
yields them in that order: `try_buffered` runs up to `in_flight` chunks
ahead while the first is drained. A chunk's morsels can still finish out
of order, so each chunk holds its own batches until it is done and sorts
them by start row. An ordered scan never runs chunks inline, because inline
chunks interleave on one run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBCdGzsRfdpWndH9q21
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
vortex-file's V2 ScanDriver had the same collect-and-sort as the
DataFusion driver: an ordered scan held every batch until the scan was
done. It now streams chunks in planning order through try_buffered, each
chunk holding only its own batches until it is done, and never runs
ordered chunks inline on the root run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9i8ce2ie4BaCQkHSc
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant