Repository navigation
Stream ordered V2 DataFusion scans chunk by chunk - #11448
Draft
joseph-isaacs wants to merge 2 commits into
Draft
joseph-isaacs wants to merge 2 commits into
joseph-isaacs wants to merge 2 commits into
Conversation
An ordered V2 scan collected every batch of the partition and sorted them before yielding anything, so it held the whole partition decoded in memory and an ORDER BY ... LIMIT could not stop early. The chunk planner already spawns chunks in row order, so the driver now yields them in that order: `try_buffered` runs up to `in_flight` chunks ahead while the first is drained. A chunk's morsels can still finish out of order, so each chunk holds its own batches until it is done and sorts them by start row. An ordered scan never runs chunks inline, because inline chunks interleave on one run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBCdGzsRfdpWndH9q21 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
vortex-file's V2 ScanDriver had the same collect-and-sort as the DataFusion driver: an ordered scan held every batch until the scan was done. It now streams chunks in planning order through try_buffered, each chunk holding only its own batches until it is done, and never runs ordered chunks inline on the root run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9i8ce2ie4BaCQkHSc Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
force-pushed
the
claude/datafusion-scan-v2-m5iwn0
branch
from
November 16, 2026 14:58
1ae5933 to
9439ca4
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Plan(replacingLayoutReaders) #9062When output ordering is required, the DataFusion V2 scan driver collected every batch of the partition and sorted them by start row before yielding anything (
ScanDriver::run). An ordered scan therefore held the whole partition decoded in memory, emitted nothing until the partition was done, andORDER BY … LIMIT kcould not stop early because theLimitStreamsits after the collect. This is one of the blockers to making V2 the DataFusion default.Changes
vortex-datafusion/src/persistent/scan_v2.rs:ChunkPlanneralready spawns chunks in row order, so in ordered mode the driver keeps that order instead of re-sorting the partition:try_buffered(in_flight)spawns up toin_flightchunks ahead and yields their streams in order, andtry_flattendrains them one at a time. Memory is now bounded by the chunks in flight, not the partition.chunk_streamholds that chunk's batches, at most 16, until the chunk is done and sends them sorted by start row.in_flight <= 1), because inline chunks are separate roots of oneRunand their output interleaves. Within_flight <= 1they run one at a time on their own tasks.Test:
test_ordered_v2_scan_yields_chunks_in_row_orderinopener.rsscans a 300,000-row file (several chunks) with ordering atscan_concurrency0, 1 and 4, and checks the output equals the input in order. It covers the new ordered path, including concurrency 0, which used to run inline. It checks order only, not memory or early output: the old collect-and-sort passes it too.vortex-file/src/v2/scan.rs(second commit) had the same collect-and-sort and gets the same change: ordered chunks stream in planning order throughtry_buffered, each holding only its own batches, and never run inline. Test:ordered_scan_yields_chunks_in_row_orderinvortex-file/src/v2/scan/tests.rs, at concurrency 1 and 4. Like the DataFusion test, it checks order only.Known limits, unchanged here:
in_flightis stillconcurrency * cores; Fix and speed up the V2 scan's planner chain #10789 changes that toconcurrency, and whichever merges second must keep both that and&& !self.ordered.Checks
Rebased onto
develop(2026-11-16), no conflicts.cargo clippy -p vortex-file -p vortex-datafusion --all-targets --all-features -- -D warnings: cleancargo +nightly-2026-11-13 fmt -p vortex-file -p vortex-datafusioncargo nextest run -p vortex-file -p vortex-datafusion: 1193 passedGenerated by Claude Code