Repository navigation
Preserve ordered Scan API partitions in the v2 DataFusion adapter - #10062
Conversation
Merging this PR will not alter performance3 warnings, 1 note
Comparing Footnotes
|
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
f429e9e to
4c62dae
Compare
VortexDataSource::open consumed partitions with try_flatten_unordered even when ordering was requested. - When ordering is requested, flatten split streams sequentially so results are emitted in partition order; otherwise retain concurrent consumption. - The existing LimitStream trims the combined output to the fetch limit. - Add a regression test with a delayed first partition that asserts ordered output and a global fetch of 1. Co-authored-by: Nicholas Gates <gatesn@users.noreply.github.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
…undaries Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
4c62dae to
4b3ade5
Compare
Ordered scans must emit partitions in source order even when later partitions are ready first. Concatenate ordered partition streams while preserving bounded unordered concurrency and the existing global fetch trimming.
Closes #10056.
Deterministic regressions cover complete output, fetch zero, the first row, and a fetch crossing partition/batch boundaries. Three cases fail with unordered flattening; all four pass with the fix. The combined v2 module passes 17 tests; scoped Clippy and pinned formatting pass.
AI assistance: Claude Code and Codex, under @gatesn's direction.