Skip to content

Preserve ordered Scan API partitions in the v2 DataFusion adapter - #10062

Merged
gatesn merged 3 commits into
developfrom
claude/issue-10056-20260925-1405
Oct 26, 2026
Merged

gatesn merged 3 commits into
developfrom
claude/issue-10056-20260925-1405

Conversation

@vortex-claude

@vortex-claude vortex-claude Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Ordered scans must emit partitions in source order even when later partitions are ready first. Concatenate ordered partition streams while preserving bounded unordered concurrency and the existing global fetch trimming.

Closes #10056.

Deterministic regressions cover complete output, fetch zero, the first row, and a fetch crossing partition/batch boundaries. Three cases fail with unordered flattening; all four pass with the fix. The combined v2 module passes 17 tests; scoped Clippy and pinned formatting pass.

AI assistance: Claude Code and Codex, under @gatesn's direction.

@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 20 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

✅ 2439 untouched benchmarks
⏩ 12 skipped benchmarks1


Comparing claude/issue-10056-20260925-1405 (4b3ade5) with develop (33675d9)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Oct 10, 2026
@robert3005
robert3005 force-pushed the claude/issue-10056-20260925-1405 branch from f429e9e to 4c62dae Compare October 12, 2026 09:51
@robert3005 robert3005 added changelog/fix A bug fix and removed stale This PR is stale and will be auto-closed soon labels Oct 12, 2026
github-actions Bot and others added 2 commits October 26, 2026 12:59
VortexDataSource::open consumed partitions with try_flatten_unordered even
when ordering was requested.

- When ordering is requested, flatten split streams sequentially so results
  are emitted in partition order; otherwise retain concurrent consumption.
- The existing LimitStream trims the combined output to the fetch limit.
- Add a regression test with a delayed first partition that asserts ordered
  output and a global fetch of 1.

Co-authored-by: Nicholas Gates <gatesn@users.noreply.github.com>
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
…undaries

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn marked this pull request as draft October 26, 2026 17:16
@gatesn
gatesn force-pushed the claude/issue-10056-20260925-1405 branch from 4c62dae to 4b3ade5 Compare October 26, 2026 17:16
@gatesn gatesn changed the title vortex-datafusion v2 scan should preserve ordered splits and enforce fetch across splits Preserve ordered Scan API partitions in the v2 DataFusion adapter Oct 26, 2026
@gatesn gatesn self-assigned this Oct 26, 2026
@gatesn
gatesn marked this pull request as ready for review October 26, 2026 19:42
@gatesn
gatesn enabled auto-merge (squash) October 26, 2026 19:42
@gatesn
gatesn merged commit 967e2f5 into develop Oct 26, 2026
90 checks passed
@gatesn
gatesn deleted the claude/issue-10056-20260925-1405 branch October 26, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vortex-datafusion v2 scan should preserve ordered splits and enforce fetch across splits

2 participants