Skip to content

Split V2 Scan API scans into row-range partitions - #11649

Closed
gatesn wants to merge 2 commits into
ngates/plan-data-sourcefrom
ngates/scan-request-target-partitions
Closed

gatesn wants to merge 2 commits into
ngates/plan-data-sourcefrom
ngates/scan-request-target-partitions

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

A single file scanned through the Scan API ran as one partition, so DataFusion's VortexTable spread a one-file table over one output partition. PlanDataSource now splits an unordered, unlimited scan into contiguous row ranges, and the new ScanRequest::target_partitions lets the engine say how many it wants instead of the source guessing; DataFusion passes its target partitions, JNI and the C API pass None, which keeps the available-parallelism default.

Because the split count now depends on the request, split numbers cannot be stable partition indices, which the DataSource docs promise. A partition's index now names the unit of the source it reads (a PlanDataSource is one unit, index 0), partition_range/partition_selection select units, and the docs say that row_range and selection are in each unit's own row numbering and apply once across a unit's partitions.

Breaking: ScanRequest has a new public field, so struct literals without ..Default::default() must set it. No in-tree caller selects individual partitions of a single-file source.

Stacked on #11648.

🤖 Generated with Claude Code

A PlanDataSource scan had one partition, so engines that parallelise by
partition read a whole file on one worker. Unordered, unlimited scans now
split the row range into near-equal partitions, sized by the new
ScanRequest::target_partitions hint or the available parallelism, with at
most one partition per started 65,536 rows. DataFusion passes its target
partition count.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gatesn gatesn added the changelog/break A breaking API change label Nov 19, 2026
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gatesn

gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by split/open (#11662, #11665): row-range partitions re-planned and read boundary chunks twice, which splitting on the source's own units avoids. #11626 will move onto #11665.

@gatesn gatesn closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant