Repository navigation
Conversation
PlanDataSource exposes any source that lowers its data to plans over placed segments, such as a Vortex file, through the Scan API: one partition per child, each run with a ScanDriver, and deferred children that a scan expands as it reaches them. Splitting and a pool of its own follow in later changes; until then a scan is one piece and opens into a PartitionPool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
PlanDataSource, a Scan API source over V2 plans. It is not tied to one format. A ready child (PlanChild) is aScanInput, a plan over placed segments, together with what is known about it without reading payloads: its row count and a pruner. A deferred child (DeferredPlanChild) expands into further children only when a scan reaches it, so a file to open or a manifest entry costs nothing until it is read. With this split, any store that lowers its data to plans can serve any Scan API host.A scan has one partition per child, and each partition runs on a
ScanDriver.splitreturns the scan as one piece, andopenreads its partitions throughPartitionPool. #11702 and #11703 give the source its ownsplitand pool.Contracts to check:
row_rangeandselectionare in each child's own rows.partition_rangeandpartition_selectionselect children.RowLimit.split_concurrencycaps the chunks a partition'sScanDriverdecodes at once (ScanDriver::with_max_in_flight).Stacked on #11663.
🤖 Generated with Claude Code