Skip to content

Add a Scan API source over V2 plans - #11699

Draft
gatesn wants to merge 2 commits into
ngates/scan-native-poolfrom
ngates/v2-plan-source
Draft

gatesn wants to merge 2 commits into
ngates/scan-native-poolfrom
ngates/v2-plan-source

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Adds PlanDataSource, a Scan API source over V2 plans. It is not tied to one format. A ready child (PlanChild) is a ScanInput, a plan over placed segments, together with what is known about it without reading payloads: its row count and a pruner. A deferred child (DeferredPlanChild) expands into further children only when a scan reaches it, so a file to open or a manifest entry costs nothing until it is read. With this split, any store that lowers its data to plans can serve any Scan API host.

A scan has one partition per child, and each partition runs on a ScanDriver. split returns the scan as one piece, and open reads its partitions through PartitionPool. #11702 and #11703 give the source its own split and pool.

Contracts to check:

  • Rows are per child. A request's row_range and selection are in each child's own rows.
  • Partition index is the child's ordinal. So partition_range and partition_selection select children.
  • Order. Children are read in order, and a deferred child's children are read in its place.
  • Dtype. A child that an expansion produces with a dtype other than the source's fails the scan.
  • Pruning. A child its pruner rules out yields no partition.
  • Limits.
    • Unordered partitions share one RowLimit.
    • Ordered partitions each apply the limit, and the caller trims the concatenated result.
  • In-flight chunks. split_concurrency caps the chunks a partition's ScanDriver decodes at once (ScanDriver::with_max_in_flight).

Stacked on #11663.

🤖 Generated with Claude Code

PlanDataSource exposes any source that lowers its data to plans over
placed segments, such as a Vortex file, through the Scan API: one
partition per child, each run with a ScanDriver, and deferred children
that a scan expands as it reaches them. Splitting and a pool of its own
follow in later changes; until then a scan is one piece and opens into a
PartitionPool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant