Skip to content

Expose Vortex files as Scan API sources over V2 plans - #11700

Draft
gatesn wants to merge 2 commits into
ngates/v2-plan-sourcefrom
ngates/v2-plan-file-sources
Draft

gatesn wants to merge 2 commits into
ngates/v2-plan-sourcefrom
ngates/v2-plan-file-sources

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Exposes Vortex files through PlanDataSource:

  • VortexFile::plan_data_source scans one file.
  • VortexFile::plan_child makes a file a child of any plan source.
  • MultiFileDataSource::build_plan scans many files. It opens the first file for the schema and every later one only when a scan reaches it.
  • layout_reader_data_source names the layout-reader path explicitly. data_source() and build() keep scanning on layout readers, so this PR changes no default.

Contracts to check:

  • Shared cache. Every scan of a file's plan source shares one ScanCache: the file's own with with_caching, otherwise one owned by the source.
  • Pruning. A scan whose filter the file statistics prove false reads nothing.
  • Opening is free. Opening a file's plan source reads nothing and builds no layout reader.
  • List layouts. The V2 scan cannot read list layouts yet, so build_plan falls back to build's layout-reader source whenever the schema could hold one. may_contain_list_layout is conservative: maps and variants count.

Stacked on #11699.

🤖 Generated with Claude Code

VortexFile::plan_data_source scans one file through PlanDataSource, with
the file's statistics pruning scans they rule out, and
MultiFileDataSource::build_plan scans many, opening each file after the
first as a scan reaches it. Both are opt-in: data_source() and build()
still scan with layout readers. build_plan falls back to build() when
the schema could hold a list layout, which the V2 scan cannot read yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant