Skip to content

Scan Vortex files on the V2 scan through the Scan API - #11648

Closed
gatesn wants to merge 2 commits into
ngates/scan-api-bench-flagfrom
ngates/plan-data-source
Closed

gatesn wants to merge 2 commits into
ngates/scan-api-bench-flagfrom
ngates/plan-data-source

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

The Scan API (DataSource) could only run Vortex files on the V1 layout reader; V2 was reachable only by engines that build its ScanBuilder themselves (the DataFusion file opener and DuckDB). PlanDataSource exposes any ScanInput as a DataSource whose partition runs on ScanDriver, so a format only has to produce a ScanInput to be scannable through the Scan API. VortexFile::data_source() now returns it, falling back to LayoutReaderDataSource for files whose layout tree has a list layout, which V2 cannot execute yet; plan_data_source() is always V2. As before, a scan whose filter the footer statistics prove false reads nothing.

Behaviour change: VortexFile::data_source() scans most files on V2. Its only in-tree caller is JNI's single-file source, which no benchmark suite exercises; multi-file sources (and so VortexTable) move in the last PR of this stack.

Stacked on #11647.

🤖 Generated with Claude Code

Engines that consume the Scan API still scanned Vortex files with the V1
layout reader. PlanDataSource exposes any ScanInput as a DataSource whose
single partition runs on the planner-driven ScanDriver, and
VortexFile::data_source now uses it, falling back to the layout reader for
files with list layouts, which the V2 executor cannot read yet. As the
layout reader did, a planned file scan whose filter the footer statistics
prove false reads nothing.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Engines that keep a V1/V2 switch while they move onto the Scan API, and
benchmarks comparing the two executors, need each one as a DataSource.
layout_reader_data_source() returns the layout-reader source that
data_source() already falls back to, with the same footer statistics, and
data_source() now calls it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #11665, which carries reworked versions of both commits (the V2 file source and the layout-reader source) on top of split/open.

@gatesn gatesn closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant