Skip to content

Scan Vortex files on the V2 scan through data_source - #11705

Draft
gatesn wants to merge 2 commits into
ngates/v2-plan-datafusion-tablesfrom
ngates/v2-plan-default-source
Draft

gatesn wants to merge 2 commits into
ngates/v2-plan-datafusion-tablesfrom
ngates/v2-plan-default-source

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Behaviour change: VortexFile::data_source() returns the file's plan data source, so Scan API hosts that call it read the file on the V2 scan. Today those callers are JNI's file scan and DataFusion's V2 tests.

Files with a list layout still scan on the layout reader, because the V2 scan cannot read list layouts yet. planned_scan_supported walks the layout tree only when the dtype could hold a list layout. layout_reader_data_source keeps the layout-reader path reachable for every file.

Benchmarks: pending. This changes a default path, so the PR stays a draft until a table comparing V2 through open with the layout reader is posted here.

Stacked on #11704.

🤖 Generated with Claude Code

VortexFile::data_source now returns the plan data source, so Scan API
hosts such as the Java bindings read files on the V2 scan. Files with a
list layout, which the V2 scan cannot read yet, still scan on the layout
reader; layout_reader_data_source keeps that path reachable for every
file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the changelog/feature A new feature label Nov 20, 2026
…urce

Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant