Skip to content

Run DataFusion's Scan API tables on the V2 scan - #11704

Draft
gatesn wants to merge 2 commits into
ngates/v2-plan-openfrom
ngates/v2-plan-datafusion-tables
Draft

gatesn wants to merge 2 commits into
ngates/v2-plan-openfrom
ngates/v2-plan-datafusion-tables

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

datafusion-bench's Scan API tables (--scan-api true) follow --scan-v1:

  • On V2 they register MultiFileDataSource::build_plan.
  • On V1 they register build.

The bench orchestrator accepts datasource-integration targets on V2, and the nightly TPC-DS SF100 matrix gains a "DataFusion, Vortex, datasource v2" job, which costs one more machine. The orchestrator passes --scan-v1 explicitly for every target, so every other target runs the same scan as before.

Stacked on #11703.

🤖 Generated with Claude Code

With `--scan-api true --scan-v1 false`, datafusion-bench registers a `VortexTable` over `MultiFileDataSource::build_plan`, so the datasource integration can run the V2 scan. The orchestrator now accepts datafusion datasource-v2 targets, and every-scan coverage runs them. Folds in #11626.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant