Skip to content

Experiment: expose Vortex files as V2 plan sources - #11716

Draft
gatesn wants to merge 5 commits into
ngates/scan-exp-a-plan-sourcefrom
ngates/scan-exp-a-file-plan-source
Draft

gatesn wants to merge 5 commits into
ngates/scan-exp-a-plan-sourcefrom
ngates/scan-exp-a-file-plan-source

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Part of #9062

Adds VortexFile::plan_data_source and plan_child, which expose a file as a PlanDataSource child. data_source() still returns the layout-reader source, and layout_reader_data_source names that path explicitly.

A file's child carries the three things the source needs without reading payloads: the file's row count, a pruner that tests the filter against the footer's file statistics, and a splitter. The splitter cuts the file's work at its layout splits over the fields a scan reads, the same boundaries a V1 scan of those fields uses, so V1 and V2 parallelise a file at the same grain. The splitter reads the file's layout reader, so a file opened with_caching builds that reader once. FileDataSource<S> wraps either executor's source with the file's footer statistics, so both answer field_statistics without reading payloads. Tests run the same reads on both executors and compare them.

This PR is one layer of experiment A, which tests one decision about the Scan API: whether DataSourceScan::open(), a cloneable async ScanStream that each host thread polls, is enough for the V2 scan engine, or whether a host needs V2's synchronous step. The stack puts the V2 engine behind open() and moves DuckDB's V2 scan onto it. The gate is DuckDB V2 on TPC-H and ClickBench within noise of develop's DuckDB V2, measured on the last layer with rotated runs. If V2 cannot reach parity through the stream, the experiment stops and reports what the engine needs that ScanStream cannot express, with profile evidence, rather than tuning around it.

Stacked on #11715

🤖 Generated with Claude Code

VortexFile::plan_data_source scans one file through PlanDataSource and
plan_child makes a file a child of one, with the file's statistics
pruning scans they rule out and its layout splits over the fields a scan
reads cutting the scan's parts, as a V1 scan cuts its splits. Both are
opt-in: data_source() still scans with the layout reader, which
layout_reader_data_source names explicitly.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
gatesn and others added 4 commits November 20, 2026 17:20
…source

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…source

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…source

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant