Skip to content

Experiment: share the V2 part planner through vortex-scan - #11714

Draft
gatesn wants to merge 2 commits into
ngates/scan-exp-a-row-limitfrom
ngates/scan-exp-a-part-root
Draft

gatesn wants to merge 2 commits into
ngates/scan-exp-a-row-limitfrom
ngates/scan-exp-a-part-root

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Part of #9062

Moves the V2 part planner out of vortex-duckdb/src/scan_v2.rs into vortex_scan::prepare::part_root, so that every host cuts a prepared V2 scan into parts the same way. DuckDB calls the shared function. The code moves verbatim, so DuckDB's behaviour does not change.

part_root(scan, splits, dynamic) returns the root planner of a prepared scan. The root spawns one part per range between consecutive splits, clamped to the scan's rows and selection. With dynamic, it first prunes the whole scan against the filter's current dynamic bound, which reads zone maps. Without it, the root reads nothing, so a host may run it synchronously.

This PR is one layer of experiment A, which tests one decision about the Scan API: whether DataSourceScan::open(), a cloneable async ScanStream that each host thread polls, is enough for the V2 scan engine, or whether a host needs V2's synchronous step. The stack puts the V2 engine behind open() and moves DuckDB's V2 scan onto it. The gate is DuckDB V2 on TPC-H and ClickBench within noise of develop's DuckDB V2, measured on the last layer with rotated runs. If V2 cannot reach parity through the stream, the experiment stops and reports what the engine needs that ScanStream cannot express, with profile evidence, rather than tuning around it.

Stacked on #11713

🤖 Generated with Claude Code

Every host that reads a prepared V2 scan in parts should cut it the same
way, so DuckDB's part planner becomes vortex_scan::prepare::part_root and
DuckDB calls it. The code moves unchanged.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant