Skip to content

Experiment: move RowLimit into vortex-scan - #11713

Draft
gatesn wants to merge 2 commits into
ngates/duckdb-scan-poolfrom
ngates/scan-exp-a-row-limit
Draft

gatesn wants to merge 2 commits into
ngates/duckdb-scan-poolfrom
ngates/scan-exp-a-row-limit

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Part of #9062

Moves RowLimit and limit_rows from vortex-layout into a new vortex_scan::limit module, so that a scan in vortex-scan can enforce one limit across every clone of its stream. vortex-layout re-exports the type instead of keeping its own copy. RowLimit is one cloneable budget: every worker takes rows from it, so together they return at most the limit. limit_rows trims a stream to the rows it is granted and drops upstream work once the budget is spent. The code and its tests are unchanged.

This PR is one layer of experiment A, which tests one decision about the Scan API: whether DataSourceScan::open(), a cloneable async ScanStream that each host thread polls, is enough for the V2 scan engine, or whether a host needs V2's synchronous step. The stack puts the V2 engine behind open() and moves DuckDB's V2 scan onto it. The gate is DuckDB V2 on TPC-H and ClickBench within noise of develop's DuckDB V2, measured on the last layer with rotated runs. If V2 cannot reach parity through the stream, the experiment stops and reports what the engine needs that ScanStream cannot express, with profile evidence, rather than tuning around it.

Stacked on #11664

🤖 Generated with Claude Code

Scans in vortex-scan need the row budget that layout scans share, so
the budget and the stream adapter that trims to it move down a crate
and vortex-layout uses them from there.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant