Repository navigation
Conversation
A PartitionPool runs each partition of a PlanDataSource scan with its own ScanDriver, which spends a driver, an I/O session and a task per partition. open instead runs the scan as develop's V2 DuckDB scan does: it prepares each child once and queues its parts, and each worker drives a planning run of its own over them, on the task that polls it. A file's parts follow its layout splits over the fields the scan reads, as a V1 scan's do, and a dynamic filter prunes the whole child before its parts are cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
This was referenced Nov 20, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gives
PlanDataSourcea pool of its own:openruns the scan on V2's planner driver, in the shape ofvortex-duckdb/src/scan_v2.rs.openprepares each ready child once and queues the child's parts. Each worker owns a planningRun, driven on the task that polls the worker. A worker takes parts from the shared queue, runs a part's chunks side by side, yields batches as they are produced, and awaits reads withpoll_completion. A host thread therefore does its own CPU work, and nothing is spawned per part.The part planner lives in vortex-scan as
part_rootso that hosts can share it.Contracts to check:
open.split_concurrencyraises that.RowLimit.About half of the diff is
vortex-file/src/plan_tests.rs. Those tests run real files through the session's I/O, with a reader that pends before each read completes.Stacked on #11702.
🤖 Generated with Claude Code