Repository navigation
Conversation
Layout scans now split on their layout splits and open into a pool whose workers share those splits through a channel. Multi-file scans queue an open for each deferred file; the worker that opens one sends the file's splits back into the channel for any worker to take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Engines that consume the Scan API still scanned Vortex files with the V1 layout reader. PlanDataSource exposes any ScanInput as a DataSource whose single partition runs on the planner-driven ScanDriver, and VortexFile::data_source now uses it, falling back to the layout reader for files with list layouts, which the V2 executor cannot read yet. As the layout reader did, a planned file scan whose filter the footer statistics prove false reads nothing. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Engines that keep a V1/V2 switch while they move onto the Scan API, and benchmarks comparing the two executors, need each one as a DataSource. layout_reader_data_source() returns the layout-reader source that data_source() already falls back to, with the same footer statistics, and data_source() now calls it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
PlanDataSource now covers one or many files, the later ones opened during scans, and implements the Scan API's split and open natively. split cuts the scan's known rows on the 8,192-row block grid into contiguous pieces of about equal rows, from metadata only, keeping unopened files whole. open prepares each open file once and hands its parts (its chunks, or its splits when given) to workers through the V1 pool's pattern: a channel of units, deferred files as queued opens that push their parts back, and up to split_concurrency parts in flight per worker, each driven on a planning run on the worker's own thread. MultiFileDataSource::build_plan builds the multi-file source. RowLimit, group_contiguous and the in-flight default move from vortex-layout into vortex-scan so both pools share them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The plan source tests read files small and compressible enough that the footer read held every segment, so no part ever waited on a read, and every filtered scan was a single part. They now read larger files with a scrambled second column through a reader whose reads pend once, filter on that column, and cover parts cut on a file's splits. A worker's output also ends after its first error, as the V1 pool's does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
16 of 46 tasks
The Scan API sits between any host and any format, so the V2 source's pool should not be about Vortex files. A deferred child now expands into children, which may be deferred in turn, as a table format's manifest lists files; the pool queues the work of whatever a child expands into, and an ordered read expands each child in its place. PlanFile, DeferredPlanFile and PlanSourceFile become PlanChild, DeferredPlanChild and PlanSourceChild; a multi-file Vortex scan is one deferred child per file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Layout scans now split on their layout splits and open into a pool whose workers share those splits through a channel. Multi-file scans queue an open for each deferred file; the worker that opens one sends the file's splits back into the channel for any worker to take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Decoding a split inline kept each worker's splits on its own thread. Spawning them onto the session's runtime, as V1 scans do, lets idle runtime threads decode them and starts their I/O at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn
force-pushed
the
ngates/scan-native-pool
branch
from
November 19, 2026 23:30
d862569 to
540b8cc
Compare
V2 pool parts now spawn onto the session runtime, matching the base's new ScanPool contract for V1 splits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The base now spawns each split onto the session runtime, but the test session had no runtime handle. Give it a current-thread runtime with background workers, and stop assuming an unordered piece completes in order or that a spawned split is ready on the first poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Match develop's V2 DuckDB scan: each child is prepared once and its root runs up front, spawning one part per layout split over the fields the scan reads. A dynamic filter prunes the child as a whole before its parts are cut, as queued work since it reads zone maps. PartPlanner moves into vortex-scan as part_root so hosts can share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn
force-pushed
the
ngates/scan-native-pool
branch
from
November 20, 2026 00:17
540b8cc to
c638db4
Compare
Take the base's reworked layout pool as is. V2 pools now report progress and exhaustion through the base's ScanPool methods, and scale parts in flight to the core count, at most split_concurrency, as the layout pool does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ex into ngates/v2-scan-pool
…ex into ngates/v2-scan-pool
Requests stay unbound, as vortex#11654 settled: each source binds them against its own dtype in scan(). An engine that opens a plan source per file, as DuckDB does, therefore bound and optimized the same projection and filter once per file, where the V2 DuckDB path bound them once per query. A small per-session cache keyed by expression and dtype restores binding once per (expression, dtype). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
This was referenced Nov 20, 2026
gatesn
added a commit
that referenced
this pull request
Nov 20, 2026
#11626's union layer is superseded by PlanDataSource::from_children on #11665, which MultiFileDataSource::build_plan already builds. This merge keeps the branch's history and takes #11665 plus the table path change as its tree. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
With `--scan-api true --scan-v1 false`, datafusion-bench registers a `VortexTable` over `MultiFileDataSource::build_plan`, so the datasource integration can run the V2 scan. The orchestrator now accepts datafusion datasource-v2 targets, and every-scan coverage runs them. Folds in #11626. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
An opened PlanDataSource scan now runs as develop's V2 DuckDB scan does: each worker owns a planning run, driven on the task that polls it, fed with parts from a queue every worker shares, yielding batches as they are produced. Parts are no longer spawned onto the runtime with several in flight per worker; a worker runs one part at a time unless the request's split_concurrency says otherwise, as a DuckDB thread runs one chunk at a time. Ordered pools run their parts in row order on the first worker's run. SPLITS_IN_FLIGHT_PER_WORKER only fed the old pool, so it goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Takes the reworked #11663, which reads V1 layout scans through split and open on develop's pipeline and includes #11662's required split and open. The V1 layout files are #11663's as they stand; vortex-scan's pool keeps V2's RowLimit, trim_to_limit and group_contiguous beside them. The resolution is the tree of #11665's commits replayed on the new base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
This was referenced Nov 20, 2026
Contributor
Author
|
Split into a chain of smaller PRs. The chain's last PR has the same tree as this branch, plus #11663's latest test fix:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by a chain of smaller PRs, which together scan V2 plans through the Scan API's
splitandopen:PlanDataSource, a format-agnostic source over V2 plans.VortexFile::data_source()scan on V2. This is the only default-path change, held for benchmarks.🤖 Generated with Claude Code