Skip to content

Run V2 plan scans through split and open - #11665

Closed
gatesn wants to merge 22 commits into
ngates/scan-native-poolfrom
ngates/v2-scan-pool
Closed

gatesn wants to merge 22 commits into
ngates/scan-native-poolfrom
ngates/v2-scan-pool

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by a chain of smaller PRs, which together scan V2 plans through the Scan API's split and open:

  1. Add a Scan API source over V2 plans #11699 adds PlanDataSource, a format-agnostic source over V2 plans.
  2. Expose Vortex files as Scan API sources over V2 plans #11700 exposes Vortex files through it.
  3. Bind a request once per (expression, dtype) in each session #11701 binds a request once per (expression, dtype) in each session.
  4. Split V2 plan scans along the chunk grid #11702 splits scans along the chunk grid.
  5. Run opened V2 plan scans on V2's own driver #11703 runs opened scans on V2's planner driver.
  6. Run DataFusion's Scan API tables on the V2 scan #11704 runs DataFusion's Scan API bench tables on V2.
  7. Scan Vortex files on the V2 scan through data_source #11705 makes VortexFile::data_source() scan on V2. This is the only default-path change, held for benchmarks.

🤖 Generated with Claude Code

gatesn and others added 4 commits November 19, 2026 15:54
Layout scans now split on their layout splits and open into a pool
whose workers share those splits through a channel. Multi-file scans
queue an open for each deferred file; the worker that opens one sends
the file's splits back into the channel for any worker to take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Engines that consume the Scan API still scanned Vortex files with the V1
layout reader. PlanDataSource exposes any ScanInput as a DataSource whose
single partition runs on the planner-driven ScanDriver, and
VortexFile::data_source now uses it, falling back to the layout reader for
files with list layouts, which the V2 executor cannot read yet. As the
layout reader did, a planned file scan whose filter the footer statistics
prove false reads nothing.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Engines that keep a V1/V2 switch while they move onto the Scan API, and
benchmarks comparing the two executors, need each one as a DataSource.
layout_reader_data_source() returns the layout-reader source that
data_source() already falls back to, with the same footer statistics, and
data_source() now calls it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the changelog/feature A new feature label Nov 19, 2026
gatesn and others added 2 commits November 19, 2026 16:14
PlanDataSource now covers one or many files, the later ones opened during
scans, and implements the Scan API's split and open natively. split cuts
the scan's known rows on the 8,192-row block grid into contiguous pieces
of about equal rows, from metadata only, keeping unopened files whole.
open prepares each open file once and hands its parts (its chunks, or its
splits when given) to workers through the V1 pool's pattern: a channel of
units, deferred files as queued opens that push their parts back, and up
to split_concurrency parts in flight per worker, each driven on a planning
run on the worker's own thread. MultiFileDataSource::build_plan builds the
multi-file source. RowLimit, group_contiguous and the in-flight default
move from vortex-layout into vortex-scan so both pools share them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The plan source tests read files small and compressible enough that the
footer read held every segment, so no part ever waited on a read, and
every filtered scan was a single part. They now read larger files with a
scrambled second column through a reader whose reads pend once, filter
on that column, and cover parts cut on a file's splits. A worker's output
also ends after its first error, as the V1 pool's does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn mentioned this pull request Nov 19, 2026
16 of 46 tasks
gatesn and others added 3 commits November 19, 2026 18:00
The Scan API sits between any host and any format, so the V2 source's
pool should not be about Vortex files. A deferred child now expands into
children, which may be deferred in turn, as a table format's manifest
lists files; the pool queues the work of whatever a child expands into,
and an ordered read expands each child in its place. PlanFile,
DeferredPlanFile and PlanSourceFile become PlanChild, DeferredPlanChild
and PlanSourceChild; a multi-file Vortex scan is one deferred child per
file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Layout scans now split on their layout splits and open into a pool
whose workers share those splits through a channel. Multi-file scans
queue an open for each deferred file; the worker that opens one sends
the file's splits back into the channel for any worker to take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Decoding a split inline kept each worker's splits on its own thread.
Spawning them onto the session's runtime, as V1 scans do, lets idle
runtime threads decode them and starts their I/O at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/scan-native-pool branch from d862569 to 540b8cc Compare November 19, 2026 23:30
gatesn and others added 3 commits November 19, 2026 18:34
V2 pool parts now spawn onto the session runtime, matching the base's
new ScanPool contract for V1 splits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The base now spawns each split onto the session runtime, but the test
session had no runtime handle. Give it a current-thread runtime with
background workers, and stop assuming an unordered piece completes in
order or that a spawned split is ready on the first poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Match develop's V2 DuckDB scan: each child is prepared once and its root
runs up front, spawning one part per layout split over the fields the
scan reads. A dynamic filter prunes the child as a whole before its
parts are cut, as queued work since it reads zone maps. PartPlanner moves
into vortex-scan as part_root so hosts can share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/scan-native-pool branch from 540b8cc to c638db4 Compare November 20, 2026 00:17
gatesn and others added 2 commits November 19, 2026 19:21
Take the base's reworked layout pool as is. V2 pools now report progress
and exhaustion through the base's ScanPool methods, and scale parts in
flight to the core count, at most split_concurrency, as the layout pool
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn and others added 4 commits November 19, 2026 20:15
Requests stay unbound, as vortex#11654 settled: each source binds them
against its own dtype in scan(). An engine that opens a plan source per
file, as DuckDB does, therefore bound and optimized the same projection
and filter once per file, where the V2 DuckDB path bound them once per
query. A small per-session cache keyed by expression and dtype restores
binding once per (expression, dtype).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
#11626's union layer is superseded by PlanDataSource::from_children on
#11665, which MultiFileDataSource::build_plan already builds. This merge
keeps the branch's history and takes #11665 plus the table path change
as its tree.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gatesn and others added 3 commits November 20, 2026 12:53
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
With `--scan-api true --scan-v1 false`, datafusion-bench registers a `VortexTable` over `MultiFileDataSource::build_plan`, so the datasource integration can run the V2 scan. The orchestrator now accepts datafusion datasource-v2 targets, and every-scan coverage runs them. Folds in #11626.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
An opened PlanDataSource scan now runs as develop's V2 DuckDB scan does: each worker owns a planning run, driven on the task that polls it, fed with parts from a queue every worker shares, yielding batches as they are produced. Parts are no longer spawned onto the runtime with several in flight per worker; a worker runs one part at a time unless the request's split_concurrency says otherwise, as a DuckDB thread runs one chunk at a time. Ordered pools run their parts in row order on the first worker's run. SPLITS_IN_FLIGHT_PER_WORKER only fed the old pool, so it goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn changed the title Scan Vortex files on the V2 engine through split and open Run V2 plan scans through split and open Nov 20, 2026
Takes the reworked #11663, which reads V1 layout scans through split and open on develop's pipeline and includes #11662's required split and open. The V1 layout files are #11663's as they stand; vortex-scan's pool keeps V2's RowLimit, trim_to_limit and group_contiguous beside them. The resolution is the tree of #11665's commits replayed on the new base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Split into a chain of smaller PRs. The chain's last PR has the same tree as this branch, plus #11663's latest test fix:

  1. Add a Scan API source over V2 plans #11699 Add a Scan API source over V2 plans
  2. Expose Vortex files as Scan API sources over V2 plans #11700 Expose Vortex files as Scan API sources over V2 plans
  3. Bind a request once per (expression, dtype) in each session #11701 Bind a request once per (expression, dtype) in each session
  4. Split V2 plan scans along the chunk grid #11702 Split V2 plan scans along the chunk grid
  5. Run opened V2 plan scans on V2's own driver #11703 Run opened V2 plan scans on V2's own driver
  6. Run DataFusion's Scan API tables on the V2 scan #11704 Run DataFusion's Scan API tables on the V2 scan
  7. Scan Vortex files on the V2 scan through data_source #11705 Scan Vortex files on the V2 scan through data_source (the only default-path change, held for benchmarks)

@gatesn gatesn closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant