Skip to content

Experiment: read DataFusion file scans through the Scan API - #11718

Draft
gatesn wants to merge 4 commits into
ngates/duckdb-scan-poolfrom
ngates/scan-exp-b-fileopener
Draft

gatesn wants to merge 4 commits into
ngates/duckdb-scan-poolfrom
ngates/scan-exp-b-fileopener

Conversation

@gatesn

@gatesn gatesn commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

This experiment tests one decision about the Scan API: whether DataSourceScan::split(n) is the right primitive for hosts that cut work themselves, or whether those hosts need the source's cut points, with split(n) derived from them. DataFusion's default FileOpener assigns byte ranges of each file to its partitions, so on its V1 path it now holds only a DataSource and a ScanRequest. It reads scan(ScanRequest { row_range, .. }) followed by open() on a LayoutReaderDataSource shared by the partitions of a file, in place of a ScanBuilder, and converts arrays with the to_arrow stream extension. The V2 path is unchanged.

The port needed one addition to the trait: DataSource::cut_points(&ScanRequest), which returns the sorted row boundaries at which a scan of the request can be cut without splitting the source's units of work. They are computed from metadata only, depend on the fields the projection and filter read, and are clamped to the request's row range. The default returns None. The opener maps its byte range onto these boundaries with its existing midpoint rule and passes the result as the request's row_range. The layout source computes the boundaries once for each set of fields, caches them, and cuts its own later scans at them, so the opener no longer hands natural splits back to a builder. It also needed LayoutReaderDataSource::with_scan_metrics, so that the per-file sources add to the opener's shared counters.

On the V1 source, split(n) for a request without a selection is group_contiguous(cut_points, n): it groups the same boundaries the opener maps byte ranges onto into n contiguous runs of about equal rows. A selection makes the source cut at the selection's ranges first. That is the evidence this experiment was after. Cut points are the primitive that both host-cut scans and split(n) are built from.

The alternative was a byte range on ScanRequest that the source snaps to its own boundaries. It was rejected because the layout source does not know the file's byte size, because it would build one host's assignment rule into every source, and because a byte range means nothing for a multi-file source. Spark's SplitRows.of maps byte ranges to rows by proportion, with no alignment, and could read the same boundaries.

Each DataFusion partition reads its scan through one stream. To keep as many splits in flight as the scan pipeline did, the opened layout scan now honours the request's split_concurrency per stream as given, rather than capping each stream at the pipeline's per-thread default of four. A single stream spawns every split it takes, so it can hold that many. The opener sets split_concurrency to the pipeline's total, its per-thread scan_concurrency or four, times the machine's threads. split_concurrency therefore means splits in flight per opened stream, or per partition on the partition path. Neither is per thread, so a host has to multiply by the thread count itself, and it has to know the source's default of four to match it. On the ordered and limited partition path, the value is a total cap above the pipeline's per-thread default, so a scan_concurrency above four is not honoured there. That option is off by default.

The gate is the default datafusion:vortex targets on TPC-H and ClickBench within noise of develop. The kill criterion is that parity needs the opener to reach around the Scan API into ScanBuilder internals.

Stacked on #11664. Part of #9062.

🤖 Generated with Claude Code

DataFusion's default FileOpener reads its V1 path through
`DataSource::scan(ScanRequest { row_range, .. })` and `open()` on a
`LayoutReaderDataSource` shared per file, instead of building a
`ScanBuilder`. The byte-range-to-row mapping uses a new
`DataSource::cut_points`, which the layout source caches and reuses for
its own scans in place of the opener handing natural splits back.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@gatesn gatesn added the changelog/chore A trivial change label Nov 20, 2026
gatesn and others added 3 commits November 20, 2026 17:18
An opened layout scan keeps the request's `split_concurrency` in flight
per stream as given, instead of capping each stream at the pipeline's
per-thread default. The opener sets it to the scan pipeline's total,
per thread times the machine's threads, and reads one stream per
partition instead of merging a clone per core.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant