Repository navigation
Conversation
DataFusion's default FileOpener reads its V1 path through
`DataSource::scan(ScanRequest { row_range, .. })` and `open()` on a
`LayoutReaderDataSource` shared per file, instead of building a
`ScanBuilder`. The byte-range-to-row mapping uses a new
`DataSource::cut_points`, which the layout source caches and reuses for
its own scans in place of the opener handing natural splits back.
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An opened layout scan keeps the request's `split_concurrency` in flight per stream as given, instead of capping each stream at the pipeline's per-thread default. The opener sets it to the scan pipeline's total, per thread times the machine's threads, and reads one stream per partition instead of merging a clone per core. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This experiment tests one decision about the Scan API: whether
DataSourceScan::split(n)is the right primitive for hosts that cut work themselves, or whether those hosts need the source's cut points, withsplit(n)derived from them. DataFusion's defaultFileOpenerassigns byte ranges of each file to its partitions, so on its V1 path it now holds only aDataSourceand aScanRequest. It readsscan(ScanRequest { row_range, .. })followed byopen()on aLayoutReaderDataSourceshared by the partitions of a file, in place of aScanBuilder, and converts arrays with theto_arrowstream extension. The V2 path is unchanged.The port needed one addition to the trait:
DataSource::cut_points(&ScanRequest), which returns the sorted row boundaries at which a scan of the request can be cut without splitting the source's units of work. They are computed from metadata only, depend on the fields the projection and filter read, and are clamped to the request's row range. The default returnsNone. The opener maps its byte range onto these boundaries with its existing midpoint rule and passes the result as the request'srow_range. The layout source computes the boundaries once for each set of fields, caches them, and cuts its own later scans at them, so the opener no longer hands natural splits back to a builder. It also neededLayoutReaderDataSource::with_scan_metrics, so that the per-file sources add to the opener's shared counters.On the V1 source,
split(n)for a request without a selection isgroup_contiguous(cut_points, n): it groups the same boundaries the opener maps byte ranges onto intoncontiguous runs of about equal rows. A selection makes the source cut at the selection's ranges first. That is the evidence this experiment was after. Cut points are the primitive that both host-cut scans andsplit(n)are built from.The alternative was a byte range on
ScanRequestthat the source snaps to its own boundaries. It was rejected because the layout source does not know the file's byte size, because it would build one host's assignment rule into every source, and because a byte range means nothing for a multi-file source. Spark'sSplitRows.ofmaps byte ranges to rows by proportion, with no alignment, and could read the same boundaries.Each DataFusion partition reads its scan through one stream. To keep as many splits in flight as the scan pipeline did, the opened layout scan now honours the request's
split_concurrencyper stream as given, rather than capping each stream at the pipeline's per-thread default of four. A single stream spawns every split it takes, so it can hold that many. The opener setssplit_concurrencyto the pipeline's total, its per-threadscan_concurrencyor four, times the machine's threads.split_concurrencytherefore means splits in flight per opened stream, or per partition on the partition path. Neither is per thread, so a host has to multiply by the thread count itself, and it has to know the source's default of four to match it. On the ordered and limited partition path, the value is a total cap above the pipeline's per-thread default, so ascan_concurrencyabove four is not honoured there. That option is off by default.The gate is the default
datafusion:vortextargets on TPC-H and ClickBench within noise of develop. The kill criterion is that parity needs the opener to reach around the Scan API intoScanBuilderinternals.Stacked on #11664. Part of #9062.
🤖 Generated with Claude Code