Repository navigation
perf(scan): share decoded segments while live work covers their rows - #11248
joseph-isaacs wants to merge 9 commits into
Conversation
Extract the row-claim read lifetime from the closed #10857 on its own, without its layout, DuckDB, or DataFusion changes. `FileReads` shares pending and completed reads across all workers of one file execution. Work owns a `RowClaim` over the file-local rows it may still read, and moves, splits (`split_to`), narrows (`restrict_to`), or shares that claim as it is handed between planners. A read whose `ReadLifetime` bounds its consumers to a row range is evicted once no live claim covers that range; reads with unknown lifetimes live until the last claim drops. Nothing constructs these types yet; wiring them into the V2 scan planners is left to a follow-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Row claims are a timing system for when in-memory scan state can be dropped, not an IO concern, so they now live in the planner driver. A `Liveness` tracks which root rows of one file live work may still touch. `Run::admit_with_liveness` gives every planner and morsel descended from a root a `RowClaim` over its scope's rows: taken when the work is spawned, narrowed after each compute to the work's new optional `remaining_rows()`, and released when the work retires or is cancelled. Children claim their rows before their parent narrows, so handed-off rows are never uncovered in between. Planners no longer pass claims around themselves. Any value decoded for those rows (arrays, dictionary values, buffers) can be pinned to a `Lifetime` with `Liveness::pin` and is dropped once no claim covers its rows. Caches that want to find it again keep a weak reference. This removes the IO-layer `FileReads` coordinator and its read lifetimes. `Planner` and `Morsel` gain `remaining_rows()`, which defaults to keeping the whole scope, so existing implementations are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A V2 scan decodes every segment once per reader: the filter conjuncts and the projection of a chunk each decode the columns they share, and chunks overlapping one stored segment each decode it again. Develop only keeps raw segment bytes, in a 64 MiB LRU, so those decodes are repeated. Keep each whole decoded segment in a per-file `DecodedSegments` cache instead, pinned in the file's driver `Liveness` to the root rows the segment serves. A later reader of the segment takes the kept decode and issues no read. The decode is dropped as soon as no live planner or morsel claims any of those rows, so the cache holds only what work in flight can still use, with no byte budget to tune. - `ScanCache` owns the file's `Liveness` and indexes each segment's root rows from the layout once. Dictionary values and zone maps serve their parent's rows. - `ScanSource` takes a kept decode before reading, and keeps its own. The filter planner no longer prefetches segments already decoded. - Engines (DataFusion, DuckDB, vortex-file) admit roots with the scan's liveness. `Progress::Spawn` now carries the child's claim, admitted with `Run::admit_spawned`, so a chunk's rows stay claimed while it waits to be admitted. - `ChunkPlanner` reports its remaining rows. The driver clamps reported rows to the claim instead of failing. - `ScanCache::with_retained_bytes(0)` keeps disabling all segment sharing, as tests that count reads expect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Previous Runs (16)
Powered by Polar Signals Cloud |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.948x ➖, 3↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.031x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.034x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.017x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.016x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.013x ➖, 1↑ 3↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.815x ➖, 3↑ 1↓)
datafusion / parquet / ns (0.841x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed / ns (0.861x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.801x ➖, 4↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.895x ✅, 38↑ 8↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.018x ➖, 0↑ 7↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 1↓)
|
Indexing every segment's root rows walked the whole stored layout on a file's first scan, materializing every child layout, including the columns a query never reads. On ClickBench's wide table that added about 20 ms to each query's cold run (cold geomean 1.13x) without changing hot runs. Pin each decode to the root rows of the split that read it instead, which the pipeline already knows when it compiles the split. A later reader that takes the decode pins it to its own split's rows too, and the decode is dropped with its last pin. No layout walk remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The filter planner skipped prefetching segments another reader had decoded, but a decode can be dropped between that check and the read, for example when the neighbouring chunk that pinned it finishes. The read then misses with no prefetch ahead of it, which cost cold runs up to about 11% on PolarSignals against a control. Prefetch as develop does; a reader that finds the decode still skips its own read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Stacked on #11201. This PR includes its commits; review only the last one,
perf(scan): share decoded segments…. It targetsdeveloprather than the #11201 branch because the SQL benchmark workflow only runs on PRs intodevelop. #11201 changes no runtime behaviour, so these benchmarks measure only this change.A V2 scan decodes each segment once per reader:
Develop keeps only raw segment bytes, in a 64 MiB least-recently-used cache, so those decodes are repeated.
This PR keeps each whole decoded segment in a per-file cache instead. Each entry is pinned in the file's driver
Liveness(from #11201) to the root rows the segment serves. A later reader takes the kept decode and issues no read. The decode is dropped as soon as no live planner or morsel claims any of those rows. The cache therefore holds only what in-flight work can still use, and has no byte budget to tune. #10849 tried the same reuse behind a fixed 16 MiB first-in-first-out cache. It made ClickBench Q2 9% faster, but FineWeb Q7 3.7% slower and peak memory 51 MiB higher. Exact lifetimes are meant to keep the hits without those costs.Changes
DecodedSegments(vortex-layout, pipeline): a per-file map from segment id to the whole decoded array.Liveness::pin(Lifetime::Rows(segment rows), …). When liveness drops the pin, a guard removes the entry.ScanSourcetakes a kept decode before reading and keeps its own after decoding. The filter planner no longer prefetches segments that are already decoded.ScanCacheowns the file'sLivenessand indexes each segment's root rows from the layout once. Dictionary values and zone maps serve their parent's rows.ScanCache::with_retained_bytes(0)still turns off all segment sharing, as the tests that count reads expect.PreparedScan::liveness().Progress::Spawnnow carries the child'sRowClaim, which the engine admits withRun::admit_spawned. A chunk's rows stay claimed while it waits to be admitted, which for DuckDB includes sitting in the part queue.ChunkPlannerreports its remaining rows, so its claim shrinks as it hands out chunks.Tests:
decoded_segments_are_shared_while_their_rows_are_claimed. A second scan of a claimed chunked column issues no read and returns the same rows. After the claim is dropped, it reads both segments again.segment_rowstests cover nested chunks, dictionary values, and segments shared by several layouts.API Changes
Progress::Spawngains a third field,Option<RowClaim>. This breaks code that matches on it.Run::admit_spawned,RowClaim::liveness,PreparedScan::liveness,ScanCache::liveness,Scan::with_decoded_segments, andDecodedSegments.Generated by Claude Code