Repository navigation
feat(scan): track row liveness in the planner driver to time dropping in-memory scan state - #11201
joseph-isaacs wants to merge 5 commits into
Conversation
Extract the row-claim read lifetime from the closed #10857 on its own, without its layout, DuckDB, or DataFusion changes. `FileReads` shares pending and completed reads across all workers of one file execution. Work owns a `RowClaim` over the file-local rows it may still read, and moves, splits (`split_to`), narrows (`restrict_to`), or shares that claim as it is handed between planners. A read whose `ReadLifetime` bounds its consumers to a row range is evicted once no live claim covers that range; reads with unknown lifetimes live until the last claim drops. Nothing constructs these types yet; wiring them into the V2 scan planners is left to a follow-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Previous Runs (8)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.968x ➖, 2↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 1↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.816x ➖, 3↑ 1↓)
datafusion / parquet / ns (0.845x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed / ns (0.883x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.832x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.890x ✅, 36↑ 7↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.027x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.019x ➖, 0↑ 0↓)
|
Row claims are a timing system for when in-memory scan state can be dropped, not an IO concern, so they now live in the planner driver. A `Liveness` tracks which root rows of one file live work may still touch. `Run::admit_with_liveness` gives every planner and morsel descended from a root a `RowClaim` over its scope's rows: taken when the work is spawned, narrowed after each compute to the work's new optional `remaining_rows()`, and released when the work retires or is cancelled. Children claim their rows before their parent narrows, so handed-off rows are never uncovered in between. Planners no longer pass claims around themselves. Any value decoded for those rows (arrays, dictionary values, buffers) can be pinned to a `Lifetime` with `Liveness::pin` and is dropped once no claim covers its rows. Caches that want to find it again keep a weak reference. This removes the IO-layer `FileReads` coordinator and its read lifetimes. `Planner` and `Morsel` gain `remaining_rows()`, which defaults to keeping the whole scope, so existing implementations are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Builds on the row claims from #10857, but places them in the planner driver instead of the IO layer. Claims are a timing system: they decide when in-memory structures decoded for a file's rows (arrays, dictionary values, buffers) can be dropped. The driver already knows every live planner and morsel and the rows each covers, so it owns the claims, and planners no longer pass them around.
Nothing in the V2 scan uses this yet. Wiring the layout planners and caches to it is left to follow-ups.
Changes
New module
vortex-scan/src/planning/driver/liveness/:Livenesstracks which root rows of one file live work still claims. Clones share state, so the runs of every thread scanning a file report to one liveness.Run::admit_with_liveness(root, scope, io, liveness)gives every planner and morsel descended from the root aRowClaimover its scope's rows. The claim is taken when the work is spawned and released when the work retires or is cancelled. A child claims its rows before its parent narrows, so handed-off rows are never uncovered in between.Run::admitis unchanged and tracks nothing.Planner::remaining_rows()andMorsel::remaining_rows()are new optional methods. After eachcompute(), the driver narrows the item's claim to them. They may only shrink, and the run fails if they widen. The defaultNonekeeps the whole scope, so existing implementations are unaffected.Liveness::pin(Lifetime, value)holds any value until no claim covers its rows (Lifetime::Rows) or until no claim is live at all (Lifetime::File). Pinning rows that are already unclaimed drops the value at once. A cache that wants to find a pinned value again keeps a weak reference.Liveness::claim(rows)over a whole scan, so values outlive the gap between one root finishing and the next being admitted.This removes the IO-layer
FileReadscoordinator and its per-read lifetimes from the first revision of this PR.Tests in
liveness/tests.rscover:API Changes
Additive and experimental:
vortex_scan::planning::driver::{Liveness, Lifetime, RowClaim}Run::admit_with_livenessPlanner::remaining_rowsandMorsel::remaining_rows, with default implementations🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW