Repository navigation
perf(scan): keep dictionary values only while live work claims their rows - #11355
joseph-isaacs wants to merge 16 commits into
Conversation
Extract the row-claim read lifetime from the closed #10857 on its own, without its layout, DuckDB, or DataFusion changes. `FileReads` shares pending and completed reads across all workers of one file execution. Work owns a `RowClaim` over the file-local rows it may still read, and moves, splits (`split_to`), narrows (`restrict_to`), or shares that claim as it is handed between planners. A read whose `ReadLifetime` bounds its consumers to a row range is evicted once no live claim covers that range; reads with unknown lifetimes live until the last claim drops. Nothing constructs these types yet; wiring them into the V2 scan planners is left to a follow-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Row claims are a timing system for when in-memory scan state can be dropped, not an IO concern, so they now live in the planner driver. A `Liveness` tracks which root rows of one file live work may still touch. `Run::admit_with_liveness` gives every planner and morsel descended from a root a `RowClaim` over its scope's rows: taken when the work is spawned, narrowed after each compute to the work's new optional `remaining_rows()`, and released when the work retires or is cancelled. Children claim their rows before their parent narrows, so handed-off rows are never uncovered in between. Planners no longer pass claims around themselves. Any value decoded for those rows (arrays, dictionary values, buffers) can be pinned to a `Lifetime` with `Liveness::pin` and is dropped once no claim covers its rows. Caches that want to find it again keep a weak reference. This removes the IO-layer `FileReads` coordinator and its read lifetimes. `Planner` and `Morsel` gain `remaining_rows()`, which defaults to keeping the whole scope, so existing implementations are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…rows The V2 scan's ValueCache kept every dictionary's decoded values for the whole life of the ScanCache. Pin each value in the file's Liveness to the file rows its dictionary covers instead, so it is dropped once no live work claims any of those rows. Values recorded while no work claims their rows, as in a scan driven without liveness, are kept as long as the cache, as before. The compiler now tracks the file row of each plan's row zero through concatenations, and detaches it for dictionary values and zone tables, whose rows are not file rows. Engines admit their scans with the cache's liveness and pass claims with spawned chunks, so a dictionary stays while any chunk over it is planned or running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…alues Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # vortex-layout/src/plan/physical/plans/share.rs
Polar Signals Profiling ResultsLatest Run
Previous Runs (49)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.967x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.012x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.970x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 1↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.891x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.899x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.974x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.970x ➖, 1↑ 1↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Every planner and morsel the driver spawned took its own RowClaim, which locks the file's Liveness to add its rows and again to release them. DuckDB runs many threads over one file's Liveness, so on cheap, high-throughput scans such as TPC-DS's inventory queries that mutex was taken for every child of every chunk. A child whose scope lies within its parent's claim now shares that claim through an Arc, so spawning and retiring it never touch the Liveness. A parent that narrows while children still share its claim claims the narrower rows afresh and leaves the shared claim to them, so rows handed to a child stay claimed until it retires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A planner handing out chunks in order spawned each with a new claim and then narrowed its own, taking the liveness lock twice per chunk. DuckDB plans every chunk of a file on one thread before its threads start, so on files with thousands of chunks this delayed the whole scan. RowClaim::split_front moves a claim's leading rows into a claim of their own without touching the liveness, since the rows stay claimed. A spawned child at the front of a claim only its parent holds now takes those rows over, and the parent's narrowing then finds nothing to release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A benchmark-only experiment, to be reverted: with the driver and engine liveness wiring unchanged, the ValueCache keeps every value as on develop. If TPC-DS inventory queries stay slower, the wiring costs the time; if they recover, early eviction of dictionary values does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A benchmark-only experiment, to be reverted with af51447: DuckDB scans run without claims, as before the engines were wired to liveness, while every other change stays. If TPC-DS inventory queries recover on DuckDB, its use of liveness costs the time; otherwise a change that does not involve liveness does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Reverts 261823a (DuckDB scans without liveness) and af51447 (dictionary values kept for the cache's life). With either change the TPC-DS SF10 inventory queries stayed equally slow, and an unrelated PR measured against the same base shows the same numbers, so their cost comes from develop since that base rather than from this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A VortexFile opened with caching shares one ScanCache across its scans. Each scan's claims were released when it ended, which drained every pin, so the next scan decoded every dictionary again, as random-access takes on a cached file do once per take. The file's scan cache now holds a claim on all of its rows for as long as the file caches, so the values its scans decode outlive each scan, as before liveness. A file without caching, and an engine's per-query cache, still drop them once their scan's work retires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The previous commit kept a caching file's dictionary values by holding a claim on the file's rows beside its ScanCache. A caller holding only the ScanFile lost that claim with the file, and the file's scans still took the liveness lock for claims that could never evict anything. ScanCache::retaining_values builds a cache whose values live as long as it does and which tracks no liveness, so its scans run as before liveness, with no claims. ScanCache and PreparedScan::liveness return an Option, and the engines admit through Run::admit_tracked, which tracks liveness only when the cache has one. VortexFile::with_caching uses a retaining cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Say which ScanCache constructors share values only between overlapping scans and that retaining_values keeps them for the cache's life, state how long a cached file's V2 state lives, rename the vortex-file test to what it proves, and reset only the plans when a cached file's segment source changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (0.974x ➖, 0↑ 0↓)
random-access / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
random-access / parquet / ns (1.001x ➖, 0↑ 0↓)
random-access / lance / ns (0.989x ➖, 0↑ 0↓)
random-access / litdata / ns (0.991x ➖, 0↑ 0↓)
|
A V2 scan kept up to 64 MiB of each file's recently read segment bytes in a least recently used cache behind one mutex, which every segment request took. Scans that track liveness now keep the arrays decoded from the segments neighbouring chunks share instead: a segment a chunk slices, which spans a chunk boundary, and a zone table every chunk reads whole. Each array is pinned to the file rows it covers and dropped once no live work claims them, so later chunks reuse it without reading or decoding the segment again. Lookups take a read lock on one of 32 shards rather than updating a shared LRU. A 1 GiB budget only bounds memory if work is claimed far ahead of its reads. Caches without liveness, as for VortexFile::with_caching, keep retaining segment bytes as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Stacked on #11201 (driver-owned liveness). It targets
developso the SQL benchmarks run. Review the commits after the merge of #11201:0ff47fa3,63656cf0and2ff72324. The two commits markedexperiment:were benchmark-only and are reverted by96d17fc5, so the tree is identical to2ff72324.This uses liveness for one thing: limiting how long the V2
ValueCachekeeps dictionary values. Today every decoded dictionary stays for the whole life of theScanCache. With this change, each one is pinned to the file rows its dictionary covers and dropped once no live work claims any of those rows. The goal is lower memory at the same speed.Changes
ValueCache(pipeline/values.rs):with_liveness(liveness)pins each recorded value to its file rows. Dropping the pin evicts the value. A generation check stops an old pin from evicting a value recorded again later. A value recorded while no work claims its rows is kept as long as the cache, as before, so scans driven without liveness don't re-decode dictionaries for every chunk.compile_atadds a chunk's offset inside a concatenation, andcompile_detachedis used for dictionary values and zone tables, whose rows aren't file rows.TakeandSharepin tofile_rows(plan.row_count()), or for the whole file when the rows aren't known.ScanCache: owns the file'sLiveness, builds itsValueCachefrom it, and exposesliveness().PreparedScan::liveness()passes it to the engines.Progress::Spawncarries the child'sRowClaim, andRun::admit_spawnedadmits it with that claim. DataFusion, DuckDB andvortex-fileadmit with the cache's liveness, so a dictionary stays while any chunk over its rows is planned or running.ChunkPlanner::remaining_rowsnarrows the root's claim as it hands chunks out.63656cf0,2ff72324):Arcinstead of locking the liveness to claim and release its own rows.RowClaim::split_front, which never touches the liveness.dictionary_values_live_while_their_rows_are_claimedcovers two dictionaries: one evicted after its claim drops, and the keep-for-the-cache's-life fallback.a_split_claim_keeps_its_rows_until_each_part_is_dropped.releasedcase showing values are read again once nothing claims them.Benchmarks
Measured at
2ff72324againstdevelop. Most suites come out the same or slightly faster after subtracting the Parquet drift: ClickBench −0.5%, FineWeb −2.1%, TPC-H SF1 +0.4% and SF10 −0.7%, TPC-DS SF1 +0.4%. TPC-H on S3 is marked too noisy. PolarSignals has no Parquet control.TPC-DS SF10 inventory queries are slower against the base
4598f857: DuckDB q21 +26%, q37 +10%, q82 +11%. This doesn't appear to come from this PR:So they most likely come from a
developcommit after that base, probably #11331, which changes how integer columns get compressed. That's my best guess and isn't confirmed.API Changes
ScanCacheandPreparedScangainliveness().Progress::Spawngains a third field,Option<RowClaim>;Run::admit_spawnedandRowClaim::split_frontare added. Dictionary values are no longer kept between scans of a sharedScanCacheunless a caller holds a claim on their rows.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBBmU6aRNmSqEsMdKGW