Repository navigation
Description
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
RowGroupSnapshots(src/datafusion/src/cache/mod.rs:66-87) holds anArcto each decoded batch, with no remove path and no charge.- The prefetch path snapshots every batch unconditionally (
src/datafusion/src/reader/plantime/morselizer.rs:847). The reader snapshots onCacheFull(src/datafusion/src/reader/runtime/liquid_cache_reader.rs:435-436). - Eviction subtracts the entry from the budget while the snapshot still holds it, so the budget drops and real memory does not.
- The headroom gates (
morselizer.rs:695-707,:794-801) read the very counter that excludes snapshots, and they take no reservation. The first row group of each file is not gated (morselizer.rs:653-654); only the per-batch brake applies. - This is a concrete case of the "evicted while an
Arcis still alive" candidate that Memory budget excludes hydrate/squeeze transients: tier reports at limit while the process holds 3x that #43 lists as unestablished.
Evidence.
| Scenario | Snapshot bytes | Cache budget used | Limit |
|---|---|---|---|
| Single prefetch | 448 | 448 | 448 |
| 4 concurrent prefetches | 11,648 | 7,168 | 7,168 |
Reader after CacheFull, 16 batches |
3,584 | 0 | 1 |
flush_data while 64 snapshot entries are alive |
7,168 | 0 |
- For the 4 concurrent prefetches, snapshots alone are 1.62x the limit and snapshots plus cache are 2.6x.
- The
CacheFullrow holds the whole row group, growing linearly until the stream yieldsNone. - After
flush_data, all 64 evicted arrays were still allocated. - First row group in a full cache: 896 snapshot bytes, so snapshots plus cache are 1.12x the limit.
- Fixtures are 4-row Int32 batches, so real-workload scaling is inferred.
Tests. All run in a single-threaded runtime because the registry is thread-local, so use --test-threads=1.
| Test | File | Command | Flags |
|---|---|---|---|
repro_prefetch_snapshots_exceed_memory_budget |
src/datafusion/src/reader/plantime/morselizer.rs:1943 |
cargo +1.96.0 test -p liquid-cache-datafusion --lib repro_prefetch_snapshots_exceed_memory_budget -- --nocapture --test-threads=1 |
|
repro_concurrent_prefetch_snapshots_exceed_memory_budget |
src/datafusion/src/reader/plantime/morselizer.rs:2055 |
cargo +1.96.0 test -p liquid-cache-datafusion --lib repro_concurrent_prefetch_snapshots_exceed_memory_budget -- --nocapture --test-threads=1 |
|
repro_cache_full_reader_parks_whole_row_group_in_snapshots |
src/datafusion/src/reader/plantime/morselizer.rs:2000 |
cargo +1.96.0 test -p liquid-cache-datafusion --lib repro_cache_full_reader_parks_whole_row_group_in_snapshots -- --nocapture --test-threads=1 |
|
audit_evicting_snapshotted_entries_releases_or_charges_them |
src/datafusion/src/reader/plantime/morselizer.rs:2155 |
cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_evicting_snapshotted_entries_releases_or_charges_them -- --nocapture --test-threads=1 |
|
audit_first_row_group_prefetch_is_bounded_by_headroom |
src/datafusion/src/reader/plantime/morselizer.rs:2287 |
cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_first_row_group_prefetch_is_bounded_by_headroom -- --nocapture --test-threads=1 |
The tests use the test-only snapshot registry.
Heads-up: the existing test snapshots_survive_eviction (morselizer.rs:1553) asserts this pinning as intended behaviour. A fix has to either charge snapshots or release them, and that test changes either way.
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity
fix(cache): count memory that row-group snapshots hold