Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't workingmemoryMemory accounting or memory usageMemory accounting or memory usage
Description
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
LiquidCache::reset(core.rs:376-379) only clears the index and zeroes both counters. It does not:- delete store objects;
- clear the policy queues;
- fence work that is in flight.
- It is reachable on a live server through the admin
/reset_cachehandler (src/datafusion-server/src/admin_server/handlers.rs:50-55). Theunsafedoc onsrc/datafusion/src/cache/mod.rs:532says to call it only when nothing else uses the cache, but nothing enforces that. - A spill that finishes or is dropped after the reset wraps the disk counter (cache: budget counters wrap below zero #72), and that disables the disk tier.
Evidence.
| Case | Result |
|---|---|
Spill then reset() |
The store still holds 1 object that no record names |
Server /reset_cache |
Reports 0 disk usage while 16 objects remain (593,920-byte file) |
Spill dropped after reset() |
Disk usage 18,446,744,073,709,550,128; a 528-byte spill is then refused with CacheFull |
Spill that installs its record after reset() |
Usage 0 while the index charges 1,488; removing the record wraps the counter |
Queues after reset() |
The policy still offers 4 memory and 6 disk victims for an empty index |
reset() racing inserts |
Budget 5 entries short of the index |
Tests.
| Test | File | Command | Flags |
|---|---|---|---|
audit_reset_deletes_disk_objects |
src/core/src/cache/core.rs:4063 |
cargo +1.96.0 test -p liquid-cache --lib audit_reset_deletes_disk_objects |
Inline |
audit_reset_under_a_dropped_spill_keeps_the_disk_counter_in_bounds |
src/core/src/cache/core.rs:3185 |
cargo +1.96.0 test -p liquid-cache --lib audit_reset_under_a_dropped_spill -- --nocapture |
|
audit_reset_under_a_finishing_spill_leaves_its_record_charged |
src/core/src/cache/core.rs:3241 |
cargo +1.96.0 test -p liquid-cache --lib audit_reset_under_a_finishing_spill -- --nocapture |
|
audit_reset_leaves_the_policy_no_victims |
src/core/src/cache/core.rs:3278 |
cargo +1.96.0 test -p liquid-cache --lib audit_reset_leaves_the_policy_no_victims -- --nocapture |
|
audit_reset_racing_inserts_leaves_the_memory_budget_matching_the_index |
src/core/src/cache/core.rs:3319 |
cargo +1.96.0 test -p liquid-cache --lib audit_reset_racing_inserts -- --nocapture |
P |
audit_reset_cache_handler_empties_the_disk_it_reports_as_empty |
src/datafusion-server/src/admin_server/handlers_audit_tests.rs:48 |
cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_reset_cache_handler_empties_the_disk_it_reports_as_empty -- --nocapture |
Inline test (uses the existing flushed and assert_disk_matches_index):
#[tokio::test]
async fn audit_reset_deletes_disk_objects() {
let entry = EntryID::from(1010usize);
let array: ArrayRef = Arc::new(Int32Array::from_iter_values(0..256));
let (cache, store, _directory) = flushed(entry, &array).await;
assert_eq!(store.len().await.unwrap(), 1);
cache.reset();
assert!(!cache.contains(&entry, 0));
assert_eq!(cache.budget.disk_usage_bytes(), 0);
assert_disk_matches_index(&cache, &store).await;
}Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingmemoryMemory accounting or memory usageMemory accounting or memory usage
Activity