Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't workingperformanceWasted work, over-eviction or excess IOWasted work, over-eviction or excess IO
Description
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
insert_innerevictsfind_memory_victim(8)(core.rs:557), andwrite_batch_to_diskremoves all offind_disk_victim(8)(core.rs:1073-1079). Neither checks how many bytes are actually needed.- All 8 victims are also serialized before any of their writes completes, which multiplies the transients in Memory budget excludes hydrate/squeeze transients: tier reports at limit while the process holds 3x that #43.
Evidence.
- Inserting one 352-byte entry into a memory tier full of 20 such entries evicted 8 of them.
- Spilling one 720-byte record into a disk tier full of 20 such records removed 8 of them, leaving disk usage at 9,360 of 14,400.
Tests.
| Test | File | Command | Flags |
|---|---|---|---|
audit_one_insert_into_full_memory_evicts_only_what_it_needs |
src/core/src/cache/core.rs:3632 |
cargo +1.96.0 test -p liquid-cache --lib audit_one_insert_into_full_memory_evicts_only_what_it_needs -- --nocapture |
|
audit_one_spill_into_full_disk_removes_only_what_it_needs |
src/core/src/cache/core.rs:3678 |
cargo +1.96.0 test -p liquid-cache --lib audit_one_spill_into_full_disk_removes_only_what_it_needs -- --nocapture |
Inline |
Inline test (uses the existing arrow_copy, cache_over_counted_store and spill):
#[tokio::test]
async fn audit_one_spill_into_full_disk_removes_only_what_it_needs() {
let array: ArrayRef = Arc::new(Int32Array::from_iter_values(0..64));
let copy = arrow_copy(&array);
let (cache, store, _directory) = cache_over_counted_store(1 << 20, 20 * copy).await;
let ids: Vec<EntryID> = (9730usize..9750).map(EntryID::from).collect();
for id in &ids {
cache.insert(*id, 0, array.clone()).await.unwrap();
}
cache.flush_all_to_disk().await.unwrap();
assert!(ids.iter().all(|id| cache.contains(id, 0)));
assert_eq!(cache.budget.disk_usage_bytes(), 20 * copy);
let extra = EntryID::from(9750usize);
cache.insert(extra, 0, array.clone()).await.unwrap();
spill(&cache, extra, &array).await;
let removed = ids.iter().filter(|id| !cache.contains(id, 0)).count();
let usage = cache.budget.disk_usage_bytes();
println!(
"record size={copy}, max_disk={}; spilling 1 more removed {removed} of 20; \
disk usage now {usage}, store holds {} objects",
20 * copy,
store.len().await.unwrap(),
);
assert!(
removed <= 1,
"spilling one {copy}-byte record into a disk tier full of 20 such records \
removed {removed} of them; 1 was needed"
);
}(The println! is shortened from the branch. The assertion is unchanged.)
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingperformanceWasted work, over-eviction or excess IOWasted work, over-eviction or excess IO
Activity
Merge origin/main into fix/eviction-accounting
fix(cache): keep the memory budget matching the index (#100)