Skip to content

fix(cache): keep the memory budget matching the index - #100

Merged
anoop-narang merged 9 commits into
mainfrom
fix/eviction-accounting
Nov 18, 2026
Merged

anoop-narang merged 9 commits into
mainfrom
fix/eviction-accounting

Conversation

@zfarrell

@zfarrell zfarrell commented Nov 17, 2026 •

Copy link
Copy Markdown

Keeps the memory budget counting what the index holds, and keeps eviction from taking more than an insert needs.

  • Budget counters no longer wrap: a release past zero stops at zero, is counted in the new budget_underflows runtime stat and logged once; a reservation that would overflow usize is refused. (cache: budget counters wrap below zero #72)
  • An insert charges its entry in full before the entry enters the index, and releases the slot it displaced after, so a removal that follows at once never finds the entry uncharged.
  • Eviction takes victims until what they hold covers what the insert lacks (at most 8), instead of a fixed 8; a replacement asks only for its growth over the slot it replaces. (cache: eviction always takes 8 victims #70)
  • Victims of an eviction dropped before it finishes are offered to the policy again (VictimLease). Only an eviction waiting on a disk write can be dropped part-way, so this matters with a disk tier. (cache: failed or cancelled eviction strands its victims #66)
  • evict_victims / evict_victim_inner no longer return a Result: since fix(cache): keep admitting once a memory-only cache is full #103 a victim the disk cannot take is dropped, so eviction cannot fail.

Settling an insert against the slot it displaced (#67) is already on main through #103 and is only tested here. #69 (one failed victim refusing the insert) cannot happen since #103.

Closes #72
Closes #66
Refs #70
Refs #67
Refs #69

https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar

A release larger than the usage wrapped the counter to near usize::MAX,
after which the tier refused every reservation, and a reservation whose
sum overflowed panicked in debug and wrapped in release. Releases now
stop at zero and reservations that overflow are refused.

Closes #72
try_insert moved the memory budget by the size of the slot it read,
then installed over whatever held the key by then. Two inserts of one
key, or an eviction racing a re-insert, left the budget over- or
under-counting the index for good. The insert now corrects the budget
by the size of the slot it actually displaced.

Closes #67
insert_inner takes up to 8 victims off the policy's queues, and only a
victim whose eviction installs its new form is filed again. A victim
whose eviction failed, or never ran because the insert was dropped,
stayed in the index and charged with no queue offering it.

With no disk tier this is every refused insert: a Liquid victim has
nowhere to go, so evict_victims fails with CacheFull. Victims are now
held in a lease that files each unfinished one back with the policy.

Closes #66
evict_victims failed the whole insert as soon as any victim failed, even
when other victims had already freed enough memory. With no disk tier a
batch of victims mixes Arrow entries, which transcode, and Liquid ones,
which cannot move, so an insert that fit was refused with CacheFull. It
now fails only when no victim was evicted, and the insert retries.

Closes #69
insert_inner always took 8 memory victims, whatever the insert needed.
One insert into a full tier of Arrow entries transcoded 8 of them, each
costing an encode and leaving data slower to read. Victims are now taken
one at a time until what they hold covers what the budget lacks, up to
8; an insert that still lacks room comes back for more.

Two datafusion tests relied on the 8-victim batch to transcode both
their columns; their dummy column is now wide enough to need both.

Refs #70
claude[bot]
claude Bot previously approved these changes Nov 17, 2026
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.00797% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/core/src/cache/core.rs 97.85% 7 Missing and 1 partial ⚠️
src/core/src/cache/budget.rs 98.29% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@zfarrell
zfarrell marked this pull request as ready for review November 17, 2026 20:35
@zfarrell
zfarrell requested a review from a team as a code owner November 17, 2026 20:35
@zfarrell
zfarrell requested review from eddietejeda and removed request for a team November 17, 2026 20:35
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

📊 Benchmark Comparison

Current: 55cd8194 (Liquid) vs Baseline: 55cd8194 (DataFusionDefault)

Query Cold Time Δ Warm Time Δ CPU Time Δ
Q1 3.0ms (2.0ms) +50.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q2 11.0ms (6.0ms) +83.3% 5.0ms (4.0ms) +25.0% 8.0ms (7.0ms) +14.3%
Q3 18.0ms (11.0ms) +63.6% 7.0ms (11.5ms) -39.1% 4.0ms (23.0ms) -82.6%
Q4 16.0ms (11.0ms) +45.5% 6.0ms (10.0ms) -40.0% 3.0ms (19.0ms) -84.2%
Q5 69.0ms (50.0ms) +38.0% 59.5ms (49.5ms) +20.2% 6.5ms (21.0ms) -69.0%
Q6 264.0ms (105.0ms) +151.4% 88.0ms (101.0ms) -12.9% 35.5ms (76.5ms) -53.6%
Q7 1.0ms (1.0ms) +0.0% 1.0ms (0.000ms) +inf% 0.000ms (0.000ms) +0.0%
Q8 10.0ms (6.0ms) +66.7% 6.5ms (6.0ms) +8.3% 10.5ms (6.5ms) +61.5%
Q9 114.0ms (92.0ms) +23.9% 94.0ms (80.0ms) +17.5% 9.0ms (39.0ms) -76.9%
Q10 117.0ms (90.0ms) +30.0% 90.5ms (86.0ms) +5.2% 9.0ms (58.5ms) -84.6%
Q11 46.0ms (24.0ms) +91.7% 23.5ms (23.0ms) +2.2% 44.5ms (30.5ms) +45.9%
Q12 49.0ms (30.0ms) +63.3% 24.5ms (27.5ms) -10.9% 47.5ms (38.5ms) +23.4%
Q13 279.0ms (112.0ms) +149.1% 118.0ms (110.5ms) +6.8% 60.0ms (79.0ms) -24.1%
Q14 576.0ms (138.0ms) +317.4% 147.5ms (137.0ms) +7.7% 83.5ms (97.5ms) -14.4%
Q15 348.0ms (105.0ms) +231.4% 121.5ms (104.5ms) +16.3% 75.5ms (94.0ms) -19.7%
Q16 143.0ms (109.0ms) +31.2% 140.5ms (113.0ms) +24.3% 6.5ms (21.0ms) -69.0%
Q17 631.0ms (223.0ms) +183.0% 248.0ms (226.0ms) +9.7% 65.0ms (98.5ms) -34.0%
Q18 623.0ms (223.0ms) +179.4% 246.0ms (223.5ms) +10.1% 66.5ms (96.0ms) -30.7%
Q19 1.05s (415.0ms) +152.0% 391.5ms (444.0ms) -11.8% 94.0ms (140.0ms) -32.9%
Q20 14.0ms (10.0ms) +40.0% 4.0ms (10.5ms) -61.9% 8.5ms (20.5ms) -58.5%
Q21 671.0ms (172.0ms) +290.1% 470.0ms (172.5ms) +172.5% 282.0ms (269.5ms) +4.6%
Q22 1.04s (172.0ms) +504.7% 622.5ms (166.5ms) +273.9% 168.0ms (332.5ms) -49.5%
Q23 2.22s (464.0ms) +378.2% 1.72s (466.0ms) +268.1% 545.5ms (733.0ms) -25.6%
Q24 26.50s (893.0ms) +2867.7% 984.0ms (912.0ms) +7.9% 630.0ms (2.51s) -74.9%
Q25 354.0ms (71.0ms) +398.6% 16.0ms (55.0ms) -70.9% 43.0ms (112.0ms) -61.6%
Q26 160.0ms (43.0ms) +272.1% 19.0ms (45.5ms) -58.2% 50.0ms (83.0ms) -39.8%
Q27 358.0ms (60.0ms) +496.7% 25.0ms (56.5ms) -55.8% 72.5ms (115.0ms) -37.0%
Q28 841.0ms (212.0ms) +296.7% 944.0ms (213.0ms) +343.2% 218.0ms (276.0ms) -21.0%
Q29 1.55s (973.0ms) +58.9% 1.16s (981.0ms) +18.2% 415.5ms (339.0ms) +22.6%
Q30 35.0ms (27.0ms) +29.6% 22.0ms (27.0ms) -18.5% 7.0ms (21.5ms) -67.4%
Q31 739.0ms (104.0ms) +610.6% 84.0ms (103.5ms) -18.8% 75.0ms (140.5ms) -46.6%
Q32 1.15s (103.0ms) +1020.4% 147.5ms (99.0ms) +49.0% 113.5ms (144.5ms) -21.5%
Q33 372.0ms (356.0ms) +4.5% 349.5ms (350.0ms) -0.1% 8.0ms (69.5ms) -88.5%
Q34 965.0ms (421.0ms) +129.2% 474.0ms (528.5ms) -10.3% 135.5ms (335.5ms) -59.6%
Q35 1.00s (488.0ms) +105.9% 462.5ms (417.5ms) +10.8% 128.5ms (264.0ms) -51.3%
Q36 106.0ms (131.0ms) -19.1% 100.5ms (110.5ms) -9.0% 5.5ms (22.0ms) -75.0%
Q37 325.0ms (101.0ms) +221.8% 78.5ms (98.0ms) -19.9% 27.5ms (71.0ms) -61.3%
Q38 75.0ms (49.0ms) +53.1% 27.0ms (45.0ms) -40.0% 16.0ms (23.5ms) -31.9%
Q39 296.0ms (46.0ms) +543.5% 31.0ms (46.0ms) -32.6% 17.5ms (70.5ms) -75.2%
Q40 724.0ms (189.0ms) +283.1% 234.5ms (184.0ms) +27.4% 61.5ms (128.0ms) -52.0%
Q41 23.0ms (19.0ms) +21.1% 11.0ms (20.5ms) -46.3% 7.0ms (18.5ms) -62.2%
Q42 24.0ms (20.0ms) +20.0% 10.5ms (19.0ms) -44.7% 8.0ms (16.5ms) -51.5%
Q43 21.0ms (15.0ms) +40.0% 12.0ms (15.0ms) -20.0% 8.5ms (10.5ms) -19.0%

⚠️ LiquidCache is slower on 13 queries (warm)

  • Q7: warm +inf% (1.0ms vs 0.000ms)
  • Q28: warm +343.2% (944.0ms vs 213.0ms)
  • Q22: warm +273.9% (622.5ms vs 166.5ms)
  • Q23: warm +268.1% (1.72s vs 466.0ms)
  • Q21: warm +172.5% (470.0ms vs 172.5ms)
  • Q32: warm +49.0% (147.5ms vs 99.0ms)
  • Q40: warm +27.4% (234.5ms vs 184.0ms)
  • Q2: warm +25.0% (5.0ms vs 4.0ms)
  • Q16: warm +24.3% (140.5ms vs 113.0ms)
  • Q5: warm +20.2% (59.5ms vs 49.5ms)
  • Q29: warm +18.2% (1.16s vs 981.0ms)
  • Q9: warm +17.5% (94.0ms vs 80.0ms)
  • Q15: warm +16.3% (121.5ms vs 104.5ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.

Rust 1.100 deprecates AtomicUsize::fetch_update in favour of try_update,
and CI's clippy denies warnings. try_update is missing from older
toolchains and from shuttle's atomics, so the update loop is written out
with compare_exchange_weak.
claude[bot]
claude Bot previously approved these changes Nov 17, 2026
Brings in #103, which fixes the memory-only freeze and settles
try_insert's charge against the slot actually displaced.

- try_insert: #103's growth-only reservation and settle replace ours
  (#67); settle_memory_release goes, and its test becomes one for
  #103's charge_memory.
- budget.rs: our reserve/release keep the overflow and underflow guards
  (#72); charge_memory saturates through update() rather than fetch_add.
- insert_inner: #103's guard for an entry larger than the memory tier,
  then find_memory_victims_for (#70).
- Two #66 tests assumed a full memory-only cache refuses inserts; they
  now check it admits and keeps the rest evictable.
- Snapshots provide_schema2 and url_selection_and_ordering: victims by
  need transcode fewer Arrow entries, and shift one more entry through
  the latter's hydrate rotation.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merge of origin/main is consistent with this branch. In try_insert, the reserve-then-settle split charges or releases the gap between the slot read and the slot displaced, through charge_memory and the saturating release. evict_victim_inner now always returns Ok, so evict_victims fails only for an empty victim list. Dropped victims leave the VictimLease through evicted, and a newer entry is filed again by drop_memory_victim. CI was still running when this review started, so no test result is confirmed here.

claude[bot]
claude Bot previously approved these changes Nov 18, 2026
Resolve against the snapshot accounting from #104: sweep_held releases
through the saturating release, and try_reserve_memory sweeps held
snapshots and retries once when the reservation is refused.

Claude-Session: https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar
…rowth

- try_insert charges the part of an entry it did not reserve before the
  entry enters the index, and releases the slot it displaced after. A
  removal that follows the insert at once never finds the entry uncharged,
  so a saturating release cannot clip and leave a permanent over-count.
- An insert that replaces a key sizes its eviction by the growth over the
  slot it replaces, which is what try_insert reserves, not the whole entry.
- evict_victims and evict_victim_inner no longer return a Result: a victim
  the disk cannot take is dropped, so eviction cannot fail, and the partial
  failure handling was unreachable. VictimLease still re-offers victims of
  an eviction dropped while it waits on a disk write.
- A budget release larger than the usage is counted in the new
  budget_underflows runtime stat and logged once.
- Tests: try_reserve_memory sweeps dropped snapshots and retries; a
  replacement evicts only for its growth.

Claude-Session: https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar
@anoop-narang
anoop-narang merged commit a3996c7 into main Nov 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cache: budget counters wrap below zero cache: failed or cancelled eviction strands its victims

2 participants