Skip to content

fix(cache): keep admitting once a memory-only cache is full - #103

Merged
anoop-narang merged 6 commits into
mainfrom
fix/memory-only-eviction
Nov 18, 2026
Merged

anoop-narang merged 6 commits into
mainfrom
fix/memory-only-eviction

Conversation

@anoop-narang

Copy link
Copy Markdown
Collaborator

Summary

A cache built with with_max_disk_bytes(0) (memory only) stopped admitting new data once its memory filled: it kept whatever it cached first and refused every later insert. This change makes a full memory-only cache drop old entries to make room, keeps filtered queries correct while it does, and fixes two accounting races that the new drop path made easier to hit.

Problem

Eviction moves an Arrow entry to Liquid in memory, and a Liquid entry to disk. With no disk tier, the disk write of a Liquid victim failed with CacheFull, evict_victims returned the error, and insert_inner gave up without freeing anything. Once memory held only Liquid entries, every insert was refused.

Reproduced with LiquidCacheLocalBuilder, an 8 MiB memory budget, max_disk_bytes(0), default policies, and an admission gate of 4/1/1/strict: after six tables fill the cache, a seventh table queried three times caches 1 of its 25 batches, with 72 refused inserts. With a disk tier the same table caches 25 of 25.

Change

Eviction (src/core/src/cache/core.rs)

  • A memory victim the disk cannot take is dropped (drop_memory_victim). With no disk tier, a Liquid victim is dropped before the eviction policy serialises it for a write that cannot happen.
  • The drop removes the slot only if it still holds the exact entry the eviction picked (pointer equality), so a newer write of the key, under any identity, survives.
  • If a write installs the key again between the drop's removal and its notice to the policy, the notice would take the new entry out of the policy too, leaving it unevictable. The drop files the key again when the index holds it.
  • An entry larger than the whole memory tier no longer evicts anything before it is refused (it previously emptied the cache first), and with no disk tier a Liquid entry is refused before it is serialised.

Accounting (try_insert, BudgetAccounting)

  • try_insert charged the budget against the slot it read, then installed over whatever slot was there by then. A concurrent drop or replacement of the key in between left the budget off by that slot's size; enough of that can wrap the counter and refuse every insert. Only growth over the slot read is now reserved up front, and the rest is settled against the slot actually displaced. The budget can over-read for a moment, never under-read. try_reserve_memory uses a saturating add.

Reader (src/datafusion/src/reader/runtime/liquid_cache_reader.rs)

  • Dropping victims means a batch's earlier columns can leave the cache while its later ones are inserted. The reader evaluated predicates from the cache after filling it, and failed with predicate unavailable after materialization (seen on Boolean and timestamp-with-time-zone filter columns, which Liquid does not encode). It still tries the cache first, and now falls back to evaluating the predicates on the record batch it just read.

Behaviour changes worth knowing

  • Eviction still takes up to 8 victims per failed insert. With a disk tier those spill; with none they are dropped, so a full memory-only cache runs a few entries under its budget.
  • Under memory pressure, a memory-only cache retains columns of types Liquid cannot encode (Boolean, timestamp with time zone, ...) only briefly: the Arrow queue is popped before the Liquid queue, and such an entry cannot be transcoded, so it is dropped. Queries stay correct; those columns are re-read from parquet. Not addressed here.
  • Two snapshots changed. insert_wont_fit_cache: the oversized entry no longer evicts entry 0 before going to disk. test_provide_schema2: an oversized batch is transcoded instead of first spilling two other entries to disk; results are unchanged.

Tests

  • src/datafusion-local/tests/memory_only_freeze.rs: the reproduction above, and its disk-tier control.
  • src/datafusion-local/tests/memory_only_workloads.rs, every result checked against DataFusion with no cache, and after each phase the charged memory checked against the bytes the index holds:
    • filtered queries on timestamp-with-time-zone, Boolean, Int64 and Utf8 columns once full;
    • 24 tables in rotation for 4 passes: the table queried last is always cached;
    • 8 concurrent query loops, then a table not yet queried is cached;
    • 120 queries cancelled at varying points, then a table not yet queried is cached.
  • Core unit tests: a full memory-only cache admits 200 inserts in a row within budget; an oversized entry leaves the cache as it was; a drop does not remove an entry written after the victim was picked; a key written again between a drop and its notice stays evictable (deterministic, through a policy hook); 16 writers on 8 keys keep the budget equal to what the index holds and every cached key known to the policy.
  • Each new test was run with its part of the fix switched off and fails: the rotation, concurrency and cancellation tests on main (nothing cached after the first pass); the filter test with the cache fix but not the reader fix (predicate unavailable after materialization); the accounting stress test without the settle (5 of 5 runs off by up to 13 KB); the re-file test without the re-file.
  • Three reader tests in morselizer.rs asserted the old behaviour (a full cache keeps its first batches and refuses the last); they now assert that the last batch is cached.
  • cargo fmt --check, cargo clippy --workspace --exclude dev-tools --all-targets --all-features -- -D warnings, and cargo test --workspace --exclude dev-tools pass on Rust 1.96.

With no disk tier, eviction spilled a Liquid victim to disk, the write
failed with CacheFull, and the insert was refused without freeing
anything. A full cache then kept what it cached first and refused every
new entry.

Eviction now drops a memory victim the disk cannot take, and drops a
Liquid victim before serialising it when the disk tier is zero. The
drop removes the slot only if it still holds the entry the eviction
picked, so a newer write of the key survives. An entry larger than the
whole memory tier no longer evicts anything before it is refused.

Dropping victims means a batch's earlier columns can leave the cache
while its later ones are inserted. The reader evaluated predicates from
the cache after filling it, and failed with "predicate unavailable
after materialization". It now falls back to the batch it just read.
try_insert charged the memory budget against the slot it read, then
installed over whatever slot was there by then. A concurrent drop or
replacement of the key in between left the budget off by that slot's
size, and enough of that can wrap the counter and refuse every insert.
The charge is now settled against the slot actually displaced.

A drop that removed a key and then told the policy could also remove a
newer entry of the key from the policy, leaving it unevictable. The
drop now files the key again if the index holds it.
Releasing a shrink before the insert could under-read the budget while
another write of the key raced it. Only growth is now reserved up
front, and the rest is settled once the insert knows what it displaced,
so the budget can over-read for a moment but never under-read.

Also test the re-file of a key written again while its victim drops,
and note that a policy's notify_insert must be an upsert.
@anoop-narang
anoop-narang requested a review from a team as a code owner November 17, 2026 20:32
@anoop-narang
anoop-narang requested review from rohan-hotdata and removed request for a team November 17, 2026 20:32
Comment thread src/core/src/cache/core.rs Resolved
claude[bot]
claude Bot previously approved these changes Nov 17, 2026
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.63958% with 18 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...tafusion/src/reader/runtime/liquid_cache_reader.rs 74.41% 8 Missing and 3 partials ⚠️
src/core/src/cache/core.rs 96.88% 7 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

📊 Benchmark Comparison

Current: 78279f05 (Liquid) vs Baseline: 78279f05 (DataFusionDefault)

Query Cold Time Δ Warm Time Δ CPU Time Δ
Q1 2.0ms (2.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q2 5.0ms (4.0ms) +25.0% 3.0ms (4.0ms) -25.0% 6.0ms (5.0ms) +20.0%
Q3 11.0ms (11.0ms) +0.0% 4.5ms (9.0ms) -50.0% 3.0ms (18.5ms) -83.8%
Q4 10.0ms (9.0ms) +11.1% 4.0ms (7.5ms) -46.7% 2.5ms (14.5ms) -82.8%
Q5 46.0ms (45.0ms) +2.2% 36.0ms (40.5ms) -11.1% 4.0ms (16.5ms) -75.8%
Q6 164.0ms (93.0ms) +76.3% 67.5ms (87.5ms) -22.9% 18.0ms (63.0ms) -71.4%
Q7 1.0ms (1.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q8 7.0ms (5.0ms) +40.0% 5.0ms (4.5ms) +11.1% 8.0ms (5.0ms) +60.0%
Q9 77.0ms (69.0ms) +11.6% 63.5ms (68.0ms) -6.6% 5.0ms (31.0ms) -83.9%
Q10 80.0ms (70.0ms) +14.3% 61.0ms (73.0ms) -16.4% 7.0ms (48.0ms) -85.4%
Q11 77.0ms (18.0ms) +327.8% 63.0ms (18.5ms) +240.5% 175.5ms (24.0ms) +631.2%
Q12 80.0ms (21.0ms) +281.0% 65.5ms (22.0ms) +197.7% 178.0ms (28.0ms) +535.7%
Q13 257.0ms (99.0ms) +159.6% 110.0ms (100.5ms) +9.5% 120.5ms (64.0ms) +88.3%
Q14 424.0ms (119.0ms) +256.3% 132.0ms (119.5ms) +10.5% 86.0ms (82.0ms) +4.9%
Q15 330.0ms (87.0ms) +279.3% 131.0ms (95.0ms) +37.9% 121.5ms (81.0ms) +50.0%
Q16 122.0ms (93.0ms) +31.2% 112.5ms (93.0ms) +21.0% 6.0ms (17.5ms) -65.7%
Q17 504.0ms (187.0ms) +169.5% 252.0ms (189.5ms) +33.0% 70.5ms (81.5ms) -13.5%
Q18 581.0ms (199.0ms) +192.0% 256.0ms (190.0ms) +34.7% 72.5ms (80.5ms) -9.9%
Q19 1.00s (367.0ms) +172.8% 402.0ms (354.5ms) +13.4% 71.0ms (124.0ms) -42.7%
Q20 10.0ms (9.0ms) +11.1% 3.0ms (9.0ms) -66.7% 7.0ms (15.5ms) -54.8%
Q21 2.19s (146.0ms) +1400.7% 430.5ms (147.0ms) +192.9% 224.5ms (232.5ms) -3.4%
Q22 1.59s (142.0ms) +1021.1% 544.5ms (141.5ms) +284.8% 146.5ms (287.0ms) -49.0%
Q23 3.65s (391.0ms) +833.8% 1.34s (402.5ms) +231.9% 434.0ms (627.5ms) -30.8%
Q24 24.39s (775.0ms) +3047.4% 1.02s (813.0ms) +25.0% 452.0ms (2.28s) -80.2%
Q25 315.0ms (65.0ms) +384.6% 29.0ms (48.5ms) -40.2% 50.0ms (95.5ms) -47.6%
Q26 181.0ms (36.0ms) +402.8% 47.5ms (36.5ms) +30.1% 120.0ms (66.0ms) +81.8%
Q27 307.0ms (50.0ms) +514.0% 34.5ms (48.5ms) -28.9% 74.5ms (95.0ms) -21.6%
Q28 1.36s (176.0ms) +672.2% 723.5ms (180.5ms) +300.8% 153.5ms (235.5ms) -34.8%
Q29 1.96s (814.0ms) +140.9% 963.0ms (815.0ms) +18.2% 337.0ms (291.5ms) +15.6%
Q30 24.0ms (22.0ms) +9.1% 18.0ms (22.0ms) -18.2% 6.0ms (17.5ms) -65.7%
Q31 595.0ms (91.0ms) +553.8% 75.0ms (95.0ms) -21.1% 71.5ms (121.0ms) -40.9%
Q32 801.0ms (84.0ms) +853.6% 102.0ms (87.5ms) +16.6% 85.0ms (120.0ms) -29.2%
Q33 326.0ms (305.0ms) +6.9% 275.5ms (300.0ms) -8.2% 8.5ms (61.5ms) -86.2%
Q34 1.50s (384.0ms) +290.1% 562.5ms (392.5ms) +43.3% 114.0ms (226.0ms) -49.6%
Q35 1.49s (384.0ms) +288.3% 670.0ms (400.0ms) +67.5% 113.0ms (227.0ms) -50.2%
Q36 89.0ms (92.0ms) -3.3% 74.5ms (88.5ms) -15.8% 4.0ms (19.0ms) -78.9%
Q37 1.28s (86.0ms) +1386.0% 65.5ms (92.0ms) -28.8% 21.0ms (62.5ms) -66.4%
Q38 78.0ms (37.0ms) +110.8% 24.5ms (38.0ms) -35.5% 16.0ms (20.0ms) -20.0%
Q39 289.0ms (39.0ms) +641.0% 23.0ms (40.0ms) -42.5% 12.5ms (59.5ms) -79.0%
Q40 788.0ms (163.0ms) +383.4% 208.0ms (158.5ms) +31.2% 45.0ms (109.5ms) -58.9%
Q41 24.0ms (16.0ms) +50.0% 9.5ms (18.0ms) -47.2% 6.0ms (17.0ms) -64.7%
Q42 17.0ms (17.0ms) +0.0% 8.0ms (14.0ms) -42.9% 6.0ms (12.0ms) -50.0%
Q43 14.0ms (12.0ms) +16.7% 8.5ms (11.5ms) -26.1% 6.0ms (8.0ms) -25.0%

⚠️ LiquidCache is slower on 17 queries (warm)

  • Q28: warm +300.8% (723.5ms vs 180.5ms)
  • Q22: warm +284.8% (544.5ms vs 141.5ms)
  • Q11: warm +240.5% (63.0ms vs 18.5ms)
  • Q23: warm +231.9% (1.34s vs 402.5ms)
  • Q12: warm +197.7% (65.5ms vs 22.0ms)
  • Q21: warm +192.9% (430.5ms vs 147.0ms)
  • Q35: warm +67.5% (670.0ms vs 400.0ms)
  • Q34: warm +43.3% (562.5ms vs 392.5ms)
  • Q15: warm +37.9% (131.0ms vs 95.0ms)
  • Q18: warm +34.7% (256.0ms vs 190.0ms)
  • Q17: warm +33.0% (252.0ms vs 189.5ms)
  • Q40: warm +31.2% (208.0ms vs 158.5ms)
  • Q26: warm +30.1% (47.5ms vs 36.5ms)
  • Q24: warm +25.0% (1.02s vs 813.0ms)
  • Q16: warm +21.0% (112.5ms vs 93.0ms)
  • Q29: warm +18.2% (963.0ms vs 815.0ms)
  • Q32: warm +16.6% (102.0ms vs 87.5ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.

Comment thread src/core/src/cache/core.rs Resolved
claude[bot]
claude Bot previously approved these changes Nov 17, 2026
@anoop-narang
anoop-narang merged commit 1beb151 into main Nov 18, 2026
14 checks passed
zfarrell added a commit that referenced this pull request Nov 18, 2026
Brings in #103, which fixes the memory-only freeze and settles
try_insert's charge against the slot actually displaced.

- try_insert: #103's growth-only reservation and settle replace ours
  (#67); settle_memory_release goes, and its test becomes one for
  #103's charge_memory.
- budget.rs: our reserve/release keep the overflow and underflow guards
  (#72); charge_memory saturates through update() rather than fetch_add.
- insert_inner: #103's guard for an entry larger than the memory tier,
  then find_memory_victims_for (#70).
- Two #66 tests assumed a full memory-only cache refuses inserts; they
  now check it admits and keeps the rest evictable.
- Snapshots provide_schema2 and url_selection_and_ordering: victims by
  need transcode fewer Arrow entries, and shift one more entry through
  the latter's hydrate rotation.
@anoop-narang
anoop-narang deleted the fix/memory-only-eviction branch November 18, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant