Skip to content

fix(cache): forget files that hold nothing - #102

Merged
anoop-narang merged 2 commits into
mainfrom
fix/side-table-growth
Nov 19, 2026
Merged

anoop-narang merged 2 commits into
mainfrom
fix/side-table-growth

Conversation

@zfarrell

@zfarrell zfarrell commented Nov 17, 2026 •

Copy link
Copy Markdown

The per-file registry and lineage hints now drop files that no handle refers to and that hold no cached entries, and reuse their key slots, so they stop growing for the life of the process.

Forget files that hold nothing. A file is forgotten, with its hints, once no file, row group or column handle refers to it and the cache holds no entry for it. A sweep for such files runs on registration. The cache key's 16-bit file field becomes a slot recycled from forgotten files, while the identity checked before serving an entry is never reused, so registered files keep distinct keys until 65,536 are registered at once instead of until 65,536 have ever been registered.

Sweep before key slots run out. The next sweep is due once the registry has doubled or could take every slot, whichever comes first, and at least 1,024 registrations away. A file counts as holding entries only if the cache has one for its identity, not merely one under its slot, so a file sharing a slot with another that keeps cached data can still be forgotten. Without both, the registry grew again once more than about 32,768 files held cached data. A panic during a sweep no longer disables sweeping, and a sweep cannot forget a file re-registered after reset().

Known and not addressed: an eviction already transcoding an entry when its file is forgotten can re-insert it under the recycled slot. The entry is unreachable (identity mismatch), charged and evictable; closing that window needs a conditional write in core's eviction path.

Closes #84

https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar

@zfarrell
zfarrell marked this pull request as ready for review November 17, 2026 20:35
@zfarrell
zfarrell requested a review from a team as a code owner November 17, 2026 20:35
@zfarrell
zfarrell requested review from rohan-hotdata and removed request for a team November 17, 2026 20:35
@anoop-narang
anoop-narang force-pushed the fix/liquid-memory-charge branch from c7555e6 to 1b7a6f1 Compare November 18, 2026 16:24
The file registry and the per-column lineage hints gained an entry for
every file ever read and never dropped one, so a long-running process
reading ever new files (compaction replaces them) grew both without
bound, with or without a disk tier.

A file is now forgotten, with its hints, once no file, row group or
column handle refers to it and the cache holds no entry under its key.
Registration sweeps for such files whenever the registry has doubled.

Keys and identities are split: the key's 16-bit file field is a slot
recycled from forgotten files, while the identity the cache checks
before serving an entry is never reused. Registered files keep distinct
keys until 65,536 are registered at once, instead of until 65,536 have
ever been registered. reset() now also drops the hints.

Refs #84
@anoop-narang
anoop-narang changed the base branch from fix/liquid-memory-charge to main November 19, 2026 03:52
The sweep for forgettable files ran only once the registry had doubled.
When the files left after a sweep numbered more than half the 65,536 key
slots, the next sweep was scheduled past the point where slots run out,
so new files shared slots although many idle files could have given
theirs back. And a file counted as holding entries if anything sat under
its slot, so a file sharing a slot with one that kept cached data was
never forgotten. Past about 32,768 files holding data, the registry grew
again.

The next sweep is now due once the registry has doubled or could take
every slot, whichever comes first, and at least 1,024 registrations
away. A file holds entries only if the cache has one for its identity.

Also: a panic during a sweep no longer leaves sweeping disabled; a sweep
forgets a file only under the identity it found idle, so it cannot
forget one re-registered after reset(); the shared-slot counter's doc
says it only grows.

Claude-Session: https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.47090% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/datafusion/src/cache/mod.rs 99.40% 1 Missing ⚠️
src/datafusion/src/cache/registry.rs 99.47% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison

Current: 85ab9aac (Liquid) vs Baseline: 85ab9aac (DataFusionDefault)

Query Cold Time Δ Warm Time Δ CPU Time Δ
Q1 2.0ms (2.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q2 9.0ms (5.0ms) +80.0% 3.0ms (3.5ms) -14.3% 8.0ms (4.5ms) +77.8%
Q3 15.0ms (10.0ms) +50.0% 5.0ms (9.0ms) -44.4% 5.5ms (18.0ms) -69.4%
Q4 16.0ms (8.0ms) +100.0% 3.0ms (7.0ms) -57.1% 3.0ms (14.5ms) -79.3%
Q5 46.0ms (34.0ms) +35.3% 36.0ms (32.0ms) +12.5% 5.0ms (15.0ms) -66.7%
Q6 204.0ms (85.0ms) +140.0% 68.0ms (70.5ms) -3.5% 22.0ms (63.5ms) -65.4%
Q7 1.0ms (1.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q8 8.0ms (4.0ms) +100.0% 5.0ms (4.0ms) +25.0% 8.5ms (5.0ms) +70.0%
Q9 81.0ms (52.0ms) +55.8% 64.0ms (56.0ms) +14.3% 5.5ms (30.0ms) -81.7%
Q10 82.0ms (64.0ms) +28.1% 65.5ms (63.5ms) +3.1% 6.5ms (46.0ms) -85.9%
Q11 56.0ms (17.0ms) +229.4% 35.5ms (16.5ms) +115.2% 86.0ms (21.0ms) +309.5%
Q12 48.0ms (21.0ms) +128.6% 28.0ms (18.5ms) +51.4% 69.0ms (26.0ms) +165.4%
Q13 174.0ms (83.0ms) +109.6% 65.0ms (78.5ms) -17.2% 63.0ms (66.0ms) -4.5%
Q14 442.0ms (103.0ms) +329.1% 101.5ms (108.0ms) -6.0% 65.5ms (85.5ms) -23.4%
Q15 235.0ms (85.0ms) +176.5% 66.0ms (80.0ms) -17.5% 59.0ms (77.0ms) -23.4%
Q16 83.0ms (78.0ms) +6.4% 73.0ms (80.5ms) -9.3% 4.0ms (15.0ms) -73.3%
Q17 511.0ms (161.0ms) +217.4% 187.5ms (162.0ms) +15.7% 70.0ms (82.5ms) -15.2%
Q18 471.0ms (165.0ms) +185.5% 189.0ms (162.5ms) +16.3% 69.0ms (82.0ms) -15.9%
Q19 807.0ms (304.0ms) +165.5% 275.5ms (295.0ms) -6.6% 81.0ms (118.5ms) -31.6%
Q20 12.0ms (10.0ms) +20.0% 3.5ms (8.0ms) -56.2% 8.5ms (15.0ms) -43.3%
Q21 559.0ms (161.0ms) +247.2% 112.0ms (159.5ms) -29.8% 238.5ms (258.0ms) -7.6%
Q22 620.0ms (161.0ms) +285.1% 458.0ms (156.0ms) +193.6% 125.0ms (314.5ms) -60.3%
Q23 1.50s (430.0ms) +247.7% 1.44s (439.5ms) +226.5% 431.0ms (675.0ms) -36.1%
Q24 13.02s (786.0ms) +1556.1% 399.5ms (779.5ms) -48.7% 407.0ms (2.09s) -80.5%
Q25 258.0ms (56.0ms) +360.7% 14.0ms (46.5ms) -69.9% 28.0ms (91.5ms) -69.4%
Q26 134.0ms (36.0ms) +272.2% 22.0ms (37.0ms) -40.5% 57.0ms (68.0ms) -16.2%
Q27 262.0ms (47.0ms) +457.4% 19.5ms (48.0ms) -59.4% 50.0ms (91.5ms) -45.4%
Q28 510.0ms (203.0ms) +151.2% 631.5ms (196.5ms) +221.4% 167.5ms (262.5ms) -36.2%
Q29 1.05s (752.0ms) +39.5% 835.0ms (757.5ms) +10.2% 369.0ms (315.5ms) +17.0%
Q30 24.0ms (19.0ms) +26.3% 17.0ms (22.0ms) -22.7% 5.0ms (16.0ms) -68.8%
Q31 552.0ms (82.0ms) +573.2% 59.5ms (81.0ms) -26.5% 54.0ms (116.0ms) -53.4%
Q32 570.0ms (79.0ms) +621.5% 86.5ms (81.0ms) +6.8% 77.0ms (120.5ms) -36.1%
Q33 259.0ms (262.0ms) -1.1% 228.5ms (243.5ms) -6.2% 7.0ms (57.0ms) -87.7%
Q34 655.0ms (321.0ms) +104.0% 351.5ms (319.0ms) +10.2% 128.0ms (262.5ms) -51.2%
Q35 627.0ms (305.0ms) +105.6% 331.5ms (315.5ms) +5.1% 124.0ms (264.5ms) -53.1%
Q36 78.0ms (70.0ms) +11.4% 61.5ms (70.5ms) -12.8% 3.5ms (15.5ms) -77.4%
Q37 204.0ms (84.0ms) +142.9% 56.5ms (81.0ms) -30.2% 21.5ms (69.0ms) -68.8%
Q38 71.0ms (41.0ms) +73.2% 22.0ms (38.5ms) -42.9% 13.5ms (22.0ms) -38.6%
Q39 187.0ms (46.0ms) +306.5% 15.5ms (43.0ms) -64.0% 11.0ms (65.0ms) -83.1%
Q40 396.0ms (166.0ms) +138.6% 164.0ms (164.0ms) +0.0% 49.0ms (121.5ms) -59.7%
Q41 20.0ms (18.0ms) +11.1% 8.0ms (16.0ms) -50.0% 6.5ms (15.0ms) -56.7%
Q42 18.0ms (16.0ms) +12.5% 7.5ms (15.0ms) -50.0% 6.5ms (14.0ms) -53.6%
Q43 13.0ms (11.0ms) +18.2% 9.0ms (13.0ms) -30.8% 7.0ms (9.0ms) -22.2%

⚠️ LiquidCache is slower on 8 queries (warm)

  • Q23: warm +226.5% (1.44s vs 439.5ms)
  • Q28: warm +221.4% (631.5ms vs 196.5ms)
  • Q22: warm +193.6% (458.0ms vs 156.0ms)
  • Q11: warm +115.2% (35.5ms vs 16.5ms)
  • Q12: warm +51.4% (28.0ms vs 18.5ms)
  • Q8: warm +25.0% (5.0ms vs 4.0ms)
  • Q18: warm +16.3% (189.0ms vs 162.5ms)
  • Q17: warm +15.7% (187.5ms vs 162.0ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.

@anoop-narang
anoop-narang merged commit 74d46f9 into main Nov 19, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

datafusion: per-file side tables never shrink

2 participants