Repository navigation
fix(cache): forget files that hold nothing - #102
Merged
Merged
Conversation
zfarrell
marked this pull request as ready for review
November 17, 2026 20:35
anoop-narang
force-pushed
the
fix/liquid-memory-charge
branch
from
November 18, 2026 16:24
c7555e6 to
1b7a6f1
Compare
The file registry and the per-column lineage hints gained an entry for every file ever read and never dropped one, so a long-running process reading ever new files (compaction replaces them) grew both without bound, with or without a disk tier. A file is now forgotten, with its hints, once no file, row group or column handle refers to it and the cache holds no entry under its key. Registration sweeps for such files whenever the registry has doubled. Keys and identities are split: the key's 16-bit file field is a slot recycled from forgotten files, while the identity the cache checks before serving an entry is never reused. Registered files keep distinct keys until 65,536 are registered at once, instead of until 65,536 have ever been registered. reset() now also drops the hints. Refs #84
The sweep for forgettable files ran only once the registry had doubled. When the files left after a sweep numbered more than half the 65,536 key slots, the next sweep was scheduled past the point where slots run out, so new files shared slots although many idle files could have given theirs back. And a file counted as holding entries if anything sat under its slot, so a file sharing a slot with one that kept cached data was never forgotten. Past about 32,768 files holding data, the registry grew again. The next sweep is now due once the registry has doubled or could take every slot, whichever comes first, and at least 1,024 registrations away. A file holds entries only if the cache has one for its identity. Also: a panic during a sweep no longer leaves sweeping disabled; a sweep forgets a file only under the identity it found idle, so it cannot forget one re-registered after reset(); the shared-slot counter's doc says it only grows. Claude-Session: https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar
anoop-narang
force-pushed
the
fix/side-table-growth
branch
from
November 19, 2026 04:21
dfe90ab to
a398195
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
📊 Benchmark ComparisonCurrent:
Compared Liquid vs DataFusionDefault on the same runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The per-file registry and lineage hints now drop files that no handle refers to and that hold no cached entries, and reuse their key slots, so they stop growing for the life of the process.
Forget files that hold nothing. A file is forgotten, with its hints, once no file, row group or column handle refers to it and the cache holds no entry for it. A sweep for such files runs on registration. The cache key's 16-bit file field becomes a slot recycled from forgotten files, while the identity checked before serving an entry is never reused, so registered files keep distinct keys until 65,536 are registered at once instead of until 65,536 have ever been registered.
Sweep before key slots run out. The next sweep is due once the registry has doubled or could take every slot, whichever comes first, and at least 1,024 registrations away. A file counts as holding entries only if the cache has one for its identity, not merely one under its slot, so a file sharing a slot with another that keeps cached data can still be forgotten. Without both, the registry grew again once more than about 32,768 files held cached data. A panic during a sweep no longer disables sweeping, and a sweep cannot forget a file re-registered after
reset().Known and not addressed: an eviction already transcoding an entry when its file is forgotten can re-insert it under the recycled slot. The entry is unreachable (identity mismatch), charged and evictable; closing that window needs a conditional write in core's eviction path.
Closes #84
https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar