You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong. Two maps keyed by file only ever grow:
ParquetCacheMetadata.expression_hints (src/datafusion/src/io/mod.rs:13). Each column path is capped at 16 expressions, but the number of paths grows without bound.
Neither shrinks when the cache is emptied or a file's handles are dropped. DefaultCacheMetadata.lineages (src/core/src/cache/io_context.rs:38) is the same pattern but was not tested.
Evidence.
1,000 column paths were still tracked after flush_data() and reset() left the cache with 0 entries.
10,000 of 10,000 file identities remained after every handle was dropped.
cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_expression_hint_history_is_capped_per_column -- --nocapture
C (passes; confirms the cap of 16)
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity
fix(cache): forget files that hold nothing
fix(cache): forget files that hold nothing (#102)