You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
datafusion: cache memory never reserves from the MemoryPool #82
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong. Nothing in src/ registers a MemoryConsumer or takes a MemoryReservation. A search for MemoryReservation, MemoryConsumer or memory_pool finds nothing. So DataFusion's pool limit does not constrain cache data, snapshots or decode transients.
Evidence. A scan held 2,100,224 cache bytes plus 525,056 snapshot bytes against a 65,536-byte pool (40.1x). pool.reserved() peaked at 0, and no ResourcesExhausted error was raised.
cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_liquid_scan_reserves_from_datafusion_memory_pool -- --nocapture
Possible fix: add a with_memory_pool builder option and register a consumer against it.
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
One more thing the same pool should cover, so it is decided as part of this fix rather than separately.
A read hands the query a decompressed copy of the entry. In the audit measurements that copy was 3x to 44x the size the cache charges for the entry, depending on the encoding. The query owns it, and nothing reserves it: not the cache budget, and not the DataFusion pool.
This is a design decision, not a bug with one obvious fix, which is why it has no issue of its own. The open question is who reserves it:
the cache, against the pool passed through with_memory_pool, for the lifetime of the emitted batch, or
the caller, which reserves each batch it receives against its own pool.
Whoever picks up this issue should settle that, because either answer uses the reservation this issue adds.
Activity
zfarrell commentedon Nov 17, 2026
One more thing the same pool should cover, so it is decided as part of this fix rather than separately.
A read hands the query a decompressed copy of the entry. In the audit measurements that copy was 3x to 44x the size the cache charges for the entry, depending on the encoding. The query owns it, and nothing reserves it: not the cache budget, and not the DataFusion pool.
This is a design decision, not a bug with one obvious fix, which is why it has no issue of its own. The open question is who reserves it:
with_memory_pool, for the lifetime of the emitted batch, orWhoever picks up this issue should settle that, because either answer uses the reservation this issue adds.