Skip to content

datafusion: cache memory never reserves from the MemoryPool #82

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong. Nothing in src/ registers a MemoryConsumer or takes a MemoryReservation. A search for MemoryReservation, MemoryConsumer or memory_pool finds nothing. So DataFusion's pool limit does not constrain cache data, snapshots or decode transients.

Evidence. A scan held 2,100,224 cache bytes plus 525,056 snapshot bytes against a 65,536-byte pool (40.1x). pool.reserved() peaked at 0, and no ResourcesExhausted error was raised.

Tests.

Test File Command Flags
audit_liquid_scan_reserves_from_datafusion_memory_pool src/datafusion/src/optimizers/mod.rs:1537 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_liquid_scan_reserves_from_datafusion_memory_pool -- --nocapture

Possible fix: add a with_memory_pool builder option and register a consumer against it.

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
  2. zfarrell commented on Nov 17, 2026

    @zfarrell
    Author

    One more thing the same pool should cover, so it is decided as part of this fix rather than separately.

    A read hands the query a decompressed copy of the entry. In the audit measurements that copy was 3x to 44x the size the cache charges for the entry, depending on the encoding. The query owns it, and nothing reserves it: not the cache budget, and not the DataFusion pool.

    This is a design decision, not a bug with one obvious fix, which is why it has no issue of its own. The open question is who reserves it:

    • the cache, against the pool passed through with_memory_pool, for the lifetime of the emitted batch, or
    • the caller, which reserves each batch it receives against its own pool.

    Whoever picks up this issue should settle that, because either answer uses the reservation this issue adds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions