Skip to content

local: builder runs DataFusion on an unbounded pool #86

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong. LiquidCacheLocalBuilder::build takes only a SessionConfig and builds the context at src/datafusion-local/src/lib.rs:236. The result runs on UnboundedMemoryPool, and a caller has no way to pass a RuntimeEnv that carries a bounded one.

Evidence. With the cache configured to 1 GiB, the pool was UnboundedMemoryPool { used: 0 } and memory_limit() returned Infinite.

Tests.

Test File Command Flags
audit_local_context_has_a_bounded_memory_pool src/datafusion-local/src/audit_tests.rs:43 cargo +1.96.0 test -p liquid-cache-datafusion-local --lib audit_local_context_has_a_bounded_memory_pool -- --nocapture Inline

Inline test (needs use datafusion::execution::memory_pool::MemoryLimit; use datafusion::prelude::SessionConfig; const GIB: usize = 1 << 30;):

#[tokio::test]
async fn audit_local_context_has_a_bounded_memory_pool() {
    let dir = tempfile::tempdir().unwrap();
    let (ctx, _cache) = LiquidCacheLocalBuilder::new()
        .with_cache_dir(dir.path().to_path_buf())
        .with_max_memory_bytes(GIB)
        .build(SessionConfig::new())
        .await
        .unwrap();
    let pool = ctx.runtime_env().memory_pool.clone();
    assert!(
        matches!(pool.memory_limit(), MemoryLimit::Finite(_)),
        "the SessionContext built by LiquidCacheLocalBuilder runs on an unbounded \
         DataFusion pool: {pool:?}"
    );
}

(The branch's diagnostic println! is trimmed here.)

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions