Skip to content

cache: default memory budget ignores the cgroup limit #87

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • default_max_memory_bytes() (src/core/src/cache/builders.rs:162-170) returns half of the host's total_memory().
  • The builder docs (builders.rs:76, src/datafusion-local/src/lib.rs:111) say "half of available system memory".
  • In a container with a small limit on a large node, the default never binds, so the container is OOM-killed first.
  • Every build() call also takes its own full default budget.

Evidence.

  • Under docker run --memory=1g, the default was 8,371,793,920 bytes (about 7.8 GiB), half the Docker VM's 16 GiB. That is 15.6x the 512 MiB it should be.
  • Three default local builds summed to 1.5x host RAM. Two do not exceed it (see Dropped).

Tests.

Test File Command Flags
audit_default_memory_respects_cgroup_limit src/core/src/cache/builders.rs:560 dev/audit/run-in-container.sh mem-limit (runs cargo test -p liquid-cache --lib audit_default_memory_respects_cgroup_limit -- --nocapture in rust:1.96.0-bookworm with --memory=1g) L, D; skips with a printed note when there is no finite limit
audit_default_memory_budget_is_half_of_available_memory src/core/src/cache/builders_audit_tests.rs:14 cargo +1.96.0 test -p liquid-cache --lib audit_default_memory_budget_is_half_of_available_memory -- --nocapture E (available memory varies by host)
audit_three_default_builds_fit_in_host_memory src/datafusion-local/src/audit_tests.rs:98 cargo +1.96.0 test -p liquid-cache-datafusion-local --lib audit_three_default_builds_fit_in_host_memory -- --nocapture E; mounts the shared $TMPDIR/liquid_cache.t4 (see #96)

Not inline: helper plus test is about 49 lines and Linux-gated. In CI it drops in as one docker run --memory=1g step.

Possible fix: use sysinfo's cgroup_limits(). It reported the correct 1 GiB under the limit and needs no new feature flag.

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions