You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cache: default memory budget ignores the cgroup limit #87
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
default_max_memory_bytes() (src/core/src/cache/builders.rs:162-170) returns half of the host'stotal_memory().
The builder docs (builders.rs:76, src/datafusion-local/src/lib.rs:111) say "half of available system memory".
In a container with a small limit on a large node, the default never binds, so the container is OOM-killed first.
Every build() call also takes its own full default budget.
Evidence.
Under docker run --memory=1g, the default was 8,371,793,920 bytes (about 7.8 GiB), half the Docker VM's 16 GiB. That is 15.6x the 512 MiB it should be.
Three default local builds summed to 1.5x host RAM. Two do not exceed it (see Dropped).
dev/audit/run-in-container.sh mem-limit (runs cargo test -p liquid-cache --lib audit_default_memory_respects_cgroup_limit -- --nocapture in rust:1.96.0-bookworm with --memory=1g)
L, D; skips with a printed note when there is no finite limit
cargo +1.96.0 test -p liquid-cache-datafusion-local --lib audit_three_default_builds_fit_in_host_memory -- --nocapture
E; mounts the shared $TMPDIR/liquid_cache.t4 (see #96)
Not inline: helper plus test is about 49 lines and Linux-gated. In CI it drops in as one docker run --memory=1g step.
Possible fix: use sysinfo's cgroup_limits(). It reported the correct 1 GiB under the limit and needs no new feature flag.
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity