No resource limits on any container, and no size bound on retention #71

Activity

1. Gerrrt

   added

   enhancementNew feature or requestNew feature or request

   on Aug 21, 2026
2. Gerrrt commented on Aug 31, 2026

   

   Gerrrt

   on Aug 31, 2026

   OwnerAuthor

   More actions

   #184 takes the half of this that could be sized, and splits the rest. After it,
   nothing in this issue is untracked.

   Done in #184:

   • pids_limit (512, 1024 for Alloy) on every service, plus init: true — which
     had to come first, because verifying this issue found Grafana already leaking
     two unreaped ssl_client children every 30 seconds from its https healthcheck.
     2437 of them at the time of measurement, on course to exhaust the inherited
     pids.max of 9056 about 3.1 days after each start. pids_limit alone would
     have cut it off sooner rather than protecting it. Nothing in the stack could
     see it — container_tasks_state reads zero for every state.
   • --storage.tsdb.retention.size, at 12 GiB. Not 4: the store's current 1021 MiB
     is not a steady state (prometheus_tsdb_time_retentions_total is 0, nothing
     has ever been evicted), and the write rate went 16.1 → 72.9 MiB/day at 08-17
     when Screenshot capture, and three faults it uncovered #62 fixed cAdvisor. Real steady state is 2.28 GiB, 2.73 GiB once Saruman
     gets an agent.
   • PrometheusSizeRetentionActive, so the ceiling cannot engage silently and
     quietly shorten the 30 days this repository promises in four places.

   Split out, with the reasoning recorded in each:

   • mem_limit / memswap_limit → Set memory limits on the six services to bound the blast radius of a leak #114, with a comment arguing its 14-day gate
     cannot be satisfied on its own terms (3x every peak is 7536 MiB against 7816
     MiB of RAM) and that memswap_limit is answerable today.
   • cpus → refused, and ContainerCpuThrottled cannot fire — no service sets a CPU quota #185 instead: ContainerCpuThrottled is unfireable
     without a CPU quota, which is ContainerHighMemory cannot fire — no service sets a memory limit #63 one rule down in the same file. Adding
     cpus to make it live would be backwards — it prevents nothing at 0.4 of 4
     cores and would fire on scrape bursts.
   • ulimits → refused, with the reason written into compose.yaml. RLIMIT_NPROC
     is enforced per real UID kernel-wide, and alertmanager/snmp-exporter run as
     the operator's own uid via RENDER_UID, so an nproc there would count the
     operator's host processes against the container and could deny them a login
     shell. nofile would throttle the mmap'd chunk files Prometheus and Loki need,
     against an exhaustion never observed.
   • read_only → Run the stack read-only where it can be, and record where it cannot #186. cap_drop + no-new-privileges → Drop capabilities and set no-new-privileges on the six non-Alloy services #187. Both separable
     because the second is verifiable from the config and the first only from a
     running stack.
   • The Alloy privilege clause this issue raises in passing → Reduce Alloy's privilege — it is root on the host and can read the age key #188, which is the
     largest item in the file and worth more than everything else here combined.
   • And one this issue did not name: bounding Prometheus made Loki the only
     unbounded store on the disk → Loki is now the only unbounded store on the disk #189.
3. Gerrrt

   added a commit that references this issue on Aug 31, 2026

   Merge pull request #184 from Gerrrt/gerrrt/github-issue-71-b58993

   ...

   7529749
4. Gerrrt commented on Aug 31, 2026

   

   Gerrrt

   on Aug 31, 2026

   OwnerAuthor

   More actions

   Merged as 7529749 (#184), deployed, and confirmed on the host: Grafana flat at 24 tasks against the 2,495 it was carrying, retention_limit_bytes 12884901888, nothing evicted, 39 rules healthy.

   Closing. Nothing in this issue is untracked — the memory half is #114 (with its gate rewritten), and the rest is #185, #186, #187, #188 and #189. The split and the reasoning for each are in the comment above.
5. Gerrrt

   closed this as completedon Aug 31, 2026