[Gerrrt]

Closes #114.

No service set mem_limit, so a leak in any one container could take a host
with 8 GB soldered down with it. All seven now carry one, and a
memswap_limit equal to it.

The gate this issue set has cleared

#114 blocked itself on data, deliberately: cAdvisor has only reported
per-container series since #62 fixed cgroup: host, so there were six hours of
history when it was written. There are now fourteen days — 2026-08-21 to
2026-09-04 — and the numbers come from that window rather than from a guess.

Two of the issue's own premises did not survive it.

Prometheus is no longer the hard one. prometheus_tsdb_lowest_timestamp is
exactly 30 days back and prometheus_tsdb_time_retentions_total has
incremented, so the store is evicting and has reached the steady state that six
hours of a mostly-empty TSDB could say nothing about. What is still moving is
the series count, not the store.

loki is. The issue listed it among the low-risk services to start with, on
a 152 MiB peak. Fourteen days puts its peak at 1015 MiB.

Sized from RSS, not working set

mem_limit becomes cgroup v2 memory.max, which bounds memory.current —
anonymous pages plus page cache. The kernel reclaims cache before it OOM-kills
anything, so what a limit has to clear is the anonymous high-water mark.

The difference is not cosmetic. alloy's 512 MiB working-set peak is 221 MiB of
RSS and the rest reclaimable cache from walking /rootfs — its anonymous memory
never leaves 155-181 MiB across the whole fortnight. Sizing from working set
would have set a 1536M ceiling for a process that never exceeded 221 MiB of the
thing the ceiling is about.

It is also what retires the method the issue proposed: 3x every working-set
peak is 7600 MiB against 7816 MiB of RAM.

service	median RSS	peak RSS	peak WS	limit	×peak RSS
prometheus	116	198	244	1024M	5.2
loki	93	1015	1039	1536M	1.5
grafana	139	540	630	1280M	2.4
alloy	163	221	512	768M	3.5
alertmanager	19	25	52	256M	10.3
snmp-exporter	14	20	31	128M	6.3
blackbox-exporter	13	18	26	128M	7.2

MiB. Seven, not six — blackbox-exporter joined after the issue was written,
and it is the one service with six days of history rather than fourteen, which
its comment says.

The multipliers vary and each service says why. The sum, 5120 MiB, is a ceiling
and not a reservation: the stack's combined RSS peaked at 1448 MiB across the
fortnight and averaged 566.

loki's number belongs to a bug, and the bug is on a timer

Its 1015 MiB peak is an ingest flood — new issue #286. backup-volumes.sh
archives each volume with docker run ... tar -czf -, the gzip stream is that
container's stdout, and Alloy's loki.source.docker discovers every container
on the socket rather than this stack's seven. make backup ships its own
archives back through the log pipeline.

Loki's RSS tracks the ingest rate through that hour minute for minute:

time	loki RSS	ingest	lines/s
05:50-06:15	~100 MiB	0.0 MiB/s	3
06:16	461 MiB	2.6 MiB/s	5,501
06:18	555 MiB	2.7 MiB/s	5,693
06:29	1,015 MiB	0.6 MiB/s	1,198
06:30	960 MiB	0.0 MiB/s	2
06:39	107 MiB	0.0 MiB/s	2

Worth being explicit about cause and symptom, because the logs read the other
way round: through those minutes Loki's compactor cycled, logged "instance not
found in the ring" and returned 400s while Alloy dropped batches. That is a Loki
shedding load, not a Loki churning on its own — the containers never restarted,
and both were at baseline by 06:40 because the flood stopped.

It recurs on the backup timers: 577 MiB on 09-04, 179 on 09-01, against
107-128 MiB on the nine quiet days. So 1536M has to clear it until #286 lands,
at which point this should be re-derived down to roughly 512M. The comment
says so.

Swap, decided rather than defaulted

memswap_limit equal to mem_limit is memory.swap.max = 0. Left unset,
Docker defaults it to twice mem_limit — so adding a memory limit would have
handed every container an equal allowance of the unencrypted /swap.img this
repository already carries as a data-at-rest exposure, and the limit would not
have been a memory limit.

Not hypothetical: peak container_memory_swap over the window was 70 MiB for
prometheus, 17 for alloy, 12 for loki, 8 for grafana, 5 for snmp-exporter. Every
one had already paged out. SECURITY.md and docs/security.md are updated, and
both say what this does not close.

The rule the limits make necessary

ContainerNearMemoryLimit is the complement #114 raised as optional. It is not
optional any more: the largest limit is 1536M, which is 0.20 of this host,
against ContainerHighMemory's 0.40 — so after this change every service can
be OOM-killed by its own limit without that rule ever firing. The limits close
the gap it watches and open a new one underneath it.

It divides container_memory_rss by the limit behind the > 0 guard that #63
is about, and carries four promtool test rules cases, each paired against a
quiet one:

• fires at 93.75% of a real limit, and not before for: 15m
• silent at the worst input fourteen days offered (loki at 66%)
• silent for 200 MiB of RSS behind a 720 MiB working set — this one fails if
  the numerator is ever "simplified" to working set
• silent for 4 GiB of RSS against a limit of 0 — fails if the guard is
  dropped, which is ContainerHighMemory cannot fire — no service sets a memory limit #63 exactly

ContainerHighMemory stays host-relative. Coupling enforcement to detection is
what made it unfireable for months, and that is why #114 kept them separate.

Not done

• No limits on stacks/lab: it runs on another host with its own Prometheus and
  no series reaching 10.0.99.20 to size from. The file says so rather than
  copying these numbers.
• No limits on the five profile services (renderer, archiver, gitleaks,
  actionlint, editorconfig-checker) — none runs in a normal deploy, so there
  was nothing to measure. An unmeasured ceiling on a headless Chromium is the
  guess this work exists to avoid.

Verification

make validate passes (2 skips, both "not the deployment host"). The stack
restart and the post-deploy check are in a comment below.

🤖 Generated with Claude Code

[Gerrrt]

Deployed and verified

make converge → d168c86ea418, all seven recreated, apply=1 dirty=0 verified=1.

The limits are in the kernel, not just the file. memory.max and
memory.swap.max read from each container's cgroup:

service	memory.max	memory.swap.max
loki	1610612736	0
grafana	1342177280	0
prometheus	1073741824	0
alloy	805306368	0
alertmanager	268435456	0
snmp-exporter	134217728	0
blackbox-exporter	134217728	0

memory.swap.current is 0 on all seven, so the swap decision is enforced rather
than merely declared. (loki and alloy were read from the host — loki is
distroless and has no cat, which is the point check_compose_health.py
asserts on every CI run.)

cAdvisor now reports a real ceiling, which is what makes the new rule
possible at all — it was 0 for every service before this:

loki               limit 1536 MiB   rss 338 MiB   22.0% of its ceiling
alloy              limit  768 MiB   rss 188 MiB   24.5%
prometheus         limit 1024 MiB   rss 168 MiB   16.4%
grafana            limit 1280 MiB   rss 144 MiB   11.3%
snmp-exporter      limit  128 MiB   rss  15 MiB   11.9%
blackbox-exporter  limit  128 MiB   rss  13 MiB   10.5%
alertmanager       limit  256 MiB   rss  20 MiB    7.9%

Rules: 56 alert rules loaded, none in error. All five container rules
health=ok state=inactive, ContainerNearMemoryLimit included.

The > 0 guard is doing real work in production, not just in the fixture.
There are 7 limited containers and 12 unlimited ones on this host right now
(restartloop-probe and other sessions' test containers). The rule expression
returns no series — without the guard, all 12 would divide by zero to +Inf
and alert forever, which is #63's failure exactly.

ContainerOomKilled quiet, 0 OOM events since the deploy.

One thing worth writing down

sum(container_memory_swap{...}) read 50 MiB after the deploy, which looked
like the swap limit had not taken. It had — those were the pre-deploy
container IDs, still inside Prometheus' 5-minute staleness window. 15 series for
7 containers; every current one reads 0.0 MiB.

That is the same trap that made the six-hour sizing in #114 misleading: these
series carry the compose config hash and image digest, so a recreate starts a
new one, and any aggregation that forgets by (name) silently mixes container
generations. Worth remembering before reading anything off container_memory_*
after a deploy.

Follow-ups

• Alloy tails the backup archiver's tar stream into Loki #286 sets loki's number and should retire it. Re-derive from a fortnight
  without the flood; the comment says to expect ~512M.
• docs(runbook): say why the Rules page counts one more than we do #303 — the runbook's post-deploy check says 56 rules and the page shows
  57, because one is a recording rule. Pre-existing, found by running the check.