[Gerrrt]

Closes #114.

Sets mem_limit and memswap_limit on all seven running services — seven, not
the six #114 names, because blackbox-exporter joined the stack after the issue
was written.

The gate, and how it was cleared

#114 was blocked on data, and docs/roadmap.md had already narrowed what would
unblock it: not "wait for more history" but explain the 2026-08-29 hour in
which loki and alloy both peaked, because a peak two services share is
either real correlated load or an artefact, and only one of those is worth
sizing a ceiling around.

It is the artefact. At 06:18:14–15 both processes tore down internal components
in the same second — Alloy logged 35 node exited without error lines in three minutes (210 across the hour) and
re-registered its cAdvisor and unix collectors at 06:21; Loki's compactor module
stopped and then cycled start/stop five more times through 06:34, logging
instance not found in the ring each time and returning 400s while Alloy dropped
batches. Both were back at baseline by 06:40, and neither container restarted —
creation times either side are unchanged. In-process component churn: not a
crash, not a deploy, not a leak, and not recurred in the six days since.

There are now 14.2 days of container_memory_* history (2026-08-21 → 09-04),
so the issue's first "Done when" box is satisfied on its own terms as well.

The method had to change too

3x every peak did not fit the machine — but the reason is not that the machine
is small. It is that the peaks were being read off the working set, which
includes reclaimable page cache. mem_limit bounds a cgroup the kernel reclaims
cache from before it OOM-kills anything, so what a limit must actually clear is
the anonymous high-water mark. alloy is the worked example: a 512 MiB
working-set peak is 221 MiB of RSS and the rest is cache from walking /rootfs.

service	median RSS	peak RSS	peak WS	limit	limit / peak RSS
prometheus	145	198	244	1024M	5.2x
loki	103	1015	1039	1536M	1.5x
grafana	154	540	630	1280M	2.4x
alloy	163	221	512	768M	3.5x
alertmanager	20	25	52	256M	10.2x
snmp-exporter	15	20	31	128M	6.4x
blackbox-exporter	15	18	26	128M	7.1x

Sized from RSS with the multiplier chosen per service, the seven sum to
5120 MiB against 7816 MiB of RAM. The same 3x rule read off the
working-set column gives 7602 MiB — the limits would sum to the whole machine
and bound nothing in aggregate, which is the one thing they exist to do.

loki gets the loosest ratio and the largest number. #114 listed it among
the low-risk services to start with, on a 152 MiB peak from a six-hour window.
Fourteen days invert that: it has by far the widest dynamic range here, and
1536M is set to survive a recurrence of the 06:20 event rather than to clip it.
OOM-killing the log store during component churn turns a self-healing blip into
an outage and takes the record of it along too.

Swap: decided, not defaulted

memswap_limit equals mem_limit everywhere, so memory.swap.max is 0 and the
stack cannot page out at all. The default is the wrong way round — unset, it is
twice mem_limit, so adding memory limits alone would have made the
unencrypted-/swap.img exposure worse, by giving constrained containers a
reason to spill into it. Measured cost of switching it off: 70 MiB of peak swap
for Prometheus, 17 for Alloy, 12 for Loki, under 5 for the rest.

docs/security.md and SECURITY.md record the narrowing without overstating
it — the disk is unchanged, the host still swaps, and anything already paged out
is still there.

Enforcement stays uncoupled from detection

Which is why #63 was a separate issue in the first place. ContainerHighMemory
still divides by machine_memory_bytes. ContainerNearMemoryLimit is the
limit-relative complement, added with its own promtool test rules cases, and
it measures RSS for the same reason the limits were sized from it.

Two mutations were run against the new tests, and both fail as they should:
changing the numerator to working set fails the page-cache case, and dropping
the > 0 guard fails the unlimited-container case with +Inf%.

That the rule can fire against real data rather than only fixtures was
checked end to end with a throwaway container: compose mem_limit →
memory.max, memswap_limit → memory.swap.max = 0, cAdvisor reporting a
non-zero container_spec_memory_limit_bytes, and the expression producing a
series. That check is the one #63 never had.

Not applied to stacks/lab

It runs on its own guest with its own Prometheus and no series reaching this
one, so it has nothing of its own to size from. Copying these numbers would be a
limit derived from a different machine. Recorded there as a difference rather
than left as an omission.

What is not done

scripts/validate.sh passes (2 skips, both "not the deployment checkout").
The stack has not been restarted onto these limits — this worktree is not
the deployment checkout, so the third "Done when" box is the converge timer's
job once this is on main. Worth watching after it deploys: ContainerOomKilled
quiet, and no service settling near its ceiling.

Separately, while measuring: container_start_time_seconds did not move when
the stack was stopped and started on 09-04 — it still reports each container's
creation time. If that also holds for Docker's restart policy, then
ContainerRestartLoop cannot fire for the crash loop it exists to catch, which
is the #63 shape again. Not touched here; flagged for its own issue.

🤖 Generated with Claude Code

[Gerrrt]

Closing as superseded by #298, which is merged and deployed.

This is not a rejection of the work in here — #298 was built from this
branch's patch and kept its two best decisions:

• Sizing from container_memory_rss rather than working set. mem_limit
  bounds memory.current, which includes reclaimable page cache, so the
  anonymous high-water mark is the number a ceiling has to clear. That is the
  reasoning that retired the issue's own "3x every peak" method, and it came
  from here.
• The ContainerNearMemoryLimit fixture that pins the numerator — 200 MiB
  of RSS behind a 720 MiB working set, which fails if anyone "simplifies" the
  rule to working set. That test is the most valuable thing on this branch.

Every limit merged unchanged: prometheus 1024M, loki 1536M, grafana 1280M,
alloy 768M, alertmanager 256M, snmp-exporter 128M, blackbox-exporter 128M.

Three things were corrected on the way in, recorded here because the reasoning
matters more than the numbers:

1. The 2026-08-29 peak was an ingest flood, not component churn. Loki's RSS
   tracks the ingest rate minute for minute — ~100 MiB at 3 lines/s until 06:15,
   461 MiB the minute 5,501 lines/s arrived, 1,015 at 06:29, back to 107 nine
   minutes after the flow stopped. The compactor cycling and the 400s are a Loki
   shedding load. Source is Alloy tails the backup archiver's tar stream into Loki #286.
2. It has recurred, contrary to "not in the six days since": 577 MiB RSS on
   09-04 at 447 KiB/s peak ingest, against 107-128 MiB on the nine quiet days.
3. The ContainerHighMemory note was inverted. 1536M is 0.20 of the host,
   which is half the 0.40 threshold — so every service can now be OOM-killed
   by its own limit without that rule firing. That is the argument for
   ContainerNearMemoryLimit, not against it.

Some smaller numbers were re-measured too (medians were means; alertmanager's
range is 12-25 not 20-24; 3x the working-set peaks is 7600 MiB, not 7536).

Deploy verification is on #298. The branch is being deleted along with its
worktree.