Confirm the stack comes up clean after the Loki healthcheck fix, then capture dashboard screenshots #12

Activity

1. Gerrrt

   added

   enhancementNew feature or requestNew feature or request

   on Aug 8, 2026
2. Gerrrt commented on Aug 21, 2026

   

   Gerrrt

   on Aug 21, 2026

   OwnerAuthor

   More actions

   Verified on the lab host

   git pull (14 commits behind), make validate, make up. No legacy
   container-name collision — the docker rm -f line was not needed.

   Check	Result
   Six containers steady	all running, RestartCount=0 on every one
   Loki not blocking Grafana / Alloy	no deadlock; both start on service_started as intended
   curl -s localhost:3100/ready	ready
   SNMP targets	4/4 up — 10.0.99.1, 10.0.99.10, 10.7.7.2, 10.0.30.10
   All Prometheus targets	10 active, 0 down
   InstanceDown{job="loki"}	not firing (0 series)
   Rules loaded	34, 0 unhealthy
   Alloy components	27, 0 unhealthy
   Data on hand	Prometheus back to 2026-08-02 (19d), Loki to 2026-08-16 (5d)

   The two alerts that are firing are hardware, not stack: UpsSelfTestFailed
   (silenced until 20 Sep, as recorded in the roadmap) and IloBatteryCondition on
   shiva — the latter had no silence and no owner, so it is now #76.

   The Loki healthcheck fix from #8 is confirmed good. Nothing about it is
   implicated in anything below.

   Three faults found while doing it

   Verifying the stack meant actually looking at the dashboards, and they were
   lying. All three have the same shape — a component reporting itself healthy
   while producing nothing — and all three are fixed in #62.

   1. Alloy was answering to the name of the server. This host is called
   prometheus; render-config.sh writes ALLOY_HOSTNAME=$(hostname); compose
   passed that to hostname:, which registers the name in Docker's embedded DNS.
   Two A records for prometheus on one bridge, so every remote_write was a coin
   toss between the metrics store and the agent itself.

   Measured: host and container metrics stopped at 02:50 and did not return
   until the container was recreated at 15:29 — twelve and a half hours — while
   up{job="alloy"} read 1 for every scrape in between, because Alloy's own
   /metrics endpoint was never the thing that broke.

   Introduced by 9405de0, the commit that fixed the host label being a container ID.

   2. cAdvisor could only see its own cgroup. privileged: true is not enough
   on cgroup v2 — Docker's default namespace mode is private, so the container's
   own cgroup is the root it can see. It reported exactly one series, id="/", and
   reported it happily. No per-container metric had existed for at least 24
   hours, which makes the entire Docker Containers dashboard and four of the eight
   rules in containers.rules.yaml decorative. cgroup: host fixes it;
   docker_only = true stops the fix costing 2500 series of systemd slices.

   3. "All" selected nothing. Provisioned dashboards ship current: {} and
   options: [] because allowUiUpdates is false, so Grafana must resolve the
   option list before it can expand "All" — and the panels do not wait for it. 10 of
   the Logs dashboard's 13 panels were empty; the only one that worked was the one
   query that references no variable. allValue: ".+" makes "All" a fixed regex.

   Screenshots

   make screenshots now exists (#62). It starts a headless renderer behind a
   capture compose profile, shoots all five dashboards at a height derived from
   each dashboard's own JSON, and stops the renderer again — make up neither
   pulls nor starts it, and docker compose ps still shows six services.

   All five render correctly today. They are not committed yet, for two reasons:

   • The window is now-24h, and fault 1 was only fixed this afternoon, so the
     host and Docker dashboards currently picture their own outage. Re-running
     make screenshots after a clean day is the fix, and it is one command.
   • The Logs dashboard, now that it works, renders auth.log verbatim — real
     usernames, real source addresses. That needs a decision before it goes into a
     public repository, and docs/images/README.md now says so explicitly rather
     than leaving it to be noticed.

   So the last two boxes stay open until tomorrow's capture. Everything that was
   blocking them is done.

   Also filed

   Reading the whole repository to do this turned up 45 issues' worth of untracked
   work — every unchecked roadmap item, the ADR-0007 and ADR-0008 build-outs that
   were recorded and never tracked, and a set of verified findings that were not
   written down anywhere: #63 (ContainerHighMemory cannot fire), #64 (make backup overwrites its own last-good archive), #67 (no dead man's switch on the
   notification path), #68 (make validate and CI have drifted apart), #70
   (unauthenticated services on 0.0.0.0), #72 (rule counts stale in eight places,
   including a deploy-runbook check that now reads as a failure).
3. Gerrrt commented on Aug 21, 2026

   

   Gerrrt

   on Aug 21, 2026

   OwnerAuthor

   More actions

   Scope change, per the decision to drop the Logs dashboard: four screenshots, not five.

   homelab-logs renders auth.log verbatim in three panels, and no time range or
   filter avoids that — showing log lines is what the dashboard is for. #108 removes
   it from the capture set in the script rather than leaving a warning in
   docs/images/README.md for whoever runs the capture to remember. The doc now
   explains the absence, so nobody later "fixes" the script by adding it back.

   If a Logs screenshot is ever wanted, redaction is the thing to build first.

   The remaining four are scheduled

   The blocker was the window: the capture is now-24h, and the collection faults
   were only fixed this afternoon, so today's images picture their own outage. Host
   metrics resumed at 15:29 and per-container metrics at ~15:53, so a clean 24-hour
   window opens this time tomorrow.

   A one-shot systemd user timer on the monitoring host runs make screenshots at
   2026-08-22 16:30 UTC, logging to ~/.local/state/homelab-screenshots.log.
   Lingering is enabled, so it survives logout. The exact command path was
   smoke-tested end to end — four PNGs, exit 0, secrets decrypted non-interactively.

   It captures only. It does not commit, deliberately: these go into a public
   repository and want looking at first.

   systemctl --user list-timers homelab-screenshots   # confirm it is still armed
   cat ~/.local/state/homelab-screenshots.log          # after it fires

   Then review the four against the checklist and uncomment the block at
   README.md:177, and this can close.
4. Gerrrt commented on Aug 21, 2026

   

   Gerrrt

   on Aug 21, 2026

   OwnerAuthor

   More actions

   Two of the four are in via #115: network-snmp.png and ups-power.png, both clean against the checklist in docs/images/README.md.

   Two are held back, and neither is a cropping job:

   • host-overview.png — a 14-hour hole (02:00→16:00) where the agent was down. Half the panels are a flat line.
   • docker-containers.png — 22 of 24 hours empty, because cAdvisor only started reporting correctly in Screenshot capture, and three faults it uncovered #62.

   Both just need a re-shoot once the stack has a clean day behind it — alloy came up ~15:47 on 2026-08-21, so any capture after ~16:00 the following day gets a full window.

   Worth recording separately: shooting these found a real leak, not just bad windows. The Container inventory panel excluded table fields by name, and cAdvisor keeps adding container_label_* fields — two of which hold the absolute path of compose.yaml, so the panel was rendering a username. Fixed in #115 by switching to an allowlist. That was the pre-publish checklist doing its job, which is the argument for keeping it.

   The root README.md screenshot block stays commented until all four exist.
5. Gerrrt

   closed this as completedin #117on Aug 24, 2026
6. Gerrrt commented on Aug 24, 2026

   

   Gerrrt

   on Aug 24, 2026

   OwnerAuthor

   More actions

   Closed by #117. Both remaining boxes are done.

   The one-shot timer fired on schedule at 2026-08-22 16:30 and wrote four PNGs.
   Every one was read and checked against the checklist in docs/images/README.md:

   Check	Result
   Full 24-hour window, no gaps	all four — the 14-hour hole in host-overview and the 22 empty hours in docker-containers are gone
   Full MAC addresses	none
   WAN IP address	none — RFC1918 only
   Hostnames / usernames in tables or legends	device names only, all already in docs/hardware.md
   Container inventory panel	three allowlisted fields; the compose.yaml path leak from #115 stays fixed

   network-snmp.png and ups-power.png were replaced rather than left at their
   #115 versions, so all four now come from one run and are comparable to each other.

   Two things visible in them are real and deliberately kept rather than shot
   around: IloBatteryCondition firing on shiva (#76 — a picture of a stack with
   nothing wrong would be the less honest one), and the renderer container in two
   Docker legends, which exists only while a capture is running and is why the
   inventory lists six containers and not seven.

   README.md's screenshot block is live. docs/images/README.md lost the "why
   these two are absent" table and kept the window warning and the Logs-dashboard
   exclusion, both still load-bearing. Makefile:260 stopped claiming five PNGs.

   The two issues this verification pass filed are also done

   • urgent and default alert to the same webhook, and nothing routes on category=security #66 — urgent/default/security are three destinations now, and
     category=security routes on its own at both severities (feat(alerts): three notification channels, and security routes on its own #118).
   • No dead man's switch on the notification path #67 — there is a Watchdog rule and a heartbeat watched from off-host
     (feat(alerts): a dead man's switch on the notification path #119).

   Both are merged but not deployed: they add SOPS keys only the owner can fill
   in, so render-config.sh refuses to render until they exist. #120 has the
   commands and the precise current state of the host.
7. Gerrrt

   added a commit that references this issue on Aug 31, 2026

   feat(alerts): catch a remote-writing agent that stops pushing

   ...

   24f790e