[Gerrrt]

Closes #77.

Five commands existed and nothing ran any of them. backups/ did not exist on the host, so make backup had never been run once.

A timer alone would not have been the fix. The issue is explicit about the bar — "whatever runs it should alert on not having run, not just on failing" — and that is most of this change. A job that fails is loud; a job that stops being run is silent, and silence is also what a healthy week looks like. Same inversion watchdog.rules.yaml is built on.

What runs

Job	Command	When	Alerts if not seen in
backup-volumes	make backup	Sundays 03:30	14 days
verify-backups	make backup ARGS='--verify-only --all'	daily 05:30	3 days
backup-firewall	make backup-firewall	daily 04:30	3 days
snmp-verify	make snmp-verify	Wednesdays 06:30	14 days
verify-key-backup	human, make secrets-verify-backup KEY=…	no timer	90 days

make check-digests went to a scheduled GitHub Actions workflow instead — it needs no host and no secret, which makes it the only one of the five that is genuinely off-host.

How the "did not run" part works

Every run, timer or human, goes through scripts/run-scheduled.sh, which records four gauges into a textfile Alloy's existing unix exporter scrapes. No compose change was needed — compose.yaml already bind-mounts / at /rootfs:ro, so only the textfile block in config.alloy is new.

Thresholds are not literals in the rules. install-timers.sh writes homelab_job_max_age_seconds from the same JOBS table that decides the cadence, and the rules join against it. So no rule names a job, a sixth timer needs no rule edit, and "declared but never ran" is unless on(homelab_job) rather than six hardcoded absent() calls.

make check-timers derives each timer's real period from systemd-analyze calendar and refuses a threshold under twice it. That assertion caught snmp-verify declaring ten days against a weekly timer while this was being written.

Three findings that shaped it

The label is homelab_job, not job. discovery.relabel "metrics" sets job on every target from that exporter and scrapes default to honor_labels: false, so a job label in a .prom file arrives as exported_job and every rule keyed on it matches nothing — loaded, healthy, unfireable, the #63 shape exactly. backup.test.yaml has a case that pins this.

> on(homelab_job) silently drops instance. Caught by the unit test asserting instance and promtool returning an alert without it. group_left() is what keeps the left side's labels; without it a second monitored host would collide with the first.

backup-volumes.sh's flock does not cover --verify-only — the case block handling it returns before the lock is taken. So a nightly verify and a weekly backup could overlap today, and the backup's prune could delete a set the verifier was mid-read on. The two units share a lock name, which is what closes that.

Alertmanager is unchanged

warning + correctness already reaches default. No new route, and deliberately no new ROUTES row: it would traverse the same route as the existing capacity row, breaking the one-row-one-assertion correspondence and making both "(7 assertions)" comments false.

Verification

• Six rule mutations each caught by the unit tests — hardcoded threshold, severity drift, dropped exclusion, removed for:, removed group_left, wrong join label
• Wrapper: failure leaves last_success intact, lock timeout exits 75, SIGTERM still records
• Four classes of schedule drift caught by --check
• Textfile path end to end against the pinned Alloy image — 13 series, correct labels, node_textfile_scrape_error 0
• make validate and make lint clean

Corrected while verifying: a truncated .prom does not blank the whole directory. It raises node_textfile_scrape_error and costs only that file. The atomic write still matters, but for one job rather than all of them — four comments said otherwise and are fixed.

What this does not solve

The host still verifies its own backups. The external cron-monitor bounds that — a dead host does not read as healthy backups — but genuinely off-host verification is #92 and #99, both still open. The runbook says so in its first section rather than burying it.

Caution

The weekly backup stops the monitoring stack, which stops the Watchdog heartbeat. The run is normally ~90s against a 15m grace, but check yours — docs/runbooks/verify-the-alert-path.md now records the coupling. This is why that timer is the only one with RandomizedDelaySec=0.

Deploy step: make install-timers on the monitoring host. The installer refuses to run from a worktree, since the units carry absolute paths.

🤖 Generated with Claude Code