Repository navigation
Nothing schedules any backup, digest check or key verification #77
Description
Nothing in the repository schedules anything. No cron entry, no systemd timer,
no scheduled workflow. Every one of these is a command someone has to remember:
| Command | What lapses if it is not run |
|---|---|
make backup |
volume snapshots — and backups/ does not exist on the host, so this has never been run |
make backup-firewall |
the pfSense config export |
make secrets-verify-backup KEY=… |
proof that the age backup key still decrypts |
make check-digests |
drift between pinned digests and the registry |
make snmp-verify |
that each device still answers to its own community |
backup-firewall.sh:125-126 says the quiet part already:
This is on the same host as everything else it protects. Copy it to the backup
target and offsite.
Nothing does that copy.
The age key is the sharpest edge. verify-key-backup.sh is careful, thorough
work — it runs with env -u SOPS_AGE_KEY -u XDG_CONFIG_HOME HOME=<tmp> so the
live key cannot be silently used, refuses the live key by device:inode, and
refuses a copy sitting inside a git tree. It is also deliberately excluded from
make validate (Makefile:230-238, for good reasons — it needs a private key
on disk), which means it has no schedule and no reminder either. The one thing
that proves the secrets are recoverable runs only when someone thinks of it.
Shape of a fix
A systemd timer on the monitoring host is the obvious answer, but it makes the
host responsible for verifying its own backups. roadmap.md already carries
"Move stack deployment from make up over SSH to something pull-based" — the
same reasoning applies, and the two probably want solving together.
Whatever runs it should alert on not having run, not just on failing. See
the dead-man's-switch issue.
Found while verifying #12.
Activity
Gerrrt commentedon Aug 26, 2026
Surfaced while going through awesome-selfhosted for the ADR-0008 service build-out (#129–#150) — noting it here rather than opening a duplicate issue, since this is a proposed mechanism for the gap this issue already describes.
Healthchecks
BSD-3-ClausePython/Docker— awesome-selfhostedREADME.md, Automation section. Listens for pings and alerts when they are late.The fit here is that it inverts the problem. A cron entry that fails silently is exactly the failure this issue is about; a job that must check in turns "it did not run" into a positive signal instead of an absence. That is the same reasoning
security.rules.yamlapplies withFirewallLogsStopped, and the same reasoning behind the dead man's switch that closed #67 in #119.Applied to the four jobs named here:
make backupmake backup-firewallmake check-digestsmake secrets-verify-backupIt also alerts through ntfy, so it lands on the notification path this estate already has rather than needing a new receiver.
One caveat, which is #67's caveat again: the watcher cannot live on the host it watches. If Healthchecks runs on the Winterfell mini PC alongside the jobs it monitors, a dead box takes both. Two options that avoid it — run it on
oracle(which needs a purpose anyway, #94, and this is a genuinely tiny workload well within a dual-core A6-9200), or use the hosted free tier for this one thing on the same reasoning #67 used for keeping the outermost watchdog external.Not filing this as its own issue — if the approach is right it is an implementation detail of this one.
Generated by Claude Code