Repository navigation
make backup overwrites its own last-good archive, and nothing verifies it #64
Description
Makefile:259-267:
backup: ## Back up the stack's volumes to ./backups/
@mkdir -p backups
@for v in prometheus-data loki-data grafana-data alertmanager-data; do \
docker run --rm -v $(STACK)_$$v:/data -v "$(PWD)/backups:/backup" \
alpine tar czf "/backup/$$v.tar.gz" -C /data . ; \
doneFive distinct problems in seven lines:
- A failed run destroys the last good backup. The filename is fixed,
tar
truncates on open, so an interrupted or failing run leaves a short archive
where the previous working one was. There is no timestamp and no rotation. - Nothing verifies the result. Compare
backup-firewall.sh:118-125, which
decrypts what it just wrote and greps for<pfsense>before claiming
success. This target truststar's exit status. alloy-datais missing from the list. It holds the WAL and the log file
positions — restoring without it replays or drops whatever was in flight.prometheus-datais tarred while Prometheus is writing it. A hot copy of
an open TSDB may not be restorable, and nothing tests whether it is.- There is no restore target.
restore-the-firewall.md:168says of the
firewall runbook that "until that has been done once, this runbook is a
hypothesis" — this has no runbook at all.
Also: backups/ does not exist on the monitoring host, so this has never been
run.
Related: unpinned alpine in the same recipe, and nothing schedules any backup.
Found while verifying #12.
Activity
Gerrrt commentedon Aug 26, 2026
From the selfh.st directory — a candidate that would resolve this by replacing the hand-rolled script rather than patching it.
The two faults in
make backup— it overwrites its own last-good archive, and nothing verifies the result — are both solved-problem territory. Atarinto a fixed filename is what every backup tool exists to stop being.restic (~35.7k stars) gives you the three properties this issue is missing, by construction:
restic check --read-dataverifies, so "nothing verifies it" gets a command rather than a plan.Backrest (~7.2k stars) wraps restic with scheduling, retention policy and a web UI — which would also cover #77 for the backup jobs specifically, though not for
make check-digestsormake secrets-verify-backup. Worth considering as a pair.Kopia (~14k stars) is the closest alternative and equally defensible; Borg (~13.7k) is the veteran option but is SSH-target-oriented in a way that fits this estate less well.
Note this also disposes of #65 — an unpinned
alpineimage evading the CI pinning checks stops being a question once the backup is a pinned restic image or a host binary rather than an ad-hoc container.The sequencing that matters: #92 says the firewall backup sits on the machine it protects, and the estate has no off-host target at all. A better backup tool writing to the same disk is still one disk. The repository worth creating first is the one on the NAS (#95), with
oracleas a plausible second copy (#94). Tool choice is the easy half.Generated by Claude Code