Skip to content

feat(backup): pull Jellyfin's state off smaug from a ZFS snapshot, encrypt it here and copy it to oracle (#484) - #554

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/erebor-apps-backup-193704
Sep 20, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/erebor-apps-backup-193704

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 19, 2026

Copy link
Copy Markdown
Owner

What changed

erebor/apps was declared backed up in three files and backed up by nothing (#484). This builds the mechanism ADR-0045 decides:

  • scripts/backup-nas.sh on the monitoring host pulls Jellyfin's /config out of the newest periodic ZFS snapshot of erebor/apps over the inert 10.0.99.20 → 10.0.40.30:22 pass ADR-0016 wrote for exactly this, streams it into age here (plaintext only inside the ssh session), verifies it against a measured sentinel, and copies the set to oracle under backups/nas by the helpers make backup's volume sets stay on the host they protect: ADR-0015's copy to oracle is decided and unbuilt #535 landed today. Jellyfin never stops: the snapshot is the quiesce. A snapshot older than two days fails the run by name.
  • /config becomes a bind mount at /mnt/erebor/apps/jellyfin/config, so the dataset §4 names is the one that holds the state; jellyfin-cache stays a named volume and is DISPOSABLE by name. STACK=media backup-volumes.sh --inventory now reports nothing to archive, which is the true answer on this host.
  • backup-volumes.sh is sourceable: one guard before its argument parsing makes the tables, verify(), the set helpers and the far-side helpers a library for both scripts. jellyfin-config's sentinel is ./data/jellyfin.db, read off a boot of the pinned image (appears ~30 s into a first start; the WAL beside it is where the writes are; a clean stop checkpoints it).
  • homelab-backup-nas.timer (Saturdays 03:30, jitter and replay allowed since nothing is stopped), a JOBS row, and make verify-backups, which the existing nightly unit now runs so one job proves both directories, here and on oracle.
  • Docs: ADR-0045 with the three rejections (a second rule on the 9100 path; TrueNAS replication, which is smaug-initiated and outside age, the sentinel check and CI; encrypting on the NAS, which ADR-0016 said and gets a third NOTE clause); build-the-nas.md §4.1 (the first recorded snapshot schedule, ADR-0040 d.4), §6 migration block, §6.2 deploy checklist, §6.3 restore; the media stack's compose, env and README say what is and is not backed up; security.md, network.md, roadmap.md say port 22 is inert until §6.2; the schedule and handover runbooks gain the job.

Why

Two documents asserting a backup no script performs is worse than no backup — the next reader has no reason to look — and it is the shape #428 found in the sensitive tier, arriving a second time. Refs #484, refs #535, refs #413.

Blast radius

  • No change to network segmentation or firewall rules — the port-22 pass has existed since 2026-09-16; §6.2 switches the service on behind it, on smaug, by hand
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml — frodo on smaug gets the operator's public key; oracle receives age ciphertext to the estate's two recipients

Verification

  • make validate passes; shellcheck clean on both scripts; install-timers.sh --check passes the 2× rule for backup-nas
  • Benched end to end against a snapshot-shaped directory on oracle over real ssh, with the copy to a bench directory there: run (23 entries, sentinel present, copied and hashed on the far side), --list both sides, --verify-only --all both sides, --copy-only, make verify-backups (both halves run even when the first fails). Negative tests: a stale snapshot, a wrong zone (name in the future), an unreachable host, an unreadable path — each dies naming its cause. Bench directories removed on both sides; no real backups/nas was written on oracle.
  • Not deployed, and cannot be from here: smaug's SSH is off and nothing on it is reachable from the monitoring host. build-the-nas.md §6.2 is the checklist — snapshot task (§4.1), user frodo, ACL, SSH on, host key, the tar-to-/dev/null proof, migration of the existing volume (§6), first make backup-nas, then make install-timers. Until the first run, the nightly verify-backups will report no complete sets in backups/nas; §6.2 orders the first run before the reinstall for that reason. The issue closes on §6.2's Done block.
  • Docs updated

🤖 Generated with Claude Code

…t, encrypted here and copied to oracle — the backup three files claimed and nothing performed (#484)

Three files said erebor/apps was backed up and named backup-volumes.sh as
the mechanism; that script had no sentinel for the volume, cannot see
smaug's Docker, and the volume was on erebor/ix-apps anyway, so the
dataset §4 called backed up held the compose file and nothing Jellyfin
writes. ADR-0045 settles the mechanism and the three rejections, and this
builds it.

The pull runs on the monitoring host over the 99 → 40:22 pass ADR-0016
wrote for exactly this and left inert: scripts/backup-nas.sh logs in as
one unprivileged user, tars Jellyfin's /config out of the newest periodic
snapshot of erebor/apps, streams it into age here, verifies it against a
sentinel read off a boot of the pinned image (./data/jellyfin.db appears
about thirty seconds into a first start; the WAL beside it is where the
writes are), and copies the set to oracle by the helpers #535 landed
today. Jellyfin never stops: the snapshot is the quiesce, atomic across
db, -wal and -shm, and a snapshot older than two days fails the run by
name rather than producing a backup that stopped being current in
silence. /config becomes a bind mount on erebor/apps so the dataset the
table names is the one that holds the state; the runbook migrates what
the named volume already holds.

backup-volumes.sh gains a source guard just before its argument parsing,
so the tables, verify(), the set helpers and the far-side helpers are one
library for both scripts. Sets land in backups/nas/, apart from the volume
sets, because the nightly verification checks a set against the current
stack's volume list; a new `make verify-backups` walks both directories
and the existing timer runs it. homelab-backup-nas.timer is Saturday
03:30, jitter and replay allowed since nothing is stopped.

Benched end to end against a directory on oracle over real ssh, with the
copy to a bench directory there: the run, --list, --verify-only --all,
--copy-only, and the four failure modes (stale snapshot, wrong zone,
unreachable host, unreadable path) each behave as the runbook says.
Nothing here has touched smaug, whose SSH is still off: build-the-nas.md
§6.2 is the deploy checklist — snapshot task, user frodo, ACL, SSH on,
host key, the tar-to-/dev/null proof, first run, timers — and the issue
closes on its Done block, not on this merge.

Refs #484, refs #535, refs #413.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Gerrrt
Gerrrt force-pushed the gerrrt/erebor-apps-backup-193704 branch from 112d078 to ca494ed Compare September 20, 2026 00:28
@Gerrrt
Gerrrt merged commit fecf971 into main Sep 20, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/erebor-apps-backup-193704 branch September 20, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant