[Gerrrt]

Closes #100.

make dashboards-export pulls every dashboard back out of the running Grafana by uid and writes it over the file in grafana/dashboards/, so the loop is edit → one command → git diff instead of a hand copy out of Dashboard settings → JSON Model.

The issue's design could not work as written

The plan was to pull each dashboard from the API and write it back. That is a guaranteed no-op while allowUiUpdates is false, and the issue's premise about what that flag does is not quite right: Grafana does not discard a UI edit at the next restart, it refuses to store one at all.

POST /api/dashboards/db  →  400 {"message":"Cannot save provisioned dashboard"}

So the API can only ever return the file it was provisioned from. Every export would have succeeded, exited zero, written nothing, and left git diff reporting no change to something plainly different on the screen. Measured against the running stack before anything was built: the API's copy of homelab-docker was identical to the committed file in every field but id and version.

allowUiUpdates is therefore now true. The JSON stays the source of truth — a file change re-provisions over Grafana's copy — but an edit now survives long enough to be exported.

(Two smaller corrections to the issue: it says editable: false, but all seven dashboards carry editable: true; only allowUiUpdates was false.)

What pays for the flip

false bought a real property: the running dashboard and the committed one could not disagree. That is bought back explicitly rather than dropped.

• make dashboards-export ARGS=--check writes nothing and exits non-zero when Grafana holds an edit git does not. The new daily dashboards-drift timer runs it, so forgetting to export ages into a stale job with an alert behind it rather than into work quietly lost weeks later. The rules in backup.rules.yaml join against the JOBS table rather than naming jobs, so this needed no rule.
• make check-dashboard-roundtrip asserts in CI that a save to a provisioned dashboard is still accepted. Flipping the flag back breaks the export silently, so the setting is asserted rather than trusted.

Two findings that would have made every diff unreadable

• Grafana serialises keys alphabetically at every level, while these files put uid, title and description first (and a panel's type and title first). A naive write-back would have reordered every key in all seven files and buried the one line that changed. The committed order is preserved; only keys Grafana added are appended.
• Grafana persists whatever the browser was showing at save time. The time picker's range and each variable's selection and cached option list are read back out of the file rather than taken from the API — otherwise one person's afternoon of debugging silently becomes everyone's default time range and default host filter. Those are the one thing the round trip will not write, and the dashboards README says so.

The round-trip check

scripts/check_dashboard_roundtrip.sh boots the pinned Grafana image against a throwaway copy of the provisioning — no secret, no published port, no live stack — and asserts three things:

1. the committed JSON is what Grafana produces from it;
2. a save to a provisioned dashboard is accepted;
3. that saved edit travels through export_dashboards.py into a file.

Verified in both directions. With allowUiUpdates: false it fails and names the setting; with a hand-mangled ups-power.json it fails and prints the diff. It also caught a real regression during this work — a dashboard I had reverted mid-testing.

Verification

• make validate — all checks passed, 2 skipped (both worktree-expected: systemd-analyze verify and the deployment-host check).
• make check-dashboard-roundtrip — 3/3, ~45s.
• Export run against the live stack, then --check clean on the second run, so it is idempotent.
• The dashboard JSON changes in this PR are that normalisation applied once: seven stale "version": 1 literals dropped and escaped em dashes written as UTF-8, which three of the seven files already did. 15 lines total.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The new homelab-dashboards-drift.service is missing Docker service dependency/group settings needed for docker compose to work reliably under systemd.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds an automated “export dashboards from Grafana back into git” workflow for the observability stack, including a CI round-trip guard and a scheduled drift detector, and flips Grafana provisioning to allow UI saves so exports can capture edits.

Changes:

• Add make dashboards-export (live stack) and make check-dashboard-roundtrip (offline/CI) to ensure exports are meaningful and stable.
• Flip Grafana dashboard provisioning to allowUiUpdates: true, and introduce a daily dashboards-drift systemd timer to detect unexported UI edits.
• Normalize committed dashboards to the exporter’s canonical form (drop volatile version, normalize em-dashes).

File summaries

File	Description
systemd/homelab-dashboards-drift.timer	Adds daily scheduling for the drift check job.
systemd/homelab-dashboards-drift.service	Runs make dashboards-export ARGS=--check under systemd to detect drift.
stacks/observability/README.md	Updates operator guidance to use make dashboards-export and drift job.
stacks/observability/grafana/provisioning/dashboards/dashboards.yaml	Sets allowUiUpdates: true and documents the tradeoff and mitigations.
stacks/observability/grafana/dashboards/ups-power.json	Removes volatile version.
stacks/observability/grafana/dashboards/security-overview.json	Removes volatile version.
stacks/observability/grafana/dashboards/network-snmp.json	Removes volatile version.
stacks/observability/grafana/dashboards/logs-explorer.json	Removes volatile version.
stacks/observability/grafana/dashboards/host-overview.json	Removes volatile version and normalizes em-dash escapes.
stacks/observability/grafana/dashboards/docker-containers.json	Removes volatile version and normalizes em-dash escapes.
stacks/observability/grafana/dashboards/observability-stack.json	Removes volatile version and normalizes em-dash escapes.
stacks/observability/grafana/dashboards/README.md	Documents export workflow, preserved/dropped fields, and ordering guarantees.
scripts/validate.sh	Adds the dashboard round-trip check into make validate accounting.
scripts/install-timers.sh	Registers the new dashboards-drift job for timer installation/thresholds.
scripts/export-dashboards.sh	Live-stack exporter wrapper (auth, fetch, orphan sweep) calling canonicalizer.
scripts/export_dashboards.py	Canonicalizer: strips volatile fields, preserves session state, preserves key order, supports --check.
scripts/check_dashboard_roundtrip.sh	CI/offline validation that Grafana round-trips and accepts saves for provisioned dashboards.
Makefile	Adds dashboards-export and check-dashboard-roundtrip targets.
docs/runbooks/schedule-maintenance.md	Documents the new scheduled job and how to respond to drift.
docs/roadmap.md	Moves #100 into Done with the final implementation notes.
docs/observability.md	Updates dashboard maintenance docs to the new export workflow.
.github/workflows/ci.yml	Runs the dashboard round-trip check in CI.

Review details

Suppressed comments (1)

systemd/homelab-dashboards-drift.service:34

• dashboards-export shells out to Docker; under systemd, User=robo won't inherit the docker group membership from a login session. Without SupplementaryGroups=docker, access to /var/run/docker.sock is likely to fail (the same issue the backup/verify units document).

User=robo
Group=robo
WorkingDirectory=/home/robo/code/Gerrrt/HomeLab
Environment=STACK=observability

• Files reviewed: 22/22 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.