Skip to content

docs(alerting): the ntfy cutover as run, with its times (#136) - #722

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/ntfy-cutover-record-136
Sep 29, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/ntfy-cutover-record-136

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 29, 2026

Copy link
Copy Markdown
Owner

What changed

  • docs/runbooks/verify-the-alert-path.md records the Sensitive tier: ntfy, to bring the notification path in-house #136 cutover as run: a UTC timeline read from Caddy's access log, ntfy's cache and Prometheus's ALERTS, what the run found, and what was not run.
  • The runbook steps are corrected where the run showed them to be wrong.
  • docs/changelog.md gets an entry for 2026-09-29.

Why

The runbook says the times must be written down before the issue is closed. A notification path is believed only after it has been observed working.

Observed:

  • Both halves of the alert path: the in-house copy over Wi-Fi, and the ntfy.sh copy over mobile data, both in the new format.
  • The kill drill: docker stop sensitive-ntfy at 03:12:05; EndpointUnreachable paged through ntfy.sh at 03:18.
  • The in-house retries: after the restart, Alertmanager delivered the held-back in-house pages.
  • AlertmanagerNotificationsFailing: it fired at 03:24:26 with reason="serverError", reporting the drill.

Not run, and recorded as such: the lowered-Watchdog check and the category=security page.

Corrected in the steps:

Blast radius

Docs only.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • make validate passes
  • Deployed and confirmed working: this PR is the record of that
  • Docs updated

Closes #136

🤖 Generated with Claude Code

verify-the-alert-path.md gets the record of the cutover, in the shape
the dead man's switch drill already uses: a table of UTC times read from
Caddy's log, ntfy's cache and Prometheus's ALERTS.

It covers:
- ntfy's first leaf, the deny-all checks and the phones.
- The first in-house and first templated deliveries.
- The synthetic page seen on the ntfy.sh copy with Wi-Fi off.
- The kill drill, stop at 03:12:05 to page on ntfy.sh at 03:18.
- The retried in-house deliveries after restart.
- AlertmanagerNotificationsFailing firing afterwards, as it should.

The two parts of step 8 that were not run (the lowered Watchdog and the
security page) are written down as not run.

The steps are corrected where the run showed them wrong:
- amtool is only in the container.
- A secrets edit dirties the deployment checkout and needs a PR.
- A URL change needs `make render`.
- Resolve before re-firing.
- The kill drill's timing, and the NotificationsFailing tail.

Plus a changelog entry for the day.

Closes #136

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit b8fd0a8 into main Sep 29, 2026
7 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/ntfy-cutover-record-136 branch September 29, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sensitive tier: ntfy, to bring the notification path in-house

1 participant