Repository navigation
docs(alerting): the ntfy cutover as run, with its times (#136) - #722
Merged
Merged
Conversation
verify-the-alert-path.md gets the record of the cutover, in the shape the dead man's switch drill already uses: a table of UTC times read from Caddy's log, ntfy's cache and Prometheus's ALERTS. It covers: - ntfy's first leaf, the deny-all checks and the phones. - The first in-house and first templated deliveries. - The synthetic page seen on the ntfy.sh copy with Wi-Fi off. - The kill drill, stop at 03:12:05 to page on ntfy.sh at 03:18. - The retried in-house deliveries after restart. - AlertmanagerNotificationsFailing firing afterwards, as it should. The two parts of step 8 that were not run (the lowered Watchdog and the security page) are written down as not run. The steps are corrected where the run showed them wrong: - amtool is only in the container. - A secrets edit dirties the deployment checkout and needs a PR. - A URL change needs `make render`. - Resolve before re-firing. - The kill drill's timing, and the NotificationsFailing tail. Plus a changelog entry for the day. Closes #136 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
docs/runbooks/verify-the-alert-path.mdrecords the Sensitive tier: ntfy, to bring the notification path in-house #136 cutover as run: a UTC timeline read from Caddy's access log, ntfy's cache and Prometheus'sALERTS, what the run found, and what was not run.docs/changelog.mdgets an entry for 2026-09-29.Why
The runbook says the times must be written down before the issue is closed. A notification path is believed only after it has been observed working.
Observed:
docker stop sensitive-ntfyat 03:12:05;EndpointUnreachablepaged through ntfy.sh at 03:18.AlertmanagerNotificationsFailing: it fired at 03:24:26 withreason="serverError", reporting the drill.Not run, and recorded as such: the lowered-Watchdog check and the
category=securitypage.Corrected in the steps:
amtoolruns throughdocker execinto the container; it isn't installed on the host.make renderbefore Alertmanager sees it.NotificationsFailingtail after it.Blast radius
Docs only.
secrets/*.sops.yamlVerification
make validatepassesCloses #136
🤖 Generated with Claude Code