You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AlertmanagerNotificationsFailing catches delivery errors — a refused
connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable,
and pointed at nothing: a 200 into a deleted ntfy topic is a successful
notification by every measure Alertmanager has.
That matters here because the webhook was the ntfy.example.invalid placeholder
for the entire life of the stack and nothing noticed — that is recorded in docs/roadmap.md under Done. The same failure can recur silently at any time,
and the only symptom is that alerts stop arriving, which is also what a healthy
week looks like.
The repository already applies exactly this reasoning elsewhere. security.rules.yaml has FirewallLogsStopped because a firewall is never
silent, and security.rules.yaml:241-250 explains at length why there is
deliberately no SuricataStopped rule — "absence of alerts is indistinguishable
from absence of the service". The notification path is the one place that
reasoning was not applied to itself.
What this needs
A dead man's switch: a rule that always fires, routed to a receiver that expects
it, with something outside the stack watching for its absence. ntfy can do
this with a scheduled check; so can a healthchecks.io-style ping.
The awkward part is that the watcher has to live somewhere other than this host,
or it fails at the same time as the thing it is watching.
Activity
docs(adr): decide what oracle is for — the off-host jobs (#94)