Skip to content

No dead man's switch on the notification path #67

Closed

Description

@Gerrrt

AlertmanagerNotificationsFailing catches delivery errors — a refused
connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable,
and pointed at nothing: a 200 into a deleted ntfy topic is a successful
notification by every measure Alertmanager has.

That matters here because the webhook was the ntfy.example.invalid placeholder
for the entire life of the stack and nothing noticed — that is recorded in
docs/roadmap.md under Done. The same failure can recur silently at any time,
and the only symptom is that alerts stop arriving, which is also what a healthy
week looks like.

The repository already applies exactly this reasoning elsewhere.
security.rules.yaml has FirewallLogsStopped because a firewall is never
silent, and security.rules.yaml:241-250 explains at length why there is
deliberately no SuricataStopped rule — "absence of alerts is indistinguishable
from absence of the service". The notification path is the one place that
reasoning was not applied to itself.

What this needs

A dead man's switch: a rule that always fires, routed to a receiver that expects
it, with something outside the stack watching for its absence. ntfy can do
this with a scheduled check; so can a healthchecks.io-style ping.

The awkward part is that the watcher has to live somewhere other than this host,
or it fails at the same time as the thing it is watching.

Found while verifying #12.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions