[Gerrrt]

What changed

Two dns probe modules and two alert rules that notice when the house stops filtering DNS — AdGuardNotAnswering (does it answer at all?) and AdGuardNotFiltering (is it still blocking, or merely alive?). Both probes are aimed straight at port 53 on the filter, under a new blackbox-dns scrape job. Nothing changes in the running stack today: the two targets ship written out and commented, because AdGuard does not exist yet.

Why

Closes #126, and the monitoring half of #123 — the same failure, filed twice.

ADR-0010 leaves the public upstreams in Unbound's forwarder list so a dead AdGuard costs filtering rather than connectivity. Right trade, and not free: it converts a loud failure into a silent one. The only symptom is advertisements reappearing, which nobody reports and which is indistinguishable from a blocklist that never covered a given site.

Nothing already here would catch it — the containers rules need an agent on a box that does not exist, InstanceDown covers a host not a container, network.rules.yaml does not touch DNS. And the trap worth stating out loud: a probe down the normal resolver path always passes, because the fallback is doing its job.

Three design points:

• A separate scrape job, not a label. EndpointUnreachable is critical and routes to urgent, which a filter designed to fail open must never do; EndpointNameNotResolving joins via=dns to via=address on name, a join that means nothing when the target is a resolver. A role!= matcher on the existing rule would quietly capture the next target somebody adds without the label.
• Two questions, not one. A blocklist update that silently stopped working leaves a perfectly healthy service filtering nothing, and the liveness probe stays green through it. AdGuardNotFiltering carries an and on (name) join so a stopped AdGuard raises one alert and not two — the ContainerHighMemory cannot fire — no service sets a memory limit #63 hazard, since a join that stops matching passes promtool check rules and can never fire.
• A real blocklist entry as the canary, not a rule of our own. A custom AdGuard rule would be more stable and would test the wrong thing: it lives in AdGuard's own config and would keep working while every downloaded list failed to load.

Both rules are warning, so they route to default rather than urgent. Nothing is down, nobody is blocked, and ADR-0010 already says filtering here is a convenience rather than a control.

Blast radius

None today — the two targets are commented out, so the new job discovers nothing and neither rule has a series to evaluate. When #102 enables them, the blast radius is one warning notification to the existing default receiver; no new receiver, no routing change.

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

make validate passes (all checks passed, 2 skipped — not a deployment checkout). promtool test rules is green across all 9 test files, check rules across all 11 rule files, and Prometheus was booted against the comment-only targets/blackbox-dns.yaml to confirm an empty file_sd file loads without error.

The probes were measured against a real AdGuard Home before either rule was written — the image pulled locally, the default AdGuard DNS filter loaded (180,105 rules), probed from a throwaway prom/blackbox-exporter:v0.28.0:

Scenario	dns_answers	dns_filters
AdGuard healthy, default blocking mode	1	1
Blocklists not loaded (filtering_enabled: false)	1	0
Blocking mode changed to NXDOMAIN	1	0
AdGuard stopped	0	0
Asked Unbound at 10.0.99.1 instead — the trap	1	0

That last row is the point of the whole change: the resolver the household actually talks to passes a liveness check and fails a filtering check, which is exactly the wrong answer a probe on the normal path would have given.

Two smaller things checked rather than assumed: doubleclick.net really is on the default filter and comes back as a single 0.0.0.0 A record (six real A records unfiltered, so the probe discriminates), and blackbox v0.28.0 fails a validator over an empty answer section rather than passing vacuously — otherwise a NOERROR/NODATA reply would have looked "filtered".

The four rows above are the four cases in tests/dns.test.yaml.

• make validate passes
• Deployed to the lab and confirmed working — nothing to deploy: the targets are disabled and the box does not exist
• Docs updated

Follow-up

Enabling the two targets belongs to #102, and its roadmap entry now says so. The target file carries the address assumption to correct (10.0.99.40, the next free Winterfell reservation) and the two commands to verify before uncommenting — including the negative one against Unbound, because a filtering probe that cannot go red is measuring nothing.

#123's Monitoring entry said this was blocked on #102. It was right that the probe cannot go through morpheus and right that "answering" and "filtering" are two questions; it was wrong about one thing, and the entry has been corrected: what could not be written today was the target, not the rules.

🤖 Generated with Claude Code