TerminalSegmentReachedInternalNetwork cannot fire — the firewall logs blocks only #223

Area

Prometheus / alerting (the rule lives in loki/rules/security.rules.yaml; the root cause is what pfSense ships).

What should happen

TerminalSegmentReachedInternalNetwork should fire when a packet from a terminal VLAN (10 Degens, 20 Skids, 40 CasaBonita) is passed toward an internal segment (30 ImaginationLAN, 50 Hicks, 99 Winterfell). Its own comment says why it matters:

VLANs 10, 20 and 40 are terminal by design — egress only, no path to anything internal. A PASS from one of them to the lab, trusted or management segment means the rule set is not what this repository says it is. There is no benign version of this.

It is the alert the segmentation design in ADR-0002 depends on, and the only automated check that the design still holds.

What happens instead

It cannot fire for any input. The rule matches {app="filterlog", action="pass"}, and this firewall does not log passed packets at all. Across the full 30-day retention the action label has exactly one value — block — at roughly 88,000 lines a day.

This is the same shape as #63, where ContainerHighMemory divided by a memory limit no service sets and so could not fire for any input while showing as loaded and healthy. The rule is syntactically valid, the ruler loads it, scripts/check_loki_rules.sh passes it — because that checks whether LogQL parses, not whether the stream it selects can ever exist.

Nothing else would have caught it either: prometheus/tests/ has promtool unit tests for the metric-based rules, and there is no equivalent for the Loki rules.

Found while building the homelab-security dashboard (#82) — the dashboard is what surfaced it, because it drew the label rather than only alerting on it.

Relevant output

# action label values over the full 30-day retention
$ curl -sG http://localhost:3100/loki/api/v1/label/action/values \
    --data-urlencode "start=<now-30d>" --data-urlencode "end=<now>"
{"status":"success","data":["block"]}

# volume is not the explanation — the stream is busy, it is just all blocks
$ logcli/query: sum(count_over_time({app="filterlog"}[24h]))
88708

# per-day breakdown, last 7d: one series, no `pass` on any day
sum by (action) (count_over_time({app="filterlog"}[1d]))
  {action="block"} -> 80082, 84975, 94391, 88044, 86611, 84564, 80180

# the regex itself is fine — each half matches on its own, they never co-occur
sum(count_over_time({app="filterlog", action="block"} |~ `,10\.0\.(10|20|40)\.[0-9]+,` [24h]))   -> 63
sum(count_over_time({app="filterlog", action="block"} |~ `,10\.0\.(30|50|99)\.[0-9]+,` [24h]))   -> 2058

What is affected

• TerminalSegmentReachedInternalNetwork — the only rule matching action="pass". Dead.
• IoTAttemptedLateralMovement matches action="block" and is live; it currently reads zero because nothing on VLAN 20 is aiming at an internal segment, which is a real measurement rather than a blind spot.
• Four panels on homelab-security. They already say so rather than showing a misleading zero: the two pass-dependent stats read not logged, and the segmentation panel states that absence of a red line is not evidence of absence of a breach.

What would fix it

Enable logging on the inter-VLAN pass rules in pfSense (Firewall → Rules, per-rule Log packets that are handled by this rule), rather than logging all passes — the block stream is already ~88k lines a day and logging every pass would be a different order of magnitude against a 30-day retention on one disk.

Worth deciding at the same time:

• Whether the terminal-VLAN rules are the only ones that need it, or whether every inter-VLAN pass rule should log.
• Whether this wants a companion check so the same class of defect cannot recur silently — something that asserts a rule's selector matches a stream that actually exists, which is the gap check_loki_rules.sh does not cover by design.

Until it is armed, the segmentation design has no automated verification — only the block-side rule, which sees attempts that were stopped and not passes that were not.