Skip to content

Arm the segmentation alert with tripwire rules - #226

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/arm-segmentation-tripwire
Sep 1, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/arm-segmentation-tripwire

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Closes #223. The firewall change is already applied to morpheus; this is the repository catching up with it.

The obvious fix does not work

#223 said to enable logging on the inter-VLAN pass rules. That was wrong, and checking the rule set before touching it is what caught it: every inter-VLAN pass rule is sourced from an internal segment (vlan50 → vlan99, vlan50 → vlan30). No packet any of them passes can have a terminal VLAN as its source, and the alert needs src ∈ {10,20,40} and dst ∈ {30,50,99}. Logging them would have populated the dashboard's pass panels and left the alert exactly as dead.

Logging the terminal VLANs' own → any egress rules does work, and costs:

Interface Inbound pkts/day vs current filterlog
igc0.20 Skids/IoT 11,293,285 127×
igc0.40 CasaBonita/TV 968,266 11×
igc0.10 Degens/guest 312,031 4×
combined 12,573,581 ≈142×

On a 30-day retention on one disk, that is a disk-filling change.

What was done instead

Three tripwire rules on morpheus, one per terminal interface — pass + log for <terminal net> → Internal_Segments, placed below the eight block rules that already stop that path and above the → any egress rule:

pass in log quick on igc0.40 inet from <OPT3__NETWORK> to <Internal_Segments>
pass in log quick on igc0.20 inet from <OPT5__NETWORK> to <Internal_Segments>
pass in log quick on igc0.10 inet from <OPT6__NETWORK> to <Internal_Segments>

While segmentation holds the blocks match first, so the tripwire logs nothing and adds no volume. It matches only if those blocks are removed or reordered — the exact failure the alert exists for — and in that case → any would have passed the packet anyway, so nothing is weakened by its presence.

Applied through pfSsh.php (write_config() + filter_configure()), not by hand-editing config.xml. Encrypted backups taken before and after: config-20260901T042022Z (62 rules) and config-20260901T044457Z (65). Gateway, DNS, outbound HTTPS and log ingestion all verified afterwards; exactly 3 pass rules log, and no others.

Proving it, rather than assuming it

A tripwire that never fires is indistinguishable from a broken one — which is this entire family of defect. The monitoring host is on VLAN 99 and cannot originate terminal-segment traffic, so the logging path was proven instead: logging was briefly enabled on the lowest-volume terminal egress rule (Degens, ~3.6 lines/s) and then reverted.

  • 49 of 49 resulting pass lines carried the source address in exactly the position the alert's regex reads.
  • The destination half of that same regex already matched 2058 block lines.

Both halves are therefore verified against real traffic. Only the combination is absent, which is what "the segmentation is holding" looks like.

What changes here

  • Dashboard — no longer claims the alert cannot fire. Terminal→internal passes now reads 0 (that path is instrumented, so zero is a measurement); Passed (1h) still reads not logged, because ordinary egress genuinely is not.
  • docs/observability.md, docs/security.md — record why the obvious fix fails, what was done, and how it was verified.
  • docs/roadmap.md — TerminalSegmentReachedInternalNetwork cannot fire — the firewall logs blocks only #223 moves to Done.
  • docs/runbooks/restore-the-firewall.md — a new verification step. A config restored from a backup older than today drops the tripwires silently, they log nothing either way, and the alert goes quietly back to being unfirable. That trap is worth more than the rules themselves.

make validate passes: 7 dashboards, 140 panels, 132 PromQL and 28 LogQL expressions, docs OK.

🤖 Generated with Claude Code

TerminalSegmentReachedInternalNetwork matches {app="filterlog",
action="pass"} against a firewall that logged only blocks, so it could
not fire for any input. The control was described in security.md and not
actually watched.

The obvious fix does not work. Every inter-VLAN pass rule is sourced from
an internal segment (50->99, 50->30), so no packet any of them passes can
have a terminal VLAN as its source; logging them would populate the
dashboard's pass panels and leave the alert exactly as dead. What does
work is logging the terminal VLANs' own `-> any` egress rules, and that
costs ~12.6M lines a day against 88.7k today — 142x, on a 30-day
retention on one disk.

So instead: three tripwire rules on morpheus, one per terminal interface,
each a pass + log for `<terminal net> -> Internal_Segments`, placed below
the eight block rules that already stop that path and above the `-> any`
egress rule. While segmentation holds the blocks match first and the
tripwire logs nothing, so it adds no volume. It matches only if those
blocks are removed or reordered — the exact failure the alert exists for
— and in that case `-> any` would have passed the packet anyway, so
nothing is weakened by its presence.

A tripwire that never fires is indistinguishable from a broken one, which
is this entire family of defect, so the path was proven rather than
assumed. Logging was briefly enabled on the lowest-volume terminal egress
rule: 49 of 49 resulting pass lines carried the source address in exactly
the position the alert's regex reads, and the destination half of that
regex already matched 2058 block lines. Both halves verified against real
traffic; only the combination is absent, which is what holding looks like.

The firewall itself is not in this repository. What is here: the dashboard
no longer claims the alert cannot fire, the two pass-dependent stats now
say the right kind of nothing (Terminal->internal passes reads 0, because
that path is instrumented; Passed (1h) still reads "not logged", because
ordinary egress genuinely is not), and the restore runbook checks for the
tripwires — a config restored from a backup older than today drops them
silently and the alert goes quietly back to being unfirable.

Backups taken before and after: config-20260901T042022Z (62 rules) and
config-20260901T044457Z (65).

Closes #223

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 1, 2026 04:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The dashboard’s noValue change can mask missing/absent data as a numeric 0, and the new restore verification commands are overly broad and should be made more specific to avoid false results.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the security observability docs and Grafana dashboard text to reflect the new pfSense “tripwire” pass+log rules that instrument terminal → internal traffic, allowing TerminalSegmentReachedInternalNetwork to become meaningful without exploding log volume.

Changes:

  • Update the security-overview Grafana dashboard panel descriptions to reflect tripwire-based pass logging and treat terminal→internal passes as an instrumented signal.
  • Document the tripwire approach, rationale, and verification steps in docs/security.md and docs/observability.md.
  • Add a firewall-restore verification step for the tripwire rules and move #223 to Done in the roadmap.
File summaries
File Description
stacks/observability/grafana/dashboards/security-overview.json Updates dashboard copy and panel “no value” behavior to reflect tripwire pass logging.
docs/security.md Documents the tripwire rules as the enforcement/verification mechanism for terminal-segment isolation.
docs/runbooks/restore-the-firewall.md Adds a restore verification step to ensure tripwire rules still exist after restoring older backups.
docs/roadmap.md Moves #223 to Done and records the rationale for tripwire rules vs broad pass logging.
docs/observability.md Expands observability narrative explaining why prior alerting was unfireable and how tripwires fix it.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/runbooks/restore-the-firewall.md Resolved
Two review findings on #226, both right.

noValue fires when a query returns no series, not when it returns a
measured zero. Setting it to "0" therefore claims a measurement in
exactly the case where the stream is broken, absent or relabelled — the
failure these panels exist to make visible.

Worse, it contradicted the dashboard's own text. The caveat panel says a
quiet IDS and a stopped one produce identical output, and then Suricata
alerts (24h) rendered "0", which asserts quiet. All three count stats now
read "none": the query matched nothing, and why is left to the Firewall
log arrival rate panel next door, which is the thing that can actually
tell a quiet stream from a stopped one. Passed (1h) keeps "not logged",
which is a different and still-true claim about instrumentation.

The restore-runbook checks were also too broad. `grep "^pass" | grep -c
" log "` expecting 3 passes the moment any other rule legitimately logs a
pass, while a tripwire is missing. They now match the whole shape —
pass + log + quick + terminal interface + <Internal_Segments> — and a
second check asserts all three interfaces are present rather than one of
them three times. Both verified against the live ruleset: 3, and
igc0.10/igc0.20/igc0.40.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 1b521cf into main Sep 1, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/arm-segmentation-tripwire branch September 1, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TerminalSegmentReachedInternalNetwork cannot fire — the firewall logs blocks only

2 participants