Repository navigation
Arm the segmentation alert with tripwire rules - #226
Merged
Merged
Conversation
TerminalSegmentReachedInternalNetwork matches {app="filterlog",
action="pass"} against a firewall that logged only blocks, so it could
not fire for any input. The control was described in security.md and not
actually watched.
The obvious fix does not work. Every inter-VLAN pass rule is sourced from
an internal segment (50->99, 50->30), so no packet any of them passes can
have a terminal VLAN as its source; logging them would populate the
dashboard's pass panels and leave the alert exactly as dead. What does
work is logging the terminal VLANs' own `-> any` egress rules, and that
costs ~12.6M lines a day against 88.7k today — 142x, on a 30-day
retention on one disk.
So instead: three tripwire rules on morpheus, one per terminal interface,
each a pass + log for `<terminal net> -> Internal_Segments`, placed below
the eight block rules that already stop that path and above the `-> any`
egress rule. While segmentation holds the blocks match first and the
tripwire logs nothing, so it adds no volume. It matches only if those
blocks are removed or reordered — the exact failure the alert exists for
— and in that case `-> any` would have passed the packet anyway, so
nothing is weakened by its presence.
A tripwire that never fires is indistinguishable from a broken one, which
is this entire family of defect, so the path was proven rather than
assumed. Logging was briefly enabled on the lowest-volume terminal egress
rule: 49 of 49 resulting pass lines carried the source address in exactly
the position the alert's regex reads, and the destination half of that
regex already matched 2058 block lines. Both halves verified against real
traffic; only the combination is absent, which is what holding looks like.
The firewall itself is not in this repository. What is here: the dashboard
no longer claims the alert cannot fire, the two pass-dependent stats now
say the right kind of nothing (Terminal->internal passes reads 0, because
that path is instrumented; Passed (1h) still reads "not logged", because
ordinary egress genuinely is not), and the restore runbook checks for the
tripwires — a config restored from a backup older than today drops them
silently and the alert goes quietly back to being unfirable.
Backups taken before and after: config-20260901T042022Z (62 rules) and
config-20260901T044457Z (65).
Closes #223
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The dashboard’s noValue change can mask missing/absent data as a numeric 0, and the new restore verification commands are overly broad and should be made more specific to avoid false results.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the security observability docs and Grafana dashboard text to reflect the new pfSense “tripwire” pass+log rules that instrument terminal → internal traffic, allowing TerminalSegmentReachedInternalNetwork to become meaningful without exploding log volume.
Changes:
- Update the
security-overviewGrafana dashboard panel descriptions to reflect tripwire-based pass logging and treat terminal→internal passes as an instrumented signal. - Document the tripwire approach, rationale, and verification steps in
docs/security.mdanddocs/observability.md. - Add a firewall-restore verification step for the tripwire rules and move #223 to Done in the roadmap.
File summaries
| File | Description |
|---|---|
| stacks/observability/grafana/dashboards/security-overview.json | Updates dashboard copy and panel “no value” behavior to reflect tripwire pass logging. |
| docs/security.md | Documents the tripwire rules as the enforcement/verification mechanism for terminal-segment isolation. |
| docs/runbooks/restore-the-firewall.md | Adds a restore verification step to ensure tripwire rules still exist after restoring older backups. |
| docs/roadmap.md | Moves #223 to Done and records the rationale for tripwire rules vs broad pass logging. |
| docs/observability.md | Expands observability narrative explaining why prior alerting was unfireable and how tripwires fix it. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Two review findings on #226, both right. noValue fires when a query returns no series, not when it returns a measured zero. Setting it to "0" therefore claims a measurement in exactly the case where the stream is broken, absent or relabelled — the failure these panels exist to make visible. Worse, it contradicted the dashboard's own text. The caveat panel says a quiet IDS and a stopped one produce identical output, and then Suricata alerts (24h) rendered "0", which asserts quiet. All three count stats now read "none": the query matched nothing, and why is left to the Firewall log arrival rate panel next door, which is the thing that can actually tell a quiet stream from a stopped one. Passed (1h) keeps "not logged", which is a different and still-true claim about instrumentation. The restore-runbook checks were also too broad. `grep "^pass" | grep -c " log "` expecting 3 passes the moment any other rule legitimately logs a pass, while a tripwire is missing. They now match the whole shape — pass + log + quick + terminal interface + <Internal_Segments> — and a second check asserts all three interfaces are present rather than one of them three times. Both verified against the live ruleset: 3, and igc0.10/igc0.20/igc0.40. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #223. The firewall change is already applied to
morpheus; this is the repository catching up with it.The obvious fix does not work
#223 said to enable logging on the inter-VLAN pass rules. That was wrong, and checking the rule set before touching it is what caught it: every inter-VLAN pass rule is sourced from an internal segment (
vlan50 → vlan99,vlan50 → vlan30). No packet any of them passes can have a terminal VLAN as its source, and the alert needssrc ∈ {10,20,40}anddst ∈ {30,50,99}. Logging them would have populated the dashboard's pass panels and left the alert exactly as dead.Logging the terminal VLANs' own
→ anyegress rules does work, and costs:igc0.20Skids/IoTigc0.40CasaBonita/TVigc0.10Degens/guestOn a 30-day retention on one disk, that is a disk-filling change.
What was done instead
Three tripwire rules on
morpheus, one per terminal interface —pass+logfor<terminal net> → Internal_Segments, placed below the eight block rules that already stop that path and above the→ anyegress rule:While segmentation holds the blocks match first, so the tripwire logs nothing and adds no volume. It matches only if those blocks are removed or reordered — the exact failure the alert exists for — and in that case
→ anywould have passed the packet anyway, so nothing is weakened by its presence.Applied through
pfSsh.php(write_config()+filter_configure()), not by hand-editingconfig.xml. Encrypted backups taken before and after:config-20260901T042022Z(62 rules) andconfig-20260901T044457Z(65). Gateway, DNS, outbound HTTPS and log ingestion all verified afterwards; exactly 3 pass rules log, and no others.Proving it, rather than assuming it
A tripwire that never fires is indistinguishable from a broken one — which is this entire family of defect. The monitoring host is on VLAN 99 and cannot originate terminal-segment traffic, so the logging path was proven instead: logging was briefly enabled on the lowest-volume terminal egress rule (Degens, ~3.6 lines/s) and then reverted.
passlines carried the source address in exactly the position the alert's regex reads.Both halves are therefore verified against real traffic. Only the combination is absent, which is what "the segmentation is holding" looks like.
What changes here
0(that path is instrumented, so zero is a measurement); Passed (1h) still reads not logged, because ordinary egress genuinely is not.docs/observability.md,docs/security.md— record why the obvious fix fails, what was done, and how it was verified.docs/roadmap.md— TerminalSegmentReachedInternalNetwork cannot fire — the firewall logs blocks only #223 moves to Done.docs/runbooks/restore-the-firewall.md— a new verification step. A config restored from a backup older than today drops the tripwires silently, they log nothing either way, and the alert goes quietly back to being unfirable. That trap is worth more than the rules themselves.make validatepasses: 7 dashboards, 140 panels, 132 PromQL and 28 LogQL expressions,docs OK.🤖 Generated with Claude Code