Skip to content

feat(soc): mirror the lab bridge to Zeek with tc, and a gauge that proves it (#437) - #762

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/zeek-mirror-port-vlan30-2eaefc
Sep 30, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/zeek-mirror-port-vlan30-2eaefc

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 30, 2026

Copy link
Copy Markdown
Owner

Refs #437. This does not close it: the issue closes on the reboot proof in build-the-sensor-guest.md §7, after fenrir is built.

The decision: tc, not Open vSwitch

The issue says port mirroring needs OVS. It doesn't.

Saruman's vmbr0 is a Linux bridge on eno1, and tc is already installed. A clsact + matchall + mirred egress mirror filter on each port's ingress, plus one on vmbr0 egress, copies every frame on the bridge exactly once. So vmbr0 isn't converted, 10.0.30.110 never moves, and no VLAN-capable bridge goes under the attack segment.

ADR-0064 records this and rejects four alternatives: OVS, a SPAN from neo or the CRS326, hub mode, and Zeek on odin. ADR-0039 gets a NOTE; roadmap.md and security.md are updated.

What ships

  • scripts/zeek-mirror.sh plus a one-minute timer on the hypervisor. It keeps every port mirrored to tap190i1, matched by name. When the sensor is absent it removes the filters.

  • scripts/collect-zeek-mirror-state.sh plus a five-minute timer. This is the read-only collector behind homelab_zeek_mirror_active. It reads 1 only when all four hold:

    • the sensor VM is running;
    • its tap is up;
    • every port is mirrored;
    • the packet counter moved since the last run.

    If VM 190 doesn't exist, it publishes nothing. That way installing it before the guest is built can't page.

  • Installer: both are rows in install-agent-collectors.sh. - in the .prom column marks the non-collector.

  • ZeekMirrorInactive and ZeekMirrorStateStale in host.rules.yaml, each with a firing and a quiet promtool case. The alert counts are updated where check_docs.py asserts them.

  • stacks/sensor:

    • zeek/zeek:9.0.0 with host networking, NET_RAW/NET_ADMIN only and a read-only root, capturing on fenrir's second NIC, which sits alone on a port-less vmbr1;
    • Alloy tailing JSON logs into alexander's Loki as job="zeek", one stream per log_type;
    • no secrets, so it isn't make up (recorded in security.md and .sops.yaml, like media).
  • docs/runbooks/build-the-sensor-guest.md and a fenrir row marked Not built yet in architecture.md.

Found while building it

A sensor restart recreates tap190i1 with a new ifindex. Every existing filter then shows as Egress Mirror to device * and mirrors nothing, with no error anywhere. That's why the ensurer runs every minute and matches the tap by name rather than just checking that a filter exists. It's also why the gauge checks that packets actually moved.

Verification

  • End-to-end in a throwaway network namespace on Saruman (not on vmbr0). This covered:
    • apply, then an idempotent re-run;
    • active=1 with traffic flowing, and active=0 with a flat counter;
    • a sensor restart, which showed mirrored=0/3 with the dangling *;
    • repair on the next run, then teardown.
  • self-tests.sh: 223 fixtures across 17 suites, including 20 new collector fixtures and 5 ensurer fixtures.
  • promtool check rules and promtool test rules on the observability stack (v3.15.0, the pinned version).
  • Passing: check_docs.py, check_sops_rules.py, check_image_pins.py, check_compose_health.py, yamllint --strict, shellcheck, markdownlint-cli2.
  • Not run locally, because this host has no Docker: docker compose config and the image probes. CI covers both. local.zeek was checked against Zeek v9.0.0's script tree, and the runbook parses it on the pinned image (zeek -a) before first start.

Left for the build

  • Create vmbr1 and VM 190, deploy the stack, install both units.
  • The §7 reboot proof: disable the ensurer, reboot, the gauge falls to 0 and ZeekMirrorInactive fires.
  • JA3/JA4 fingerprints aren't in the stock Zeek image. They're a follow-up.

🤖 Generated with Claude Code

…oves it (#437)

#437 said the mirror needs Open vSwitch. It does not: a clsact matchall
mirred filter on each vmbr0 port's ingress copies every frame once, and
vmbr0 stays a Linux bridge. ADR-0064 records the choice and what it rejects.

- scripts/zeek-mirror.sh keeps the mirror on every port, each minute,
  matching the capture tap by name. A sensor restart leaves filters aimed at
  `device *`, mirroring nothing with no error; reproduced in a namespace.
- scripts/collect-zeek-mirror-state.sh reads the same state read-only and
  publishes homelab_zeek_mirror_active. ZeekMirrorInactive and
  ZeekMirrorStateStale have promtool tests.
- stacks/sensor: Zeek 9.0.0 and Alloy on fenrir (VMID 190), logs to
  alexander's Loki and never to VLAN 99. No secrets.
- build-the-sensor-guest.md, including the reboot proof that closes #437.

Refs #437

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt enabled auto-merge (squash) September 30, 2026 18:18
main took ADR-0064 through ADR-0067 meanwhile, so the tc mirror's ADR is
renumbered to ADR-0068, and only this branch's references move with it;
main's ADR-0064 (Immich's library) is untouched. Conflicts in README.md,
docs/changelog.md and docs/observability.md keep both sides, and the rule
counts are re-derived for 107 metric-based rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 8b3aa00 into main Sep 30, 2026
5 of 7 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/zeek-mirror-port-vlan30-2eaefc branch September 30, 2026 21:46
Gerrrt pushed a commit that referenced this pull request Sep 30, 2026
… passes (#437)

alloy fmt --test failed on main since #762 on one inline object:
{ log_type = "" } is written {log_type = ""}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt added a commit that referenced this pull request Sep 30, 2026
… zeek.alloy as alloy fmt writes it (#437) (#767)

* fix(soc): write the tc fixture's tabs as \t, so editorconfig passes (#437)

The self-test fixture in both Zeek mirror scripts is tc's own output, whose
action lines start with a literal tab. editorconfig-checker read those as tab
indentation and failed lint on main. The fixture is now built with $'...'
quoting and \t, byte-identical to the one it replaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(sensor): zeek.alloy as alloy fmt writes it, so Validate configs passes (#437)

alloy fmt --test failed on main since #762 on one inline object:
{ log_type = "" } is written {log_type = ""}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Your Name <you@example.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant