feat(sensor): vendor FoxIO's JA4+ scripts into stacks/sensor (#776) by Gerrrt · Pull Request #783 · Gerrrt/HomeLab

Gerrrt · 2026-10-01T13:29:15Z

Refs #776. It stays open until JA4 is live on fenrir, with a JA4 from a guest's outbound TLS and a Loki query returning it, which are its done-criteria. This PR is the vendoring and an offline replay. It also carries #782 (merged in), so the ADR-0069 note and the ~0% baseline agree whichever merges first. Implements ADR-0069.

What

The vendored package. FoxIO's ja4-zeek-scripts at 8bf9feca52790ae8c926c6a8d79a3547ac19541a, in stacks/sensor/zeek/ja4/:
- the runtime scripts only (no btest suite or traces);
- LICENSE (FoxIO License 1.1), LICENSE-JA4 (BSD 3-Clause) and zkg.meta;
- NOTICE (ours), which says which licence covers what and that this repo's MIT licence doesn't extend to the directory;
- VENDORED (ours), which records the upstream URL, the commit and the date.
Wiring. compose.yaml mounts the directory read-only beside local.zeek, which @load ./ja4. No image is built, and the zeek/zeek digest and every check are unchanged.
scripts/vendor-ja4.sh <40-char sha> re-vendors at a full SHA (a tag or branch moves), keeping NOTICE and rewriting VENDORED. --check re-fetches the recorded commit and diffs byte for byte. It passes.
Fields added:
- ja4 and ja4s → ssl.log;
- ja4h → http.log;
- ja4t, ja4ts, ja4l and ja4ls → conn.log;
- new ja4ssh.log and ja4d.log, which Alloy already ships, since it tails every *.log.
The methods enabled are FoxIO's defaults: all but JA4X.

Keeping it byte-identical

Upstream has trailing spaces and missing final newlines in 15 files. Editing them would make every re-vendor's diff partly ours and break --check. So instead:

.editorconfig-checker.json excludes ^stacks/sensor/zeek/ja4/, with the reason written in its comment block. Checked with the pinned editorconfig-checker v4.0.2: the whole tree passes, while the same files without the exclusion fail with 80 errors.
.gitattributes marks the directory -whitespace linguist-vendored, so git diff --check is clean and GitHub's language stats ignore it.
There are no CRLF files, so a fresh clone is byte-identical too.

Proof (offline, on `fenrir`, with the pinned `zeek/zeek:9.0.0`)

zeek -a local /zeek/site/local.zeek with the package loaded: parses.
A replay of a TLS session captured on Saruman (curl → Cloudflare) wrote:
- ssl.log: ja4=t13d3013h1_1d37bd780c83_8537cf56674e and ja4s=t130200_1302_234ea6891581;
- conn.log: ja4t, ja4ts, ja4l and ja4ls all populated.

Not yet deployed

The stack deploys from main on fenrir. After merge: git pull and docker compose -f stacks/sensor/compose.yaml up -d there. ADR-0069's measurement then compares a week of capture_loss.log percent_lost and stats.log pkts_dropped against the baseline. The baseline is ~0% since #782's GRO fix, not the 8.1% ADR-0069 first recorded.

Checks

These pass:

check_docs.py;
markdownlint-cli2, yamllint --strict and shellcheck;
check_image_pins.py and check_compose_health.py;
the pinned editorconfig-checker and git diff --check.

🤖 Generated with Claude Code

…f on ens19 (#437) Nothing was being dropped. Every segment reached tap190i1 in order, and pkts_dropped was 0. GRO on fenrir's ens19 held data segments while Saruman's ACKs, sent within microseconds, overtook them. Zeek saw ACKs above holes, on host-bound flows only, since guests ACK too slowly to overtake. Same 10 MB download to Saruman: 50,680 and 69,504 bytes missed with GRO on, 0 with it off, still 0 after two reboots of fenrir. - runbook §3: capture-offloads.service. It hangs off multi-user.target, because dracut brings ens19 up before switch-root and a device-wanted unit never ran (measured). - ADR-0069: a NOTE that its 8.1% baseline was this artefact; the real one is about 0. - changelog: the investigation, and dracut's brief IPv6 link-local on ens19. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ADR-0069, implemented. ja4-zeek-scripts at 8bf9feca52790ae8c926c6a8d79a3547ac19541a in stacks/sensor/zeek/ja4/: runtime scripts only, with LICENSE (FoxIO 1.1), LICENSE-JA4 (BSD) and our NOTICE. The directory is mounted read-only beside local.zeek, which @loads it. No image is built and no pin changes. - scripts/vendor-ja4.sh <sha> re-vendors at a full SHA, keeping NOTICE and rewriting VENDORED. --check re-fetches and diffs byte for byte, and passes. - Byte-identical means upstream's whitespace stays: one EditorConfig exclusion (the pinned checker v4.0.2 passes the tree; without the exclusion, 80 errors), plus .gitattributes -whitespace linguist-vendored. - Proved on fenrir offline with the pinned zeek/zeek:9.0.0. zeek -a parses. A replayed TLS session wrote ja4/ja4s to ssl.log and ja4t/ja4ts/ja4l/ja4ls to conn.log. - README, security.md and changelog updated. Not deployed until this merges; the before/after loss measurement starts at deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…437) Copilot review on #782: §3 said to change only the netplan file and .env when the NIC is not ens19, but capture-offloads.service names it in BindsTo=, After= and ExecStart=. Following the old text on such a NIC leaves GRO on and the gaps back, silently. It now lists every place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

The vendor checks permit a missing NOTICE, field documentation is incomplete, and completion depends on deployment verification and unmerged PR #782.

Review effort: Balanced
Findings: 2 · 3

Open (5)

Do not close #776 before deployment and Loki verification · New
Require NOTICE before reporting a successful check · New
Resolve #782 before merging the corrected baseline · New
Document JA4L delta fields in the Conn::Info inventory · New
Document JA4L delta fields in runtime integration comments · New

What changed in this PR

Vendors FoxIO’s JA4+ Zeek scripts and integrates fingerprint logging into the sensor stack.

Changes:

Adds the pinned JA4+ runtime scripts and licensing metadata.
Mounts and loads the scripts in Zeek.
Adds re-vendoring tooling and operational documentation.

File	Description
`.editorconfig-checker.json`	Excludes byte-identical vendor files.
`.gitattributes`	Marks JA4 files as vendored.
`docs/changelog.md`	Records integration and deployment status.
`docs/security.md`	Documents JA4+ security visibility.
`scripts/vendor-ja4.sh`	Adds update and verification workflow.
`stacks/sensor/README.md`	Documents fields, licensing, and updates.
`stacks/sensor/compose.yaml`	Mounts the vendored package read-only.
`stacks/sensor/zeek/ja4/LICENSE`	Adds FoxIO License 1.1.
`stacks/sensor/zeek/ja4/LICENSE-JA4`	Adds JA4 BSD licence.
`stacks/sensor/zeek/ja4/NOTICE`	Explains licensing boundaries.
`stacks/sensor/zeek/ja4/VENDORED`	Records upstream commit and date.
`stacks/sensor/zeek/ja4/__load__.zeek`	Loads enabled fingerprint modules.
`stacks/sensor/zeek/ja4/config.zeek`	Defines default module settings.
`stacks/sensor/zeek/ja4/ja4/__load__.zeek`	Loads JA4 components.
`stacks/sensor/zeek/ja4/ja4/helpers.zeek`	Collects TLS client-hello data.
`stacks/sensor/zeek/ja4/ja4/main.zeek`	Generates JA4 fingerprints.
`stacks/sensor/zeek/ja4/ja4d/__load__.zeek`	Loads JA4D components.
`stacks/sensor/zeek/ja4/ja4d/consts.zeek`	Defines DHCP fingerprint constants.
`stacks/sensor/zeek/ja4/ja4d/main.zeek`	Generates JA4D fingerprints.
`stacks/sensor/zeek/ja4/ja4h/__load__.zeek`	Loads JA4H components.
`stacks/sensor/zeek/ja4/ja4h/main.zeek`	Generates HTTP fingerprints.
`stacks/sensor/zeek/ja4/ja4l/__load__.zeek`	Loads JA4L components.
`stacks/sensor/zeek/ja4/ja4l/main.zeek`	Generates latency fingerprints.
`stacks/sensor/zeek/ja4/ja4s/__load__.zeek`	Loads JA4S components.
`stacks/sensor/zeek/ja4/ja4s/main.zeek`	Generates TLS server fingerprints.
`stacks/sensor/zeek/ja4/ja4ssh/__load__.zeek`	Loads JA4SSH components.
`stacks/sensor/zeek/ja4/ja4ssh/main.zeek`	Generates SSH fingerprints.
`stacks/sensor/zeek/ja4/ja4t/__load__.zeek`	Loads JA4T components.
`stacks/sensor/zeek/ja4/ja4t/main.zeek`	Generates TCP fingerprints.
`stacks/sensor/zeek/ja4/ja4x/__load__.zeek`	Provides disabled JA4X loader.
`stacks/sensor/zeek/ja4/utils/common.zeek`	Provides shared formatting utilities.
`stacks/sensor/zeek/ja4/utils/ssl-consts.zeek`	Defines TLS constants.
`stacks/sensor/zeek/ja4/zkg.meta`	Adds upstream package metadata.
`stacks/sensor/zeek/local.zeek`	Loads the vendored package.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

… close-out corrected (#776) Copilot review on #783: - --check excluded NOTICE from the diff without requiring it, so deleting the licence notice still read as a clean match. Both --check and re-vendoring now refuse when NOTICE is missing or empty. Tested both ways. - conn.log also gets ja4l_delta and ja4ls_delta (ja4l/main.zeek:61-62); the README and local.zeek now list them. - #776's criteria are a live JA4 from a guest and a Loki query, so this refers to #776 rather than closing it, and the changelog says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nto gerrrt/ja4-vendor-776 # Conflicts: # docs/changelog.md

Conflict in docs/changelog.md: both sides added a 2026-10-01 entry; both kept. Also removes a stray diff3 marker ('||||||| parent of c101757 ...') that #781's merge left in main's changelog between the #414 and odin entries; the text either side of it is intact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nto gerrrt/ja4-vendor-776

# Conflicts: # docs/changelog.md

Gerrrt and others added 2 commits October 1, 2026 06:20

Copilot AI balanced review requested due to automatic review settings October 1, 2026 13:29

Copilot started reviewing on behalf of Gerrrt October 1, 2026 13:29

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/changelog.md Resolved

Comment thread scripts/vendor-ja4.sh Resolved

Comment thread docs/changelog.md Resolved

Comment thread stacks/sensor/README.md Resolved

Comment thread stacks/sensor/zeek/local.zeek Resolved

Gerrrt and others added 4 commits October 1, 2026 06:35

Merge remote-tracking branch 'origin/gerrrt/sensor-capture-gro-off' i…

0788e51

…nto gerrrt/ja4-vendor-776 # Conflicts: # docs/changelog.md

Merge remote-tracking branch 'origin/gerrrt/sensor-capture-gro-off' i…

f4a40aa

…nto gerrrt/ja4-vendor-776

Gerrrt enabled auto-merge (squash) October 1, 2026 13:51

Merge remote-tracking branch 'origin/main' into gerrrt/ja4-vendor-776

b3a637d

# Conflicts: # docs/changelog.md

Gerrrt merged commit 76ef9f6 into main Oct 1, 2026
7 checks passed

Gerrrt deleted the gerrrt/ja4-vendor-776 branch October 1, 2026 13:53

This was referenced Oct 1, 2026

feat(scratch): a disposable investigation stack whose age is what gets noticed (#438) #784

Merged

docs(sensor): JA4+ is live on fenrir, and #776 closes (#776) #926

Merged

Navigation Menu

Repository navigation

feat(sensor): vendor FoxIO's JA4+ scripts into stacks/sensor (#776) - #783

feat(sensor): vendor FoxIO's JA4+ scripts into stacks/sensor (#776)#783
Gerrrt merged 8 commits into
mainfrom
gerrrt/ja4-vendor-776

Gerrrt commented Oct 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Gerrrt commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Keeping it byte-identical

Proof (offline, on fenrir, with the pinned zeek/zeek:9.0.0)

Not yet deployed

Checks

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Gerrrt commented Oct 1, 2026 •

edited

Loading

Proof (offline, on `fenrir`, with the pinned `zeek/zeek:9.0.0`)