[hhff]

Builds on #106 (merge that first). Monitoring, part 2 of 5: the kit reads what production already shows, with the access the project already has, and turns it into observations.

What

• Probes (kind: probe). A project file, project/monitors/<name>.mjs, that the supervisor writes per project and a reviewer reads. The kit runs main's committed blob, never a working tree, with:
  • a timeout (probe_timeout_s, 60) and output caps (2 MB, 5,000 events a run)
  • DEVTEAM_PARAMS, DEVTEAM_SINCE (its last cursor) and each credential as an environment variable, read from the encrypted store with the secret sinks' grammar (rails-credentials:, sops:, file:)
  • the CLIs in uses checked installed and logged in first (heroku auth:whoami and friends), so a lapsed login fails the monitor instead of blinding it
  • back-off on failure, doubling up to an hour
• Streams (kind: stream), for sources with only a live tail (heroku logs --tail). wait keeps a collector (monitor stream <name>) running alongside the supervisor and restarts it, at most three times in ten minutes; it stops itself 30 minutes after wait last ran. Poll drains its spool. The rollup says how much of the day it listened.
• The scrubber. Emails, phone numbers, card numbers (Luhn-checked) and SSN-like numbers, IPv4 and IPv6, JWTs, bearer tokens, URL credentials, cookies, keys with known prefixes, long random-looking strings, and values under keys named like secrets or personal data. Each becomes a typed placeholder. It runs after the redactor (every credential value the kit read) and before anything is written. The owner adds patterns in monitor_scrub (an owner setting, monitor-change); monitor scrub < lines tries them. Classes, paths, line numbers, timestamps, versions, ids and shas stay.
• What reaches the log, on subject monitor:<name>:
  • exception, log-error and platform-error: the first time a fingerprint qualifies for triage (the monitor's triage level, count and users). Fatal is 4, error is 3. Each carries the class, the scrubbed message, in-app frames mapped to repo paths, the release, counts, users and the provider link.
  • spike: a fingerprint with a day of history running at spike.factor times its 7-day hourly median, and at least spike.min. Once per spell.
  • deploy: each release, with its commit.
  • A probe's daily rollup: events by level, warnings as templates and counts, and how the probe did (runs, failures, invalid lines, run time).
• Local only (run/monitors/<name>/): hourly counts (14 days), the latest event per fingerprint (cache_days), and metrics.
• wait also returns once on a severity-4 event. Its poll budget is the slowest probe's timeout plus a minute.
• Commands.
  • monitor test <name> runs this checkout's probe and settings once, validates every line, and prints a redacted sample framed as untrusted. A worker testing an edited probe gets no credentials.
  • monitor show <fingerprint> shows the cached event, framed.
  • monitor discover [--record] finds where the app runs, error trackers, CLIs and whether they can see the app, production URLs and health routes. It proposes what isn't watched and lists gaps.
  • monitor stream.
  • monitor scrub.
• Findings.
  • monitor-failing: 3 runs in a row.
  • monitor-stale: not polled in 3× its interval while the supervisor runs.
  • probe-drift: invalid lines after a clean run.
  • probe-costly: timed out, over the caps, over budget, or quota low.
  • monitor-discovery-due: weekly.
• Dashboard. Probe cards, an exception feed (level, what, where, 24h sparkline, counts, release, provider link, its observation), and the probe sidepeek (settings with credential names only, state, today, invalid lines, what it read, history).
• Docs. supervisor.md Monitoring (probes, streams, discovery steps, production text as data), the probe review checklist in quality.md §3, monitors' access in code-style §10, and the constitution row.

No new kit files, and no probes ship in the kit.

Why

The owner asked for production signals (errors, logs, deploys) to flow into the OODA loop with as much kept as possible, scrubbed of personal data. They also asked that the kit use access the project already has and ask only for gaps. Discovery is the step that finds that access.

Not in this PR

• Triage, ignores, incidents, incident-fix and fix tickets come in PR 3.
• Deploy matching and the verify window come in PR 4.
• Remediations come in PR 5.

Tests

• The scrubber. A corpus of real-looking lines, including what must survive.
• Probes, end to end. The test covers:
  • refusing a probe not yet on main
  • monitor test from the checkout
  • credentials redacted everywhere: test output, observations, errors, every file under run/
  • each fingerprint observed once
  • a severity-4 alert
  • deploy, show, the feed and the peek
  • drift, failing and back-off
  • a spike, once per spell
  • the rollup
  • a stream collector spooled and drained, then its death noticed
  • discovery and --record

Suite: everything passes except the tmux session test, which fails on main in this environment too.

---

Generated by Claude Code

[hhff]

Upstream loop: merged #106's update, adversarial review of this PR's diff

I merged monitoring-uptime (now including main and #106's review fixes) in as a merge commit, with no rebase or force-push. The conflicts were in monitorEntries/monitorList, pollMonitor's lock, the pollMonitorLocked seed, docs and tests. One more clash came from main: main's link prober is also called runProbe, so the monitor probe runner is now runMonitorProbe.

Findings in this PR's diff

1. Blocker, fixed. Probe code ran with production credentials under a ladder-class answer.
   • The probe's file (project/monitors/*.mjs) landed as an ordinary supervisor-owned project/ change.
   • The entry that turns it on (kind, probe, credentials, uses) was monitor-change, which a chat-relayed answer or rung 3 settles.
   • Now changing any of those, per monitor, is governance-edit (owner_only, never from chat), and so is any file under project/monitors/. URL checks and thresholds stay monitor-change. See monitorOwnerChanges in ownerSettingsChanged.
   • Test: a chore that turns on a probe and adds its file, under a monitor-change answer, is refused naming monitors.errs (governance-edit) and the file.
2. Blocker, fixed. monitor test gave the supervisor's run credentials for an unmerged probe or entry. Credentials now go only to main's probe blob under main's entry, whoever runs the test. The test now checks for "credentials": false before the probe lands, and for the redacted credential after.
3. Should-fix, mostly covered by 2. A worker's monitor test could start the logged-in uses CLIs. It now runs without credentials. Deferred: the CLIs it starts still run in the owner's login session. Restricting a worker to probes with no uses would close that, and is noted for the owner.
4. Should-fix, fixed. monitor_scrub could weaken the built-in scrubber. The built-in rules now run before and after the owner's patterns, and a pattern over 200 characters is skipped.
5. Should-fix, fixed. Monitoring 1: uptime and response time #106's fixes are now carried into this PR's code.
   • The name is set last, and a name field is refused.
   • The monitor is seeded from the log only when state.json is missing.
   • monitorProblems now checks the probe path, credential names (not DEVTEAM_*), uses command names, kind, and params, which may not contain credentials.
   • The event URL in monitor show and the discovered URLs are shown without userinfo or query.
6. Nit, fixed in Monitoring 3: triage and fix #110. A hand-written observation can no longer silence alerts:
   • monitor lines count only when the supervisor or the owner wrote them;
   • observe --detector monitor is refused.
7. Nit, deferred. monitor show prints frame file names outside the untrusted frame.
8. Nit, deferred. ensureStreams passes the full environment to the collector, and probe_timeout_s has no cap.

Needs the owner's decision

• Making the code-running parts of a monitor governance-edit changes the trust model this PR proposed, in which all of a monitor was monitor-change. Discovery now asks one governance-edit question for the probes and one monitor-change question for the URL checks and thresholds (supervisor.md says so). If the owner prefers a new owner_only class (probe-change, say), that is a constitution change.

Tests: the full test/kit.test.mjs on Node 24.14.1 at the pushed head: 133/133 pass.