Skip to content

Monitoring 5: remediate - #112

Open
hhff wants to merge 10 commits into
monitoring-verifyfrom
monitoring-remediate
Open

hhff wants to merge 10 commits into
monitoring-verifyfrom
monitoring-remediate

Conversation

@hhff

@hhff hhff commented Nov 5, 2026

Copy link
Copy Markdown
Member

Builds on #111 (merge that first). Monitoring, part 5 of 5: the one place monitoring writes to production. It makes reviewed remediations, on the ladder, with a rollback across a migration as a one-way door.

What

  • remediations. An owner setting, governance-edit. Each entry names a reviewed script under project/remediations/, pinned by its blob sha, with its params, the access it uses (uses CLIs, credentials), its undo and valid_minutes (60).

  • production-remediation, a ladder class: start 1, ceiling 2. No standing rule covers it, and its default can't fire before its deadline, so the owner always sees a remediation coming, or it runs at its default after an outage's --deadline 15m.

  • The answer is bound to what it approves.

    • The question and the decision each name exactly one remediation:<action>; a rollback also names release:<version>.
    • decide refuses an answer about another remediation or release, and an answer that isn't the recommendation.
    • An act in the class only comes from remediate.
  • remediate <action> --decision <d-…> [--to <version>]:

    • runs main's blob, only if it is still the pinned one
    • runs once per decision, with a lock while it runs, and within valid_minutes of the answer
    • redacts and scrubs the output and frames it as untrusted
    • logs the act and observes remediation or remediation-failed

    A writing script isn't killed for printing too much, and a timeout says its effect is unknown and names the undo.

  • A rollback across a production-migration act since its target release first went out (by a deploy a monitor's poll observed, at the deploy's own time) is a one-way door. It needs the owner's own answer, never from chat, to an ask --kind one-way-door with no default.

  • Docs. supervisor.md gets the steps, the constitution gets the class and its row, and quality.md gets the remediation script checklist.

Why

  • Remediations stay narrow. The owner chose reversible remediations on the ladder with a ceiling of 2.
  • Rollbacks across migrations ask the owner. A rollback across a migration can lose data whatever the script says, so it is the owner's call.
  • A pinned sha covers the script body. It keeps a later PR from changing what a listed remediation does without the owner's answer.

Tests

The remediation subtest covers:

  • the refusals: no decision, a forged act, rung 1, an unknown action, two actions, no --blocks, an answer about another action, a "no" answer, and --to on a restart
  • a run with its params and credential, redacted, logged once and refused twice
  • a failing script
  • a changed script on main refused
  • rollback refusals: no target, the wrong target, an unseen release, an ordinary question across a migration, and a chat answer to a one-way door
  • the owner's own one-way-door answer running it

Suite: everything passes except the tmux session test, which fails on main in this environment too.


Generated by Claude Code

claude and others added 9 commits November 5, 2026 04:42
remediations (owner settings, governance-edit) name reviewed scripts under
project/remediations/ with their params, access and undo. remediate <action> --decision
<d-…> runs main's blob of one, once per decision, redacted and scrubbed, logs the act in
production-remediation (start 1, ceiling 2, so the owner always sees one coming or it
runs at its default) and observes what happened. A decision names exactly one
remediation:<action>; an act in the class only comes from remediate. A rollback across a
production-migration act since its target release is a one-way door: it needs the
owner's own answer to a question with no default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T5z3hZtpGaQUGsTaAMQrAg
remediations (owner settings, governance-edit) name reviewed scripts under
project/remediations/, each pinned by its blob sha, with params, access, undo and
valid_minutes. A question and a decision in production-remediation name one
remediation:<action> (and a rollback its release:<version>), and the answer must be the
recommendation; a standing rule never covers the class and its default can't fire
before its deadline. remediate <action> --decision <d-…> runs main's blob only if it is
still the pinned one, once per decision (a lock while it runs), within valid_minutes of
the answer, redacted and scrubbed, logs the act (start 1, ceiling 2) and observes what
happened; a timeout says its effect is unknown and names the undo. A rollback across a
production-migration act since its release went out needs the owner's own answer, not
from chat, to a one-way-door question with no default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T5z3hZtpGaQUGsTaAMQrAg
observe refuses the kit's own detectors, so a deploy line can't be written by hand to
dodge the rollback door. A remediation's default is its recommendation or nothing, and a
defaulted answer counts only when it is. Asking follows Upkeep's hold, and a rollback is
never asked about a release no monitor saw. The lock is taken once the script is about to
run and released if it never started. Settings entries are validated up front. The pinned
blob's exact bytes run. Output past 1 MB says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T5z3hZtpGaQUGsTaAMQrAg
…s decision to the question

- remediate re-checks the question a decision rests on: an owner's
  answer or a default only (no standing rule, no rung), asked in
  production-remediation about this remediation and release, answered
  with its recommendation, or defaulted no sooner than its deadline; a
  decision line written by hand on the owner's no runs nothing
- the script pin is the full 40-character blob id, compared exactly
- a rollback's deploy counts only from the monitors' own lines
- --deadline is for incident-fix and production-remediation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hhff

hhff commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

Upstream loop: merged #111's update, adversarial review of this PR's diff

I merged monitoring-verify in as a merge commit. The conflicts were in:

  • authorise: kept main's owner_rules and this PR's no-rule-for-production-remediation;
  • citeTargets: kept both link and remediate;
  • docs: remediations added to the governance-edit lists;
  • tests.

What I checked. Remediate is the PR that acts on production, so I checked whether it can act beyond what the owner approved.

Findings

  1. Blocker, fixed. A hand-written decision could run any listed remediation, even after the owner said no. The remediation-specific checks ran only in decide, and a decision line is the supervisor's own. remediate now re-checks the question the decision rests on:

    • The authority is answer or default only, never a standing rule or a rung.
    • The question is production-remediation (or the one-way door for a rollback).
    • Its --blocks name this remediation:<action>, and the same release: as the decision.
    • It was answered with its recommendation, or its default was applied at or after its deadline.

    Test: a decision written by hand on the owner's "no" is refused, "not its recommendation".

  2. Should-fix, fixed. A default could fire before its deadline. A hand-written status: defaulted line before the deadline is refused at run time.

  3. Should-fix, fixed. The script pin accepted a 7-character prefix. sha must now be the full 40-character blob id, compared exactly. A prefix could have been brute-forced by a supervisor-merged chore.

  4. Should-fix, fixed by 1. A standing-rule decision is refused at run time.

  5. Should-fix, partly fixed. A forged deploy line could dodge the rollback's one-way door.

    • Fixed: migrationsSince counts only the monitors' own lines.
    • Still open: a line the supervisor writes by hand into its own file looks the same. Failing closed when a release can't be anchored to a tag or commit would close it. Not done here.
  6. Needs the owner's decision. A chat-relayed answer can approve a restart, scale or rollback that isn't a one-way door. production-remediation is a ladder class, so answer --from-chat accepts it. If "the owner always sees each one" is meant literally, remediate should require channel !== 'chat', or the class should join the chat-reserved set. I left this unchanged because it's a trust-model choice.

  7. Nit, deferred. The last 600 characters of the script's output go into the observation's detail. They are redacted and scrubbed but not framed as untrusted.

  8. Nit. The run-once check counts any act carrying the decision. A forged act can only block a run, not cause one.

--deadline (#110 now limits it) is allowed for production-remediation, as this PR's docs use it.

Tests: the full test/kit.test.mjs on Node 24.14.1: 134 of 135 passed. The one failure was main's G1 golden test, which fails on #110's ready line. I fixed that in #110, merged it up, and re-ran G1, which passes. Incidents and remediation pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants