Repository navigation
feat(orchestrator): implement incident control loop skeleton on fakes (Phase 1B) - #23
Conversation
Implement the LangGraph State model and explicit field reducers to manage domain contract state across control loop nodes without implicit dict merging. Implements build-plan step B1.1.
…seam Add 15 discrete node modules under orchestrator/nodes/ implementing the full incident control loop from architecture §2.2 and error recovery from §2.9. Each node is an async function taking State and Deps and returning state updates, utilizing Deps protocols for complete decoupling and fake execution without sibling internal imports. Also update common logging to prevent closed-file errors during test capture fixtures. Implements build-plan step B1.2.
…or edges Assemble and compile the LangGraph StateGraph connecting all 15 control loop nodes. Wire conditional routing for tournament arbitration outcomes (decided winner vs. ambiguous or unviable candidate) and safety kernel formal verification verdicts (pass vs. veto or uncertain). Route node-level exceptions to handle_failure per architecture §2.9 loop failure handling, and guarantee terminal node execution of teardown_fleet and record_run. Implements build-plan step B1.3.
…ent_id Implements build-plan step B1.4. Enables LangGraph state checkpointing across incident response execution by bridging LangGraph's BaseCheckpointSaver interface to Understudy's CheckpointStore protocol. Checkpoints, channel blobs, and pending writes are keyed by incident_id (as thread_id), ensuring intermediate state transitions are durable, retrievable, and auditable while respecting architectural import boundaries (no LangGraph dependencies in store).
Add the ust demo --fake runner and ust graph CLI commands to enable end-to-end traversal of the orchestrator StateGraph on deterministic fakes and inspection of the compiled control loop diagram. Incorporate explicit force-veto flag routing through FakeSafetyKernel to exercise escalation branches, enforce strict clock determinism for synthetic alert timestamps, and maintain full test suite coverage and type-check gates. Implements build-plan step B1.5 and Checkpoint B1.
Every terminal path of the control loop must either resolve production, page a human, or record a FAILED run. Four paths did none of those. A production remediation that applied but did not resolve the incident (apply_to_production -> False) was marked FAILED and sent on to Slack, teardown, and record_run with nobody paged. actuate now leaves the outcome open and the new route_actuation router diverts to escalate_pagerduty, so the record is ESCALATED with prod_outcome=not_resolved and a human owns it. record_run was the only node with an unconditional edge to END, so a run store outage was swallowed into state.errors and run_incident returned an EXECUTED record that was never persisted while run_demo raised. Its edge is now conditional on errors like every other, reaching handle_failure. handle_failure tore twins down without unregistering them, leaking mirror registrations on every failure after register_mirrors, and paged unconditionally, so a teardown crash after a successful escalation paged on-call twice and rewrote ESCALATED to FAILED. Cleanup is now best-effort per step and always reaches the record write, paging only when no escalation was sent and appending cleanup errors to escalation_reason. observe short-circuited on non-empty evidence. Because the checkpointer is keyed by incident_id, a resumed incident re-forked its fleet but reused the previous attempt's evidence, so the kernel verified and actuate applied a winner chosen on twins that no longer existed. Also: State.resolved_outcome lets errors override a set outcome (never an escalation); checkpoint channel versions carry LangGraph's random suffix so two branches of one thread cannot collide on a blob key; build_graph rejects an unusable checkpointer instead of silently compiling without persistence; apply_candidates pairs plans to twins by candidate_index rather than list position; and Deps carries a clock threaded through every timestamping node, making run_demo(clock=FrozenClock(t)) actually deterministic for the eval harness. Full suite green at 100% branch coverage, ruff and mypy --strict clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XRtrUZZvN7JazpLQKD5SrE
Code review of the Phase 1B control loop — findings and fixesReviewed the full High / medium — fixed1. A failed production remediation paged nobody. 2. 3.
4. Resumed incidents actuated on evidence from twins that no longer existed. 5. Low — fixed
Known limitation, not changedIf the loop fails before Verification
One judgement call worth flagging: the escalation branch skips 🤖 Generated with Claude Code |
Build-Plan Step Implemented
Checkpoint Execution & Output
ADRs Relied On
Mock Registry Disclosures
Documentation Updates
Definition of Done
make checkgreen (ruff, mypy --strict, import-linter, pytest)# MOCKED:marker without a row in docs/MOCKS.md