Skip to content

feat(orchestrator): implement incident control loop skeleton on fakes (Phase 1B) - #23

Merged
Harikeshav-R merged 6 commits into
mainfrom
feat/phase-1b-orchestrator
Sep 12, 2026
Merged

Harikeshav-R merged 6 commits into
mainfrom
feat/phase-1b-orchestrator

Conversation

@Harikeshav-R

Copy link
Copy Markdown
Owner

Build-Plan Step Implemented

  • Step: Phase 1B (Steps B1.1–B1.5 & Checkpoint B1)

Checkpoint Execution & Output

$ uv run ust demo --fake --seed 42
ingest -> gather_context -> plan_candidates -> fork_fleet -> register_mirrors -> apply_candidates -> observe -> tournament -> safety_kernel -> actuate -> notify_slack -> teardown_fleet -> record_run
outcome=executed plan=plan_cand_0

$ uv run ust demo --fake --seed 42 --force-veto
ingest -> gather_context -> plan_candidates -> fork_fleet -> register_mirrors -> apply_candidates -> observe -> tournament -> safety_kernel -> escalate_pagerduty -> teardown_fleet -> record_run
outcome=escalated reason=K3 Rollback traverses schema migration boundary

$ uv run ust graph --render docs/generated/graph.png
Rendered control loop graph to docs/generated/graph.png

$ make check
lint: All checks passed!
typecheck: Success: no issues found in 144 source files
check_mocks: OK: Mock registry parity check passed.
lint-imports: Contracts: 2 kept, 0 broken.
pytest: 308 passed in 4.46s (100.00% line & branch coverage)

ADRs Relied On

  • ADR-005 (State representation & reducers)
  • ADR-006 (Dependency injection via Deps protocols)
  • ADR-007 (Fakes in fakes.py for local testing and ust demo --fake)
  • ADR-008 (LangGraph state checkpointing keyed by incident_id)
  • ADR-028 (Deterministic clock and seed isolation)

Mock Registry Disclosures

  • None

Documentation Updates

  • None required (CHECKLIST.md ticked in this PR; docs/04-build-plan.md matched exactly).

Definition of Done

  • Implementation complete for the named build-plan step
  • Unit tests written; 100% line and branch coverage of the added lines
  • make check green (ruff, mypy --strict, import-linter, pytest)
  • Checkpoint command run and output matches, OR marked unverified with a reason
  • Docs updated where this change made them untrue (or "none required", with why)
  • No new # MOCKED: marker without a row in docs/MOCKS.md
  • Conventional commit(s), conventional branch, PR opened, never pushed to main

Harikeshav-R and others added 6 commits September 11, 2026 20:29
Implement the LangGraph State model and explicit field reducers to manage
domain contract state across control loop nodes without implicit dict merging.

Implements build-plan step B1.1.
…seam

Add 15 discrete node modules under orchestrator/nodes/ implementing the full incident control loop from architecture §2.2 and error recovery from §2.9. Each node is an async function taking State and Deps and returning state updates, utilizing Deps protocols for complete decoupling and fake execution without sibling internal imports.

Also update common logging to prevent closed-file errors during test capture fixtures.

Implements build-plan step B1.2.
…or edges

Assemble and compile the LangGraph StateGraph connecting all 15 control
loop nodes. Wire conditional routing for tournament arbitration outcomes
(decided winner vs. ambiguous or unviable candidate) and safety kernel
formal verification verdicts (pass vs. veto or uncertain). Route node-level
exceptions to handle_failure per architecture §2.9 loop failure handling,
and guarantee terminal node execution of teardown_fleet and record_run.

Implements build-plan step B1.3.
…ent_id

Implements build-plan step B1.4.

Enables LangGraph state checkpointing across incident response execution
by bridging LangGraph's BaseCheckpointSaver interface to Understudy's
CheckpointStore protocol. Checkpoints, channel blobs, and pending writes
are keyed by incident_id (as thread_id), ensuring intermediate state
transitions are durable, retrievable, and auditable while respecting
architectural import boundaries (no LangGraph dependencies in store).
Add the ust demo --fake runner and ust graph CLI commands to enable end-to-end traversal of the orchestrator StateGraph on deterministic fakes and inspection of the compiled control loop diagram.

Incorporate explicit force-veto flag routing through FakeSafetyKernel to exercise escalation branches, enforce strict clock determinism for synthetic alert timestamps, and maintain full test suite coverage and type-check gates.

Implements build-plan step B1.5 and Checkpoint B1.
Every terminal path of the control loop must either resolve production,
page a human, or record a FAILED run. Four paths did none of those.

A production remediation that applied but did not resolve the incident
(apply_to_production -> False) was marked FAILED and sent on to Slack,
teardown, and record_run with nobody paged. actuate now leaves the outcome
open and the new route_actuation router diverts to escalate_pagerduty, so
the record is ESCALATED with prod_outcome=not_resolved and a human owns it.

record_run was the only node with an unconditional edge to END, so a run
store outage was swallowed into state.errors and run_incident returned an
EXECUTED record that was never persisted while run_demo raised. Its edge is
now conditional on errors like every other, reaching handle_failure.

handle_failure tore twins down without unregistering them, leaking mirror
registrations on every failure after register_mirrors, and paged
unconditionally, so a teardown crash after a successful escalation paged
on-call twice and rewrote ESCALATED to FAILED. Cleanup is now best-effort
per step and always reaches the record write, paging only when no
escalation was sent and appending cleanup errors to escalation_reason.

observe short-circuited on non-empty evidence. Because the checkpointer is
keyed by incident_id, a resumed incident re-forked its fleet but reused the
previous attempt's evidence, so the kernel verified and actuate applied a
winner chosen on twins that no longer existed.

Also: State.resolved_outcome lets errors override a set outcome (never an
escalation); checkpoint channel versions carry LangGraph's random suffix so
two branches of one thread cannot collide on a blob key; build_graph rejects
an unusable checkpointer instead of silently compiling without persistence;
apply_candidates pairs plans to twins by candidate_index rather than list
position; and Deps carries a clock threaded through every timestamping node,
making run_demo(clock=FrozenClock(t)) actually deterministic for the eval
harness.

Full suite green at 100% branch coverage, ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XRtrUZZvN7JazpLQKD5SrE
@Harikeshav-R

Copy link
Copy Markdown
Owner Author

Code review of the Phase 1B control loop — findings and fixes

Reviewed the full main...HEAD diff (state, graph, nodes, checkpointer, CLI). Eleven findings; the four that mattered all shared a theme: a terminal path that neither resolves production, nor pages a human, nor records a FAILED run. All are fixed in 4ceee10.

High / medium — fixed

1. A failed production remediation paged nobody. apply_to_production() -> False was turned into outcome=FAILED + prod_outcome="not_resolved" and returned normally, so route_linear carried the run to notify_slack → teardown_fleet → record_run → END. Production still broken, on-call never woken; the only route to escalate_pagerduty was via the tournament/kernel routers, which had already passed.
→ actuate now leaves the outcome open and sets an explicit reason; new route_actuation router diverts to escalate_pagerduty. Record is ESCALATED with prod_applied_plan_id / prod_outcome="not_resolved" preserved, so eval's did_escalate counts it.

2. record_run had no error edge. add_edge("record_run", END) was the only unconditional edge in the graph. A run-store outage got swallowed into state.errors, the graph ended, and run_incident returned an outcome=EXECUTED RunRecord that was never persisted — while run_demo on the identical failure raised "not found in store". Two entry points, two different stories about the same event.
→ Edge is now conditional on errors like every other; handle_failure pages and persists.

3. handle_failure was not a safe failure path. Three separate problems:

  • It called fleet_controller.teardown_all but never mirror_registry.unregister_twin. Every failure after register_mirrors destroyed the twins while leaving them registered — production traffic keeps being duplicated into dead twin namespaces.
  • It paged unconditionally. On the escalation path (escalate_pagerduty → teardown_fleet → record_run), a teardown flake — itself the most likely thing to fail, since it is the cleanup node — routed back here and paged on-call a second time for the same incident, rewriting ESCALATED → FAILED and discarding the real escalation reason.
  • Any exception inside it skipped the run-record write entirely.
    → Rewritten: cleanup is best-effort per step and always reaches the record write; pages only when no escalation was sent; preserves ESCALATED and appends cleanup errors to escalation_reason instead of overwriting.

4. Resumed incidents actuated on evidence from twins that no longer existed. observe was the only node with a short-circuit (if state.evidence: return {}). The checkpointer is keyed by incident_id (B1.4), so a re-invocation inherits the prior attempt's evidence/tournament — but fork_fleet/apply_candidates re-run against a brand-new fleet. The kernel then verified, and actuate applied to production, a winner scored on a fleet that had been torn down.
→ Short-circuit removed; observation always re-runs against the current fleet.

5. State.to_run_record dropped errors. outcome=self.outcome or (FAILED if errors else EXECUTED) — the errors branch was dead whenever an outcome was already set, so a run that actuated and then broke was recorded EXECUTED with the errors invisible (RunRecord has no errors field).
→ New State.resolved_outcome(): errors override a set outcome, except ESCALATED, which wins (a human already owns the incident).

Low — fixed

  • Checkpoint version collisions. get_next_version returned a bare f"{next_v:032}" where LangGraph's own savers append a random suffix. Blobs are keyed (thread_id, ns, channel, version), so two branches of one thread produced the same key and the second aput silently overwrote the first branch's channel values — time-travel or update_state would read the wrong branch's state back, with no error. Now suffixed.
  • build_graph silently dropped an unrecognised checkpointer, handing the caller a working but unpersisted graph. Now raises TypeError.
  • apply_candidates paired plans to twins by list position despite both carrying candidate_index, while reduce_twins is explicitly written to append entries it can't match by twin_id. A re-forked twin would shift every plan onto another candidate's twin, and the evidence would attribute the results to the wrong plan. Now pairs by candidate_index (newer fork wins, missing index raises, superseded twins logged).
  • run_demo(clock=...) was not actually deterministic. Five nodes called datetime.now(UTC) directly because Deps carried no clock, so only alert.fired_at was frozen — started_at, finished_at, context.gathered_at were wall time. That seam exists for the eval harness, so: Deps.clock threaded through ingest, gather_context, safety_kernel, record_run, handle_failure, and to_run_record. Asserted in test_run_demo_custom_clock.

Known limitation, not changed

If the loop fails before gather_context completes, handle_failure cannot write a record at all — RunRecord.context is non-optional, so there is nothing truthful to persist. Both entry points now raise the same OrchestratorError carrying the loop's errors, instead of disagreeing about the failure; relaxing the contract is a separate call.

Verification

pytest 322 passed at 100.00% branch coverage (gate is fail_under = 100), ruff check / ruff format --check / mypy --strict clean. ust demo --fake → outcome=executed; --force-veto → outcome=escalated reason=K3 …; both unchanged at 13 transitions. docs/generated/graph.png regenerated — actuate now has three out-edges, record_run two.

One judgement call worth flagging: the escalation branch skips notify_slack, so a failed production remediation pages but does not post to Slack. That matches the §2.2 diagram, where Slack sits only on the resolved branch — easy to change if you'd rather it posted both.

🤖 Generated with Claude Code

@Harikeshav-R
Harikeshav-R merged commit 007a1ac into main Sep 12, 2026
2 checks passed
@Harikeshav-R
Harikeshav-R deleted the feat/phase-1b-orchestrator branch September 12, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant