Repository navigation
feat(actuator): implement phase 5 actuation and end-to-end verification - #30
Merged
Merged
Conversation
Automated remediation requires a safe, deterministic mechanism to translate high-level RemediationPlan models into live Kubernetes mutations across both twin and production namespaces. Without per-action handlers and strict idempotency guarantees, repeated interventions or automated retries risk cascading failures, replica drift, or race conditions. Implements K8sPlanApplier in understudy.actuator.apply with dedicated, idempotent handlers for all six ActionTypes (rollback_deploy, restart_workload, scale_workload, disable_flag, revert_config, no_action). Supports namespace and ServiceAccount parameterization with Kubernetes API impersonation (understudy-prod for production, understudy-twin for twins). Adds comprehensive unit tests achieving 100% statement and branch coverage without cluster or network dependencies. Implements build-plan step 5.1.
Automated remediation on production infrastructure requires strict, non-bypassable safety gates and comprehensive auditability prior to execution. Mutating production without an unexpired formal PASS verdict, or without an immutable pre-actuation record in the run store, risks unconstrained remediation failures, split-brain states, and loss of incident accountability. Implements ProductionActuator in understudy.actuator.production with rigorous enforcement of Invariant K10 (actuation authorisation requiring a PASS verdict matching plan_id and evaluated within the 60s freshness window). Persists an immutable pre-actuation RunRecord with status 'actuating' to RunStore before any live mutation occurs. Performs post-apply SLO probe verification against ust-prod to determine outcome (resolved, not_resolved, worsened) and automatically triggers rollback when degradation or failure occurs. Enforces the UNDERSTUDY_ACTUATION_ENABLED kill switch and AGENTS.md §2.4 caller authorisation restrictions. Adds permanent invariant test test_actuator_requires_pass_verdict and unit tests achieving 100% statement and branch coverage. Implements build-plan step 5.2.
Implement the 7-block Slack Block Kit incident reasoning message per build-plan step 5.3. Generates the incident summary header, monospaced ASCII candidate scoreboard table, decision analysis with winning margin and blast containment, safety kernel verdict details, production actuation outcome and inverse plan, mirror traffic fidelity summary, and link to the append-only run record. Provides the SlackNotifier client with async HTTP delivery and credentials resolution from SecretSettings, updates the Notifier protocol and FakeNotifier, wires state context through orchestrator notify_slack node, and adds the ust notify slack CLI subcommand with terminal preview and JSON modes. Implements build-plan step 5.3.
When an incident cannot be safely or reliably resolved automatically—such as when the Safety Kernel vetoes a candidate due to an invariant violation, when tournament evidence is inconclusive, or when an unhandled failure occurs—on-call human operators require actionable, structured diagnostic context directly on the active PagerDuty incident rather than opaque alerts. Implements PagerDutyNotifier and CompositeNotifier in understudy.notify to post formatted escalation notes with comparative evidence to the PagerDuty REST API (api.pagerduty.com) and raise incident urgency to 'high'. Generates a 6-section comparative note detailing incident summary, Safety Kernel verification verdict and formal counterexamples (e.g., migration boundaries, data corruption risks), the candidate scoreboard table, tournament arbitration analysis explaining why candidate scores were superseded by safety constraints, diagnostic telemetry, and links to the immutable run record. Updates Notifier protocol and FakeNotifier with extended comparative parameters, wires state context into orchestrator escalate_pagerduty and handle_failure nodes, and adds the 'ust notify pagerduty' CLI subcommand with preview, JSON, and live delivery modes. Implements build-plan step 5.4.
Assemble the production Deps container from real component implementations across stores, signals, graph, playbook, planner, fleet, mirror, tournament, safety kernel, actuator, and composite notifiers. Keep deterministic in-memory fakes under create_deps(fake=True) and preserve all existing CLI --fake workflows. Maintain strict import isolation so orchestrator remains decoupled from internal database and cluster SDKs. Implements build-plan step 5.5.
Enforce architecture §2.9 execution limits with per-node timeouts and an async whole-incident watchdog. Nodes exceeding their budget raise NodeTimeoutError and route to failure handling. If the incident budget elapses, the watchdog cancels in-flight work, performs emergency twin teardown and PagerDuty escalation, and marks the incident ESCALATED. Implements build-plan step 5.6.
Add end-to-end incident scenario execution support via ust run --scenario, connecting live context ingestion, LLM candidate generation, twin fleet mirroring, tournament evaluation, safety kernel verification, and production actuation. Handle pre-actuation store record lifecycle and scenario deploy history overlays to support migration boundary verification at Checkpoint 5. Implements build-plan step 5.7.
Remediate findings identified during the Phase 5 code review pass: - Enforce strict non-null evaluated_at timestamp check for Invariant K10 authorisation in production actuator and document in architecture schema. - Persist post-actuation RunRecord to RunStore per ADR-028 and align default target workload with edge-gateway. - Deduplicate twin namespace teardown logic into orchestrator/cleanup.py and eliminate broad exception silencing across actuator and orchestrator. - Extract shared candidate tables and decision analysis formatters into notify/formatting.py and fix mirror fidelity percentage computation. - Supervise live runner with whole-incident watchdog task and isolate alert ingestion through Deps. - Register ScenarioDeployHistory in docs/MOCKS.md for mock parity. Implements build-plan step 5.2 and remediates code review findings.
Owner
Author
Code Review Remediation SummaryThis branch has been updated with commit 1. Safety & Architecture (Invariant K10 & Actuation Persistence)
2. Exception Hygiene & Teardown Deduplication
3. Notification Formatting & Fidelity Calculation
4. Runner Watchdog Supervision & Alert Ingestion
5. Mock Registry Parity
Quality Gate & VerificationAll automated quality gates pass:
|
Introduce typed ScenarioError for incident scenario loading and parsing failures per AGENTS.md §5.5. Add slack_timeout_seconds, pagerduty_timeout_seconds, and k10_max_age_seconds to TimeoutSettings to remove magic numbers per AGENTS.md §5.10. Implements build-plan step 5.6.
…ation persistence Parameterize service_account in Actuator.apply and IncidentContext in apply_to_production per build-plan step 5.1 and ADR-028. Remove redundant post-actuation run record persistence from ProductionActuator to maintain single authoritative incident records in append-only storage. Implements build-plan step 5.2.
…iene Expose workload_reader on Deps to eliminate private concrete attribute reflection in safety_kernel.py per AGENTS.md §5.3. Extract prepare_plan_for_verification helper to cleanly adapt plan resources for verification. Document emergency teardown and watchdog resilience exception handling per AGENTS.md §5.4. Implements build-plan step 5.5.
Align ServiceDependencyGraph fallback DAG edges with Architecture §2.5 (worker -> data-service). Pass workload_reader explicitly to Deps container. Implements build-plan step 5.5.
…ture Remove cross-namespace ust-prod metric peeking during twin baseline capture to strictly adhere to AGENTS.md §2.4 twin isolation guarantees. Implements build-plan step 5.5.
Read client timeouts from TimeoutSettings in SlackNotifier and PagerDutyNotifier instead of hardcoding 10.0 seconds per AGENTS.md §5.10. Implements build-plan steps 5.3 and 5.4.
…ilures Raise typed ScenarioError on scenario YAML parsing or validation failures per AGENTS.md §5.5. Read timeout settings from configured active_deps.timeouts. Implements build-plan step 5.7.
… in verify Implement ust store get <run_id> with text and --json formatting per Spec 5.3 run record link requirement. Filter intermediate pre-actuation audit records in ust store verify to inspect authoritative incident runs per Checkpoint 5. Extract _sample_action_for_candidate helper to eliminate code smell in notification previews. Implements build-plan step 5.3 and Checkpoint 5.
…tests Add integration test suite asserting real Deps assembly and verifying live cluster and PostgreSQL hooks for plan application and store verification per Spec 5.7. Implements build-plan step 5.7.
Owner
Author
Code Review Remediation Follow-up: Standards & Spec ComplianceThis PR branch has been updated with commits 1. Standards Axis Remediation (9 Findings)
2. Spec Axis Remediation (8 Findings)
3. Quality Gate Results
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build-Plan Step Implemented
Checkpoint Execution & Output
ADRs Relied On
Mock Registry Disclosures
Documentation Updates
Definition of Done
make checkgreen (ruff, mypy --strict, import-linter, pytest)# MOCKED:marker without a row in docs/MOCKS.md