Skip to content

feat(signals): complete phase 2b state, observability adapters, and dependency graph - #25

Merged
Harikeshav-R merged 8 commits into
mainfrom
feat/phase-2b
Sep 13, 2026
Merged

Harikeshav-R merged 8 commits into
mainfrom
feat/phase-2b

Conversation

@Harikeshav-R

Copy link
Copy Markdown
Owner

Build-Plan Step Implemented

  • Step: Phase 2B — B2.1 through B2.6
    • B2.1: store/ models, migrations, RunStore / PlaybookStore / EvalStore, append-only rules
    • B2.2: PagerDuty webhook receiver, HMAC verification, ust tunnel, synthetic-alert fallback
    • B2.3: Prometheus + Loki ObservabilityAdapter, namespace-parameterised
    • B2.4: GitHub DeployHistory with image digests and migration detection (Invariant K3 boundary)
    • B2.5: Datadog adapter conforming to ObservabilityAdapter protocol
    • B2.6: graph/: DAG from dependencies.yaml, cross-check against observed Prometheus traffic, and BlastRadiusCalculator

Checkpoint Execution & Output

Checkpoint B2:

  1. ust signals deploys --limit 5
Recent 5 deployment(s) for repo Harikeshav-R/Understudy:
--------------------------------------------------------------------------------
19831be | 2026-09-13 14:02:12 UTC | PR: #24 | migration: NO
b5161e7 | 2026-09-13 13:58:45 UTC | PR: #23 | migration: NO
...
  1. ust signals context
IncidentContext(
  incident_id='inc_test_context',
  service='data-service',
  p99_latency_ms=12.4,
  error_rate=0.001,
  log_signatures=['Database timeout in transaction block'],
  dependencies=['auth-service', 'worker'],
  recent_deploys=[...]
)
  1. ust graph show
edge-gateway -> auth-service -> data-service, worker -> data-service
declared graph matches observed traffic: OK
declared matches observed: OK
  1. test_run_store_append_only
tests/unit/store/test_postgres.py::test_run_store_append_only PASSED
  1. PagerDuty live test incident:
    checkpoint: unverified (PagerDuty account requires corporate email domain; synthetic fallback on port 9108 fully verified per docs/06-demo.md §6.7)

make check: 610 passed in 10.19s, 100.00% statement and branch coverage (6,344 statements, 1,604 branches), clean mypy --strict, clean ruff format/lint, clean scripts/check_mocks.py, and 5/5 import-linter contracts kept.

ADRs Relied On

  • ADR-004 — RunStore is strictly append-only; immutable run records, playbook entries, and eval metrics.
  • ADR-005 — Telemetry extraction via Prometheus and Loki with namespace scoping (ust-prod, ust-twin-*).
  • ADR-006 — Protocol-driven boundaries (ObservabilityAdapter, DeployHistory, DependencyGraph, RunStore).
  • ADR-007 — GitHub deployment history and migration flag discovery across commit boundaries for Invariant K3.
  • ADR-013 — Service dependency DAG and traffic cross-check against Prometheus request metrics.

Mock Registry Disclosures

None. No # MOCKED: markers added or modified; scripts/check_mocks.py passed clean.

Documentation Updates

  • CHECKLIST.md — Marked B2.1 through B2.6 and verified Checkpoint B2 items.

Definition of Done

  • Implementation complete for the named build-plan step
  • Unit tests written; 100% line and branch coverage of the added lines
  • make check green (ruff, mypy --strict, import-linter, pytest)
  • Checkpoint command run and output matches, OR marked unverified with a reason
  • Docs updated where this change made them untrue (or "none required", with why)
  • No new # MOCKED: marker without a row in docs/MOCKS.md
  • Conventional commit(s), conventional branch, PR opened, never pushed to main

Add SQLAlchemy 2.0 ORM models and Alembic migrations for runs, playbooks, and scenario_results matching architecture §2.12. Enforce append-only immutability on the runs table using PostgreSQL rules (runs_no_update and runs_no_delete).

Provide PostgresRunStore, PostgresPlaybookStore (with pgvector HNSW cosine similarity search), and PostgresEvalStore conforming to the store protocol interfaces.

Add live integration tests asserting UPDATE and DELETE queries against runs are no-ops, and provide full unit tests maintaining 100% line and branch coverage.

Implements build-plan step B2.1.
…rt fallback

Implement PagerDuty webhook receiver on port 9108 with HMAC-SHA256
signature verification, AlertSource protocol implementation, ust tunnel
command with cloudflared/ngrok/fake support, and ust alert inject
synthetic fallback command.

Implements build-plan step B2.2.
Add PrometheusClient, LokiClient, and PrometheusLokiAdapter implementing
ObservabilityAdapter for real telemetry ingestion across production and
twin namespaces.

Telemetry ingestion is required to capture pre- and post-incident
performance metrics and log errors during tournament evaluation and
incident diagnosis. Log messages are normalized (hiding timestamps, UUIDs,
IPs, and numbers) and fingerprinted via SHA-256 to aggregate recurring
error signatures. Metrics query p99 latency, request count, and error
rates directly from Prometheus with label-agnostic fallbacks.

Also add ust signals context CLI command to retrieve structured
incident context.

Implements build-plan step B2.3.
Implement the DeployHistory protocol for GitHub to query commit history,
map commits to container image digests, and detect schema migrations
across commit boundaries for Invariant K3.

Implements build-plan step B2.4.
Implement Datadog metrics and log telemetry query adapter adhering to
the ObservabilityAdapter protocol, supporting p99 latency, error rates,
and log signature extractions.

Implements build-plan step B2.5.
Construct service dependency DAG from dependencies.yaml, cross-check
against observed Prometheus traffic rates, and calculate candidate blast
radius using graph centrality and request share formulas.

Implements build-plan step B2.6.
Expose ust graph show CLI command for inspecting the dependency
DAG and cross-check verification status against Prometheus traffic, and
connect graph context into ust signals context. Mark completed steps in
CHECKLIST.md.

Completes Checkpoint B2.
@gitguardian

gitguardian Bot commented Sep 13, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37235456 Triggered Authentication Tuple f194966 tests/unit/graph/test_service_graph.py View secret
37235457 Triggered Authentication Tuple f194966 tests/unit/graph/test_service_graph.py View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

A /code-review pass surfaced 15 confirmed bugs across the signals
adapters, dependency graph, blast-radius calculator, CLI, and Postgres
store; separately, the branch's first CI run was failing because a test
hit a real network call instead of mocking the observability adapter.

- pagerduty: fail fast when no webhook secret is configured, fix
  substring-based service-name normalization, detect webhook server
  bind failures/timeouts instead of returning silently
- scenarios: normalize synthetic alert service names like real webhooks
- datadog: fix p99 unit-conversion heuristic (was under-reporting severe
  latency by 1000x), add missing namespace filter to service_health()
- prometheus/api: tighten service-health matching to avoid false-positive
  substring matches, add `until` param consistently across
  ObservabilityAdapter and both adapters
- github: paginate commit history fetches instead of truncating at 100
- graph: cross_check() now catches declared edges/services that are
  never observed in traffic, not just the reverse; simplify
  format_chains() (its old subpath filter was provably dead code once
  fixed to check genuine subsequences)
- calculator: fix zero-baseline degradation thresholds so a fresh error
  spike from a clean baseline is correctly flagged
- cli: log (instead of silently swallowing) fallback failures in
  `signals context`
- store: add claim_run(), an atomic advisory-lock-backed claim primitive
  for the K5 single-writer contract, without wiring it into the
  not-yet-built kernel/shadow-mode
- tests: fix test_cli_signals_context_fallback_graph to mock the
  observability adapter instead of hitting the network

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rnk8jvArFB9hormE9WbAD
@Harikeshav-R

Copy link
Copy Markdown
Owner Author

/code-review findings

A full review pass surfaced 15 confirmed findings across the signals adapters, dependency graph, blast-radius calculator, CLI, and Postgres store. All 15 are fixed in d5fc547, along with an unrelated CI test gap discovered while investigating (test_cli_signals_context_fallback_graph was hitting a real network call instead of mocking the observability adapter, failing this branch's first CI run).

Critical

  1. store/postgres.py — get_active_runs()/record_run() don't share a transaction, so the store's documented atomic check-and-claim contract (kernel invariant K5) wasn't actually true. Fix: added claim_run(), an advisory-lock-backed atomic claim primitive.
  2. signals/pagerduty.py:255 — default config (no secret set) made webhook signature verification reject every real webhook forever, with no indication why. Fix: fails fast with a clear error instead.
  3. signals/pagerduty.py:377 — WebhookReceiverServer.start() reported success even when the server failed to bind. Fix: detects bind failure/timeout and raises.
  4. signals/datadog.py:407 — p99 latency heuristic under-reported severe spikes by 1000x during real incidents. Fix: always converts seconds→ms, matching the Prometheus adapter.
  5. graph/service_graph.py:234 — cross_check() only caught undeclared traffic, never a declared edge that's never actually observed. Fix: now flags both directions.
  6. graph/calculator.py:63 — blast-radius calc undercounted newly-erroring services (absolute-vs-relative threshold mismatch on zero baselines). Fix: fresh error spikes from a clean baseline are now correctly flagged.

Medium

  1. signals/prometheus.py:309 — health check matched services by substring, could pick the wrong pod during an outage. Fix: tightened to hostname-segment/prefix matching.
  2. signals/pagerduty.py:54 — service-name normalization used substring containment, could misroute alerts (e.g. "metadata-service" → "data-service"). Fix: word-boundary matching.
  3. cli.py:659 — bare except Exception silently faked an empty dependency graph on any failure, no logging.
  4. cli.py:648 — bare except Exception silently faked an empty deploy list on any GitHub API failure, no logging. (9 & 10 fix: both now log a warning before falling back.)
  5. signals/datadog.py:487 — service_health()'s primary query ignored the namespace filter. Fix: added.
  6. signals/prometheus.py:339 — PrometheusLokiAdapter was missing the until param DatadogAdapter has, breaking interchangeability (would raise TypeError). Fix: added consistently to the ObservabilityAdapter protocol and both adapters.

Lower severity

  1. graph/service_graph.py:215 — format_chains()'s "maximal path" filter dropped distinct chains that merely shared start/end nodes. Fix: turned out the filter can never actually fire once made correct (every path runs entrypoint→terminal, so no path can be a genuine subsequence of another) — simplified by removing the dead filter entirely.
  2. signals/scenarios.py:172 — synthetic alerts skipped the service-name normalization real webhooks get. Fix: applied.
  3. signals/github.py:234 — _fetch_commits() had no pagination, silently truncating deploy history at 100 commits. Fix: follows the Link header until limit is reached.

All fixes are covered by unit tests; make check (ruff, mypy --strict, mock registry parity, import-linter contracts, pytest at 100% branch coverage) passes clean.

@Harikeshav-R
Harikeshav-R merged commit cf153bd into main Sep 13, 2026
2 checks passed
@Harikeshav-R
Harikeshav-R deleted the feat/phase-2b branch September 13, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant