arch: state backend silent error swallowing — add retry, circuit-breaker, startup verification #864

Activity

1. diberry

   added

   enhancementNew feature or improvementNew feature or improvement

   on Apr 5, 2026
2. diberry commented on Apr 6, 2026

   

   diberry

   on Apr 6, 2026

   ContributorAuthor

   More actions

   ✈️ Flight — Issue-Level PRD

   Summary

   The git-notes and orphan-branch state backends silently swallow git errors, leaving operators blind to data corruption until users report symptoms. This issue adds retry/backoff for transient failures, a circuit-breaker to prevent cascading failures, startup health verification, and observable error surfacing across all state backend operations.

   Problem Statement

   • State read/write errors are caught and silently discarded — no logging, no user notification
   • Transient git failures (network blips, lock contention) are not retried
   • One corrupted state op cascades through all subsequent operations in the session
   • Operators have zero visibility into state backend health until catastrophic failure

   Proposed Approach

   Phase 1 — Observable Errors (P0, smallest useful change)

   • Replace all silent catch {} blocks in git-notes and orphan-branch backends with logged, user-visible warnings
   • Add structured error context: operation, repo, git command, stderr output
   • No retry, no circuit-breaker — just make failures visible

   Phase 2 — Retry with Backoff (P1)

   • Add exponential backoff retry (3 attempts, 1s/2s/4s) for transient git errors
   • Classify errors: retryable (network, lock) vs permanent (auth, corruption)
   • After max retries, surface the error (Phase 1 path)

   Phase 3 — Circuit-Breaker + Startup Verification (P2)

   • Circuit-breaker: after N consecutive failures (default 5), fast-fail subsequent ops for a cooldown period
   • Startup verification: on CLI startup, run a read/write roundtrip test against the state backend
   • Surface circuit-breaker state in squad doctor output

   Acceptance Criteria

   • Zero silent catch {} blocks remain in state backend code
   • All state backend errors are logged with operation context
   • Transient git errors are retried with exponential backoff (configurable)
   • Circuit-breaker trips after N consecutive failures and fast-fails subsequent ops
   • squad doctor reports state backend health status
   • CLI startup performs state backend read/write roundtrip verification
   • Existing tests updated to verify error surfacing (not silent swallowing)

   Architecture Notes

   • State backends: packages/squad-sdk/src/state/ (git-notes, orphan-branch)
   • Error classification should be a shared utility — other backends may need it
   • Circuit-breaker state should be in-memory per session, not persisted
3. diberry

   added 2 commits that reference this issue on Apr 7, 2026

   fix(sdk): state backend hardening — retry, circuit-breaker, startup v…

   ...

   This issue will close once commit 61aec08 is merged into the 'dev' branch.

   61aec08

   fix(sdk): state backend hardening — retry, circuit-breaker, startup v…

   ...

   This issue will close once commit 17243f4 is merged into the 'dev' branch.

   17243f4
4. diberry commented on Apr 10, 2026

   

   diberry

   on Apr 10, 2026

   ContributorAuthor

   More actions

   🔒 On hold — high-touch architecture work, focusing on low/medium-touch items first.
5. diberry

   closed this as completedon Apr 12, 2026