diff --git a/docs/src/content/docs/reference/state-backend-selection.md b/docs/src/content/docs/reference/state-backend-selection.md index 1f7e0c41e..3584a02fa 100644 --- a/docs/src/content/docs/reference/state-backend-selection.md +++ b/docs/src/content/docs/reference/state-backend-selection.md @@ -120,20 +120,11 @@ git fetch origin squad-state:squad-state ## Known bug: FSStorageProvider `rootDir` (#1555) -> ⚠️ **Open bug [#1555](https://github.com/bradygaster/squad/issues/1555):** `FSStorageProvider` resolves state paths relative to the Node.js process working directory rather than the volume mount path when `rootDir` is not set in `config.json`. This causes state writes to go to the wrong path when the process `cwd` does not match `/app/.squad/`. +> ⚠️ **Open bug [#1555](https://github.com/bradygaster/squad/issues/1555):** `FSStorageProvider` is constructed without a `rootDir` argument in `resolveSquadState()`. All relative state keys resolve against the Node.js process working directory (the repo root) rather than `.squad/`, silently creating a shadow state tree that diverges from the authoritative `.squad/` copy agents actually read. > -> **Workaround:** Explicitly set `rootDir` in `.squad/config.json`: +> **There is no configuration-level workaround.** Because `FSStorageProvider` is constructed with zero arguments regardless of any user configuration, setting `rootDir` in `config.json` has no effect — the constructor call ignores it. This must be fixed in the SDK itself ([#1555](https://github.com/bradygaster/squad/issues/1555)). > -> ```json -> { -> "version": 1, -> "teamRoot": ".", -> "stateBackend": "local", -> "rootDir": "/app/.squad" -> } -> ``` -> -> This ensures Squad resolves state paths against the correct directory regardless of the process working directory. Track [#1555](https://github.com/bradygaster/squad/issues/1555) for a fix. +> **Until #1555 is resolved:** Run only a single Squad replica. Avoid deployment paths where the MCP server process `cwd` differs from the volume mount path. Track [#1555](https://github.com/bradygaster/squad/issues/1555) for the fix. --- diff --git a/docs/src/content/docs/scenarios/production-observability.md b/docs/src/content/docs/scenarios/production-observability.md index bf1a2541b..00e2f3d98 100644 --- a/docs/src/content/docs/scenarios/production-observability.md +++ b/docs/src/content/docs/scenarios/production-observability.md @@ -8,7 +8,7 @@ order: 26 > ⚠️ **Experimental** — Squad is alpha software. APIs, commands, and behavior may change between releases. -> 🔗 **Related:** [Aspire Dashboard](./aspire-dashboard) (local dev telemetry) · [Container Image — Env Var Contract](/squad/docs/reference/container-image/) · [#1144](https://github.com/bradygaster/squad/issues/1144) (embedded-host telemetry — tracked separately) +> 🔗 **Related:** [Aspire Dashboard](./aspire-dashboard) (local dev telemetry) · [Production Troubleshooting Runbook](./production-troubleshooting) · [Container Image — Env Var Contract](/squad/docs/reference/container-image/) · [#1144](https://github.com/bradygaster/squad/issues/1144) (embedded-host telemetry — tracked separately) --- @@ -205,8 +205,9 @@ Recommended alerts once traces are flowing to Azure Monitor or Grafana: | Signal | Recommended threshold | Alert type | |---|---|---| -| `squad.agent.duration` p95 > 5 minutes | Warning | Metric alert | -| `squad.issue.dispatch.errors` rate > 5/min | Critical | Log alert | +| `squad.agent.duration` p95 > 5 minutes | Warning | Metric alert (histogram) | +| `squad.agent.errors` rate > 5/min | Critical | Metric alert (counter) | +| `squad.sessions.errors` rate > 5/min | Warning | Metric alert (counter) | | Pod OOM restarts > 2 in 10 min | Critical | Container Insights alert | | GitHub API 401 errors in structured logs | Critical | Log alert | | No spans received from service in 10 min | Warning | Availability alert | @@ -217,7 +218,7 @@ Recommended alerts once traces are flowing to Azure Monitor or Grafana: When Kubernetes sends `SIGTERM`, Squad begins graceful shutdown. The default pod `terminationGracePeriodSeconds` is 30 seconds. If an agent is mid-task, in-flight work may be interrupted. Squad does not currently checkpoint in-progress tasks before shutdown. -Mitigation: Set `terminationGracePeriodSeconds: 120` in your Deployment spec to give long-running agents time to finish. There is no guarantee all work completes — monitor `squad.agent.interrupted` spans in your telemetry backend to measure the impact. +Mitigation: Set `terminationGracePeriodSeconds: 120` in your Deployment spec to give long-running agents time to finish. There is no guarantee all work completes — monitor `squad.coordinator.shutdown` spans and the `squad.agent.errors` counter in your telemetry backend to assess shutdown impact. Squad does not emit an interrupted-specific span; shutdown scope is visible only through these existing signals. --- diff --git a/docs/src/content/docs/scenarios/production-troubleshooting.md b/docs/src/content/docs/scenarios/production-troubleshooting.md index 3d5953524..1aeaf3dc7 100644 --- a/docs/src/content/docs/scenarios/production-troubleshooting.md +++ b/docs/src/content/docs/scenarios/production-troubleshooting.md @@ -47,7 +47,7 @@ Is the state backend accessible? Root causes in order of frequency: 1. **KEDA not scaling** — queue depth below threshold or scaler unreachable 2. **Wrong or missing env vars** — `GITHUB_TOKEN` absent or wrong scope -3. **State backend inaccessible** — PVC mount failure or `rootDir` mismatch (#1555) +3. **State backend inaccessible** — PVC mount failure or `FSStorageProvider` path bug (#1555) 4. **GitHub API 401** — expired token, missing scope, or org SSO not enabled 5. **No qualifying labels** — issues exist but labels don't match routing rules 6. **Image pull failure** — ACR credentials expired or image tag not found @@ -154,21 +154,13 @@ If the PVC mount is missing, the pod is writing state to the container's ephemer ### FSStorageProvider rootDir bug (#1555) -If `kubectl exec -n squad -- ls /app/.squad` shows the correct files but Squad cannot find them, the process `cwd` does not match the volume mount path. +If Squad logs show state files at the repo root (e.g., `agents/`, `log/`, `decisions.md` appearing at `/app/` rather than `/app/.squad/`), this is [#1555](https://github.com/bradygaster/squad/issues/1555): `FSStorageProvider` is constructed without a `rootDir` argument, so all state keys resolve against the process `cwd` (the repo root) instead of `.squad/`. -**Workaround:** - -```json -// /app/.squad/config.json -{ - "version": 1, - "teamRoot": ".", - "stateBackend": "local", - "rootDir": "/app/.squad" -} -``` +> ⚠️ **There is no configuration-level workaround for #1555.** Setting `rootDir` in `config.json` has no effect — `FSStorageProvider` ignores constructor arguments in the affected code path. This must be fixed in the SDK itself. +> +> **Mitigation until #1555 is resolved:** Run only a single Squad replica. Avoid deployment shapes where the MCP server process `cwd` differs from the intended state path. Track [#1555](https://github.com/bradygaster/squad/issues/1555) for status. -See [State Backend Selection — Known bug #1555](/squad/docs/reference/state-backend-selection/#known-bug-fsstorageprovider-rootdir-1555) for details. +See [State Backend Selection — Known bug #1555](/squad/docs/reference/state-backend-selection/#known-bug-fsstorageprovider-rootdir-1555) for background. ### `squad-state` branch push rejected (two-layer / orphan) @@ -319,5 +311,5 @@ Open an issue at [bradygaster/squad](https://github.com/bradygaster/squad/issues |---|---| | [#1144](https://github.com/bradygaster/squad/issues/1144) | Embedded host (copilot-sdk) telemetry not available in production — agents must use Squad CLI container mode | | [#1402](https://github.com/bradygaster/squad/issues/1402) | External state backend for multi-pod deployments — not yet GA | -| [#1555](https://github.com/bradygaster/squad/issues/1555) | FSStorageProvider `rootDir` bug — workaround documented above | +| [#1555](https://github.com/bradygaster/squad/issues/1555) | FSStorageProvider `rootDir` bug — no configuration-level workaround; run single replica and track issue for fix | | [#1577](https://github.com/bradygaster/squad/issues/1577) | HTTP health/readiness endpoints — not yet implemented |