From 555250e597a33ccdb8ccf81fd089a45efc0ec2f3 Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Sun, 2 Aug 2026 20:45:26 -0700 Subject: [PATCH] docs(gnc-revision): remove fabricated metrics/spans, fix #1555 workaround, add cross-link MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GNC revision of PR #1579 (original author Telemetry, locked out): 1. Remove fabricated squad.issue.dispatch.errors from alerting table — replaced with squad.agent.errors (counter, defined in source) and squad.sessions.errors (counter, defined in source). 2. Remove fabricated squad.agent.interrupted span from graceful-shutdown section — replaced with squad.coordinator.shutdown (span, defined in source) and squad.agent.errors (counter). Note added that Squad does not emit an interrupted-specific span. 3. Remove invalid rootDir config workaround from state-backend-selection.md and production-troubleshooting.md. Per issue #1555 source analysis: FSStorageProvider is constructed with zero arguments regardless of user config — there is no configuration-level workaround. Docs now state this accurately and advise single-replica / avoid affected path. 4. Add production-observability → production-troubleshooting cross-link in the Related header row. Source evidence: - squad.agent.errors: packages/squad-sdk/src (meter.createCounter) - squad.sessions.errors: packages/squad-sdk/src (meter.createCounter) - squad.coordinator.shutdown: packages/squad-sdk/src (tracer.startSpan) - squad.agent.duration: packages/squad-sdk/src (meter.createHistogram) - squad.issue.dispatch.errors: NOT FOUND in any source file - squad.agent.interrupted: NOT FOUND in any source file - #1555 body: "there is no configuration-level workaround" Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../docs/reference/state-backend-selection.md | 15 +++---------- .../scenarios/production-observability.md | 9 ++++---- .../scenarios/production-troubleshooting.md | 22 ++++++------------- 3 files changed, 15 insertions(+), 31 deletions(-) diff --git a/docs/src/content/docs/reference/state-backend-selection.md b/docs/src/content/docs/reference/state-backend-selection.md index 1f7e0c41e..3584a02fa 100644 --- a/docs/src/content/docs/reference/state-backend-selection.md +++ b/docs/src/content/docs/reference/state-backend-selection.md @@ -120,20 +120,11 @@ git fetch origin squad-state:squad-state ## Known bug: FSStorageProvider `rootDir` (#1555) -> ⚠️ **Open bug [#1555](https://github.com/bradygaster/squad/issues/1555):** `FSStorageProvider` resolves state paths relative to the Node.js process working directory rather than the volume mount path when `rootDir` is not set in `config.json`. This causes state writes to go to the wrong path when the process `cwd` does not match `/app/.squad/`. +> ⚠️ **Open bug [#1555](https://github.com/bradygaster/squad/issues/1555):** `FSStorageProvider` is constructed without a `rootDir` argument in `resolveSquadState()`. All relative state keys resolve against the Node.js process working directory (the repo root) rather than `.squad/`, silently creating a shadow state tree that diverges from the authoritative `.squad/` copy agents actually read. > -> **Workaround:** Explicitly set `rootDir` in `.squad/config.json`: +> **There is no configuration-level workaround.** Because `FSStorageProvider` is constructed with zero arguments regardless of any user configuration, setting `rootDir` in `config.json` has no effect — the constructor call ignores it. This must be fixed in the SDK itself ([#1555](https://github.com/bradygaster/squad/issues/1555)). > -> ```json -> { -> "version": 1, -> "teamRoot": ".", -> "stateBackend": "local", -> "rootDir": "/app/.squad" -> } -> ``` -> -> This ensures Squad resolves state paths against the correct directory regardless of the process working directory. Track [#1555](https://github.com/bradygaster/squad/issues/1555) for a fix. +> **Until #1555 is resolved:** Run only a single Squad replica. Avoid deployment paths where the MCP server process `cwd` differs from the volume mount path. Track [#1555](https://github.com/bradygaster/squad/issues/1555) for the fix. --- diff --git a/docs/src/content/docs/scenarios/production-observability.md b/docs/src/content/docs/scenarios/production-observability.md index bf1a2541b..00e2f3d98 100644 --- a/docs/src/content/docs/scenarios/production-observability.md +++ b/docs/src/content/docs/scenarios/production-observability.md @@ -8,7 +8,7 @@ order: 26 > ⚠️ **Experimental** — Squad is alpha software. APIs, commands, and behavior may change between releases. -> 🔗 **Related:** [Aspire Dashboard](./aspire-dashboard) (local dev telemetry) · [Container Image — Env Var Contract](/squad/docs/reference/container-image/) · [#1144](https://github.com/bradygaster/squad/issues/1144) (embedded-host telemetry — tracked separately) +> 🔗 **Related:** [Aspire Dashboard](./aspire-dashboard) (local dev telemetry) · [Production Troubleshooting Runbook](./production-troubleshooting) · [Container Image — Env Var Contract](/squad/docs/reference/container-image/) · [#1144](https://github.com/bradygaster/squad/issues/1144) (embedded-host telemetry — tracked separately) --- @@ -205,8 +205,9 @@ Recommended alerts once traces are flowing to Azure Monitor or Grafana: | Signal | Recommended threshold | Alert type | |---|---|---| -| `squad.agent.duration` p95 > 5 minutes | Warning | Metric alert | -| `squad.issue.dispatch.errors` rate > 5/min | Critical | Log alert | +| `squad.agent.duration` p95 > 5 minutes | Warning | Metric alert (histogram) | +| `squad.agent.errors` rate > 5/min | Critical | Metric alert (counter) | +| `squad.sessions.errors` rate > 5/min | Warning | Metric alert (counter) | | Pod OOM restarts > 2 in 10 min | Critical | Container Insights alert | | GitHub API 401 errors in structured logs | Critical | Log alert | | No spans received from service in 10 min | Warning | Availability alert | @@ -217,7 +218,7 @@ Recommended alerts once traces are flowing to Azure Monitor or Grafana: When Kubernetes sends `SIGTERM`, Squad begins graceful shutdown. The default pod `terminationGracePeriodSeconds` is 30 seconds. If an agent is mid-task, in-flight work may be interrupted. Squad does not currently checkpoint in-progress tasks before shutdown. -Mitigation: Set `terminationGracePeriodSeconds: 120` in your Deployment spec to give long-running agents time to finish. There is no guarantee all work completes — monitor `squad.agent.interrupted` spans in your telemetry backend to measure the impact. +Mitigation: Set `terminationGracePeriodSeconds: 120` in your Deployment spec to give long-running agents time to finish. There is no guarantee all work completes — monitor `squad.coordinator.shutdown` spans and the `squad.agent.errors` counter in your telemetry backend to assess shutdown impact. Squad does not emit an interrupted-specific span; shutdown scope is visible only through these existing signals. --- diff --git a/docs/src/content/docs/scenarios/production-troubleshooting.md b/docs/src/content/docs/scenarios/production-troubleshooting.md index 3d5953524..1aeaf3dc7 100644 --- a/docs/src/content/docs/scenarios/production-troubleshooting.md +++ b/docs/src/content/docs/scenarios/production-troubleshooting.md @@ -47,7 +47,7 @@ Is the state backend accessible? Root causes in order of frequency: 1. **KEDA not scaling** — queue depth below threshold or scaler unreachable 2. **Wrong or missing env vars** — `GITHUB_TOKEN` absent or wrong scope -3. **State backend inaccessible** — PVC mount failure or `rootDir` mismatch (#1555) +3. **State backend inaccessible** — PVC mount failure or `FSStorageProvider` path bug (#1555) 4. **GitHub API 401** — expired token, missing scope, or org SSO not enabled 5. **No qualifying labels** — issues exist but labels don't match routing rules 6. **Image pull failure** — ACR credentials expired or image tag not found @@ -154,21 +154,13 @@ If the PVC mount is missing, the pod is writing state to the container's ephemer ### FSStorageProvider rootDir bug (#1555) -If `kubectl exec -n squad -- ls /app/.squad` shows the correct files but Squad cannot find them, the process `cwd` does not match the volume mount path. +If Squad logs show state files at the repo root (e.g., `agents/`, `log/`, `decisions.md` appearing at `/app/` rather than `/app/.squad/`), this is [#1555](https://github.com/bradygaster/squad/issues/1555): `FSStorageProvider` is constructed without a `rootDir` argument, so all state keys resolve against the process `cwd` (the repo root) instead of `.squad/`. -**Workaround:** - -```json -// /app/.squad/config.json -{ - "version": 1, - "teamRoot": ".", - "stateBackend": "local", - "rootDir": "/app/.squad" -} -``` +> ⚠️ **There is no configuration-level workaround for #1555.** Setting `rootDir` in `config.json` has no effect — `FSStorageProvider` ignores constructor arguments in the affected code path. This must be fixed in the SDK itself. +> +> **Mitigation until #1555 is resolved:** Run only a single Squad replica. Avoid deployment shapes where the MCP server process `cwd` differs from the intended state path. Track [#1555](https://github.com/bradygaster/squad/issues/1555) for status. -See [State Backend Selection — Known bug #1555](/squad/docs/reference/state-backend-selection/#known-bug-fsstorageprovider-rootdir-1555) for details. +See [State Backend Selection — Known bug #1555](/squad/docs/reference/state-backend-selection/#known-bug-fsstorageprovider-rootdir-1555) for background. ### `squad-state` branch push rejected (two-layer / orphan) @@ -319,5 +311,5 @@ Open an issue at [bradygaster/squad](https://github.com/bradygaster/squad/issues |---|---| | [#1144](https://github.com/bradygaster/squad/issues/1144) | Embedded host (copilot-sdk) telemetry not available in production — agents must use Squad CLI container mode | | [#1402](https://github.com/bradygaster/squad/issues/1402) | External state backend for multi-pod deployments — not yet GA | -| [#1555](https://github.com/bradygaster/squad/issues/1555) | FSStorageProvider `rootDir` bug — workaround documented above | +| [#1555](https://github.com/bradygaster/squad/issues/1555) | FSStorageProvider `rootDir` bug — no configuration-level workaround; run single replica and track issue for fix | | [#1577](https://github.com/bradygaster/squad/issues/1577) | HTTP health/readiness endpoints — not yet implemented |