docs: production operations — observability to Azure Monitor, state backend selection, and troubleshooting runbook #1568

Problem

Three operational gaps exist where current docs tell only the local-dev half of the story:

1. Observability: scenarios/aspire-dashboard.md stops at local dev. Production deployments need OTel → Azure Monitor / Application Insights, Prometheus/Grafana, or structured log shipping. No guidance exists.
2. State backend selection: features/state-backends.md is thorough for developer machines but gives no guidance for containerized deployments (PVCs, multi-pod write conflicts, external state for horizontal scale).
3. Troubleshooting: scenarios/troubleshooting.md covers CLI install errors. No Kubernetes/container ops runbook exists.

Parent issue: #1564

Scope

Three new or extended pages:

Page A — Production observability: docs/src/content/docs/scenarios/production-observability.md

• OTel → Azure Monitor: OTEL_EXPORTER_OTLP_ENDPOINT + App Insights connection string example
• OpenTelemetry Collector sidecar config routing spans to Aspire (dev) and Azure Monitor (prod) simultaneously
• Prometheus scrape config + starter Grafana panel query (squad.agent.duration)
• Structured stdout JSON → Log Analytics
• "No traces appear" troubleshooting checklist (firewall, OTLP protocol, auth headers)
• Cross-link to scenarios/aspire-dashboard.md and Squad telemetry from embedded host processes (Agent Framework wrappers, no REPL): how should it reach Aspire? #1144 (embedded host telemetry — track together)

Page B — State backend selection: Extension of features/state-backends.md or new reference page

• Decision matrix: deployment shape (single dev, single pod, multi-pod) → recommended backend
• docker run -v and Kubernetes PVC examples for local-state mode
• Explicit warning: branch-based (orphan) state is NOT safe for concurrent writes from multiple pods; data corruption risk
• External state (correct for multi-pod) — cross-link to [External State] Tracking: command and docs integration gaps #1402 for remaining gaps
• Note open bug FSStorageProvider constructed without rootDir in resolveSquadState() — state writes land in repo root instead of .squad/ #1555 (FSStorageProvider rootDir) and current workaround (explicit rootDir config)

Page C — Production troubleshooting runbook: docs/src/content/docs/scenarios/production-troubleshooting.md

• Decision tree: "agent not picking up issues" — ≥5 root causes: KEDA not scaling, wrong env vars, state backend inaccessible, GitHub API 401, no qualifying labels
• kubectl logs, kubectl describe, kubectl get events command examples per failure mode
• KEDA scaler not triggering — debug steps (check ScaledObject status, external scaler service health)
• State backend contention in containers
• GitHub API rate limiting across pod fleet — budgeting guidance
• Graceful shutdown and SIGTERM handling

Acceptance Criteria

• Page A: Working OTLP → App Insights example; OTel Collector config snippet; one Grafana PromQL query
• Page A: Cross-links to aspire-dashboard.md and Squad telemetry from embedded host processes (Agent Framework wrappers, no REPL): how should it reach Aspire? #1144
• Page B: Decision matrix covers single-dev, single-pod, multi-pod deployment shapes
• Page B: Explicit multi-pod write-conflict warning for branch-based state
• Page B: Bug FSStorageProvider constructed without rootDir in resolveSquadState() — state writes land in repo root instead of .squad/ #1555 documented with workaround
• Page C: "Agent not picking up issues" decision tree covers ≥5 root causes
• Page C: kubectl command examples for ≥4 failure modes

Non-Goals

• Replacing Azure Monitor, App Insights, or Grafana official docs
• Providing SLA targets or capacity planning formulas (no verified throughput data available)
• Covering ACA-specific observability (that belongs in docs: hosting Squad in Azure — Dockerfile, ACA, and AKS deployment guides #1566)

Source / Inspiration

Tamir Dresher published posts on Squad observability and ops at scale. These are inspiration and citation sources — content must be written in Squad voice, not copied.

Relation to Existing Issues

• Squad telemetry from embedded host processes (Agent Framework wrappers, no REPL): how should it reach Aspire? #1144 — Page A cross-links for embedded host telemetry; both should move together.
• [External State] Tracking: command and docs integration gaps #1402 — Page B cross-links for external state backend gaps.
• FSStorageProvider constructed without rootDir in resolveSquadState() — state writes land in repo root instead of .squad/ #1555 — Page B documents open bug and workaround.
• docs milestone: real-world Squad hosting, integrations, and operations #1564 — Parent umbrella.

Priority

P1 — High operator value; relatively self-contained authoring scope.