diff --git a/cspell.json b/cspell.json index 2f1fc0a0c..4164b7a66 100644 --- a/cspell.json +++ b/cspell.json @@ -41,6 +41,7 @@ "benchmarkfn", "filterissuesbystream", "filterissuesbyworkstream", "Recognised", "recognised", "squadified", "TOCTOU", "unflushed", "unparseable", "pluggability", "PKCE", "MSAL", "AKIA", "runas", + "eastus", "eastus2", "kubeconfig", "healthz", "readyz", "typoo" ], "dictionaries": ["en_US", "typescript", "node", "npm", "bash"], diff --git a/docs/src/content/docs/reference/container-image.md b/docs/src/content/docs/reference/container-image.md new file mode 100644 index 000000000..35e0d6fff --- /dev/null +++ b/docs/src/content/docs/reference/container-image.md @@ -0,0 +1,291 @@ +--- +title: Container Image — Environment Variable Contract +description: Canonical reference for Squad container image configuration — environment variables, config/state paths, process lifecycle, graceful shutdown, and secret injection patterns. +order: 10 +--- + +# Container Image — Environment Variable Contract + +> ⚠️ **Experimental** — Squad is alpha software. APIs, commands, and behavior may change between releases. + +This page is the canonical reference for running `@bradygaster/squad-cli` in a container. Every environment variable, volume mount, and behavioral expectation is defined here. [Azure Container Apps](../scenarios/azure-container-apps) and [AKS deployment](../scenarios/aks-deployment) pages cross-link to this reference. + +--- + +## Environment Variables + +### Required — runtime identity + +| Variable | Required | Default | Description | +|---|---|---|---| +| `GITHUB_TOKEN` | **Required** | — | GitHub personal access token **or** (preferred) an identity token issued by workload identity / OIDC. Grants Squad access to the GitHub API for issue polling, PRs, and commits. | +| `SQUAD_DEPLOYMENT_MODE` | Optional | `agent-per-node` | Deployment topology. Set to `squad-per-pod` when multiple Squad containers run concurrently against the same repository. See [Dual-Mode Deployment](/squad/docs/features/dual-mode-deployment/). | +| `SQUAD_POD_ID` | Conditional | — | Required when `SQUAD_DEPLOYMENT_MODE=squad-per-pod`. Must be unique per pod replica. Kubernetes: set via `fieldRef.fieldPath: metadata.name`. | + +### Optional — observability + +| Variable | Required | Default | Description | +|---|---|---|---| +| `OTEL_EXPORTER_OTLP_ENDPOINT` | Optional | — | OTLP endpoint for OpenTelemetry traces and metrics (e.g., `https://otelcollector:4317`). When set, Squad emits structured spans for issue dispatch, agent lifecycle, and tool calls. | +| `OTEL_SERVICE_NAME` | Optional | `squad-agent` | Service name reported to your OTLP collector. | +| `OTEL_RESOURCE_ATTRIBUTES` | Optional | — | Key-value pairs appended to every span (e.g., `deployment.environment=production,cluster=aks-eastus`). | + +### GitHub API scopes required + +`GITHUB_TOKEN` must carry the following scopes depending on the features Squad uses: + +| Scope | Why | +|---|---| +| `repo` | Read/write issues, comments, PRs, and commits | +| `read:org` | Org-level label and project read access | +| `workflow` | Trigger GitHub Actions workflows (if used) | + +> **Important:** Azure managed identity / workload identity authenticates to **Azure** services (Key Vault, ACR, etc.), not to the GitHub API. `GITHUB_TOKEN` must always be a GitHub personal access token or GitHub App token injected at runtime (via Key Vault reference or Kubernetes Secret). Workload identity does not replace it. + +--- + +## Paths and Volumes + +### Default paths inside the container + +| Path | Purpose | +|---|---| +| `/app` | Application root; `squad watch --execute` runs here | +| `/app/.squad/` | Default Squad config and state directory | +| `/app/.squad/team.md` | Agent roster and roles | +| `/app/.squad/routing.md` | Issue routing rules | +| `/app/.squad/config.json` | Squad runtime config (state backend, etc.) | + +### Volume mount strategies + +**Strategy A — bake `.squad/` into the image (simple, immutable)** + +```dockerfile +COPY .squad/ /app/.squad/ +``` + +The team config and routing rules are pinned to the image tag. Updates require a new image push. Suitable when the Squad config rarely changes. + +**Strategy B — mount `.squad/` as a volume (mutable, shared)** + +```yaml +# Kubernetes / ACA example +volumeMounts: + - name: squad-config + mountPath: /app/.squad +``` + +The config is injected at runtime from a ConfigMap or Azure Files share. Supports live config updates without rebuilding the image. Note that FSStorageProvider may resolve state paths relative to the process working directory rather than the volume mount when `rootDir` is not set in `config.json` — see [#1555](https://github.com/bradygaster/squad/issues/1555). + +> ⚠️ **Multi-replica state warning:** The `local` state backend (files on disk) is **not safe for concurrent writes** from multiple pod replicas. If you scale beyond one replica, use the `orphan` or `two-layer` state backend, or persist state to an external store. See [State Backends](/squad/docs/features/state-backends/) and [#1402](https://github.com/bradygaster/squad/issues/1402). + +--- + +## Production Dockerfile Reference + +The following Dockerfile is the canonical starting point for Squad container images. It uses a multi-stage build, pins Node.js, and runs as a non-root user. + +```dockerfile +# syntax=docker/dockerfile:1.7 +# Stage 1 — install production dependencies +FROM node:22-alpine AS deps +WORKDIR /build + +# Copy manifests first for layer caching +COPY package.json package-lock.json ./ +RUN npm ci --omit=dev --ignore-scripts + +# Stage 2 — production image +FROM node:22-alpine AS runtime + +# Non-root user (UID 1001 avoids common conflicts with bind mounts) +RUN addgroup --system squad && adduser --system --ingroup squad --uid 1001 squad + +WORKDIR /app + +# Copy node_modules from deps stage +COPY --from=deps --chown=squad:squad /build/node_modules ./node_modules + +# Copy Squad config (baked-in strategy; use a volume mount for Strategy B) +COPY --chown=squad:squad .squad/ ./.squad/ + +# Install squad-cli globally so 'squad' is on PATH +RUN npm install -g @bradygaster/squad-cli --ignore-scripts + +# Drop privileges +USER squad + +# Squad handles SIGTERM: drains in-flight work, then exits cleanly. +STOPSIGNAL SIGTERM + +# GITHUB_TOKEN must be provided at runtime — never bake it into the image. +# Inject via Kubernetes Secret, ACA Key Vault reference, or CSI driver. +CMD ["squad", "watch", "--execute"] +``` + +> **Base image pinning:** `node:22-alpine` is used above. Pin to a specific digest in production (e.g., `node:22.x.y-alpine@sha256:...`) to prevent supply-chain drift. Rebuild on security advisories. + +--- + +## Process Lifecycle and Health + +Squad does not expose an HTTP health server or liveness/readiness endpoints. There is no `SQUAD_HEALTH_PORT`, `/healthz`, or `/readyz` — these are planned features tracked in [#1577](https://github.com/bradygaster/squad/issues/1577). + +**Current container health behavior:** +- The container is healthy as long as the `squad watch --execute` process is running. +- If `squad` exits for any reason, the container exits. Kubernetes and ACA restart exited containers automatically via the container restart policy — this is sufficient for the current release. +- Do **not** configure HTTP liveness or readiness probes — there is no listener and probes will always fail, crash-looping your pod. + +If your cluster policy requires an exec probe, use Node.js (already present in the image) to check that PID 1 is still running. This is a **shallow** check — it only confirms the process table entry for PID 1 exists; it does not verify GitHub API connectivity or queue health. An HTTP readiness endpoint is the tracked solution in [#1577](https://github.com/bradygaster/squad/issues/1577). + +```yaml +# Shallow PID-1 existence check (no procps / pgrep required) +# node:22-alpine ships without procps; this probe uses Node.js only. +livenessProbe: + exec: + command: + - node + - -e + - | + const fs = require('fs'); + fs.accessSync('/proc/1/status'); + initialDelaySeconds: 15 + periodSeconds: 30 + failureThreshold: 3 +``` + +> **Tracking:** An explicit HTTP health/readiness contract is planned. See [#1577](https://github.com/bradygaster/squad/issues/1577) for status. + +--- + +## Graceful Shutdown + +When Squad receives `SIGTERM` it: + +1. Stops accepting new issues from the queue. +2. Waits for any in-flight agent turns to complete (drain). +3. Flushes pending state to the configured backend. +4. Exits with code `0`. + +For Kubernetes, set `terminationGracePeriodSeconds` to at least `60` to give Squad time to drain in-flight work before the container runtime forcibly terminates it. + +--- + +## Secret Injection Patterns + +### Pattern 1 — environment variable (least secure) + +```bash +# Only for local testing. Never use in production. +docker run -e GITHUB_TOKEN=$GITHUB_TOKEN ghcr.io/bradygaster/squad-cli:latest +``` + +### Pattern 2 — mounted secret (Kubernetes) + +```yaml +env: + - name: GITHUB_TOKEN + valueFrom: + secretKeyRef: + name: squad-github-token + key: token +``` + +The secret value is injected into the environment at pod start. The secret object is managed separately (e.g., via External Secrets Operator or sealed secrets). + +### Pattern 3 — Azure Key Vault CSI driver (AKS, recommended) + +The [Secrets Store CSI Driver](https://learn.microsoft.com/azure/aks/csi-secrets-store-driver) mounts Key Vault secrets as files or environment variables at pod startup, with automatic rotation. + +```yaml +# SecretProviderClass (abbreviated) +apiVersion: secrets-store.csi.x-k8s.io/v1 +kind: SecretProviderClass +metadata: + name: squad-keyvault-secrets +spec: + provider: azure + parameters: + usePodIdentity: "false" + clientID: "" + keyvaultName: "" + objects: | + array: + - | + objectName: github-token + objectType: secret + tenantID: "" + secretObjects: + - secretName: squad-github-token + type: Opaque + data: + - objectName: github-token + key: token +``` + +Full example in the [AKS deployment runbook](../scenarios/aks-deployment). + +### Pattern 4 — ACA managed identity + Key Vault reference (Azure Container Apps) + +Azure Container Apps supports [Key Vault secret references](https://learn.microsoft.com/azure/container-apps/manage-secrets?tabs=azure-portal#reference-secret-from-key-vault) natively. No CSI driver required; the platform injects the secret value into the container environment automatically. See [Azure Container Apps deployment](../scenarios/azure-container-apps). + +--- + +## Current Limitations + +| Limitation | Status | Workaround | +|---|---|---| +| FSStorageProvider may resolve state paths relative to process working directory instead of volume mount when `rootDir` is not explicit in `config.json` | Open — [#1555](https://github.com/bradygaster/squad/issues/1555) | Set `rootDir` explicitly in `.squad/config.json` to the absolute volume mount path; no env var override exists yet | +| No HTTP health/readiness endpoints | Planned — [#1577](https://github.com/bradygaster/squad/issues/1577) | Use process-check exec probe; Kubernetes restarts exited containers automatically | +| `local` state backend not safe for concurrent multi-replica writes | Design gap — [#1402](https://github.com/bradygaster/squad/issues/1402) | Use `orphan` or `two-layer` backend, or scale to one replica | +| Remote dispatch (`--remote` flag) | RFC in progress — [#1189](https://github.com/bradygaster/squad/issues/1189) | Use KEDA queue-based scaling as a near-term alternative | + +--- + +## Building and Pushing the Image + +### GitHub Container Registry (GHCR) + +```bash +# Build +docker build -t ghcr.io//squad-agent:latest . + +# Authenticate (GitHub token with packages:write) +echo $GITHUB_TOKEN | docker login ghcr.io -u --password-stdin + +# Push +docker push ghcr.io//squad-agent:latest +``` + +### Azure Container Registry (ACR) + +```bash +# Authenticate with your Azure identity +az acr login --name + +# Build and push in one step using ACR Tasks (no local Docker daemon needed) +az acr build \ + --registry \ + --image squad-agent:latest \ + --file Dockerfile \ + . +``` + +For AKS, attach the ACR to your cluster to enable pull-through without credentials: + +```bash +az aks update \ + --name \ + --resource-group \ + --attach-acr +``` + +--- + +## See Also + +- [Dual-Mode Deployment](/squad/docs/features/dual-mode-deployment/) — `SQUAD_DEPLOYMENT_MODE` and `SQUAD_POD_ID` details +- [State Backends](/squad/docs/features/state-backends/) — choosing a backend safe for multi-replica deployments +- [Azure Container Apps deployment](../scenarios/azure-container-apps) — end-to-end ACA scenario +- [AKS deployment runbook](../scenarios/aks-deployment) — Kubernetes scenario with workload identity and KEDA +- [#1577](https://github.com/bradygaster/squad/issues/1577) — health/readiness endpoint product gap diff --git a/docs/src/content/docs/scenarios/aks-deployment.md b/docs/src/content/docs/scenarios/aks-deployment.md new file mode 100644 index 000000000..aae02cbc9 --- /dev/null +++ b/docs/src/content/docs/scenarios/aks-deployment.md @@ -0,0 +1,618 @@ +--- +title: AKS Deployment Runbook +description: Deploy Squad on Azure Kubernetes Service with workload identity, KEDA autoscaling, Key Vault CSI secrets, and production-grade security context. Includes troubleshooting for ≥4 failure modes. +order: 21 +--- + +# AKS Deployment Runbook + +> ⚠️ **Experimental** — Squad is alpha software. APIs, commands, and behavior may change between releases. + +This runbook walks you from a bare AKS cluster to a running, autoscaling Squad agent fleet with secretless authentication, Key Vault-backed secrets, and KEDA-driven GitHub issue queue scaling. + +**Prerequisites:** +- An AKS cluster (1.28+) with workload identity enabled +- KEDA installed on the cluster (AKS add-on or Helm) +- An Azure Container Registry attached to the cluster +- `kubectl`, `helm`, and `az` CLI installed locally +- Squad container image built and pushed (see [Container Image reference](/squad/docs/reference/container-image/)) + +--- + +## Architecture Overview + +``` +GitHub Issues + │ + ▼ +KEDA External Scaler ──── Squad KEDA ScaledObject + │ + ▼ + ┌────────────────────────┐ + │ squad-agent Deployment │ + │ • Workload Identity │ + │ • CSI Secrets Driver │ + │ • Non-root, read-only │ + └────────────────────────┘ + │ + ┌──────────┴──────────┐ + │ │ + Azure Key Vault Azure Monitor + (GitHub token, (OTel traces, + any other secrets) container logs) +``` + +--- + +## Step 1 — AKS Prerequisites + +### Enable AKS add-ons + +```bash +RESOURCE_GROUP="rg-squad-prod" +CLUSTER_NAME="aks-squad-prod" +ACR_NAME="squadregistry" +KEYVAULT_NAME="squad-kv-prod" +NAMESPACE="squad" + +# Enable workload identity (required for secretless auth) +az aks update \ + --name $CLUSTER_NAME \ + --resource-group $RESOURCE_GROUP \ + --enable-oidc-issuer \ + --enable-workload-identity + +# Enable KEDA add-on (AKS-managed KEDA — no Helm chart required) +az aks update \ + --name $CLUSTER_NAME \ + --resource-group $RESOURCE_GROUP \ + --enable-keda + +# Enable Secrets Store CSI driver add-on +az aks enable-addons \ + --name $CLUSTER_NAME \ + --resource-group $RESOURCE_GROUP \ + --addons azure-keyvault-secrets-provider + +# Attach ACR for image pull without stored credentials +az aks update \ + --name $CLUSTER_NAME \ + --resource-group $RESOURCE_GROUP \ + --attach-acr $ACR_NAME + +# Retrieve kubeconfig +az aks get-credentials \ + --name $CLUSTER_NAME \ + --resource-group $RESOURCE_GROUP +``` + +References: +- [AKS workload identity](https://learn.microsoft.com/azure/aks/workload-identity-overview) +- [AKS KEDA add-on](https://learn.microsoft.com/azure/aks/keda-about) +- [AKS Secrets Store CSI driver](https://learn.microsoft.com/azure/aks/csi-secrets-store-driver) + +--- + +## Step 2 — Workload Identity Setup + +Azure Workload Identity federates a Kubernetes ServiceAccount with an Azure AD managed identity. The Squad pod gets a short-lived OIDC token automatically — no long-lived PATs stored in secrets. + +```bash +OIDC_ISSUER=$(az aks show \ + --name $CLUSTER_NAME \ + --resource-group $RESOURCE_GROUP \ + --query oidcIssuerProfile.issuerUrl -o tsv) + +IDENTITY_NAME="squad-agent-id" +SUBSCRIPTION_ID=$(az account show --query id -o tsv) +TENANT_ID=$(az account show --query tenantId -o tsv) + +# Create managed identity +az identity create \ + --name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP + +IDENTITY_CLIENT_ID=$(az identity show \ + --name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP \ + --query clientId -o tsv) + +IDENTITY_OBJECT_ID=$(az identity show \ + --name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP \ + --query principalId -o tsv) + +# Kubernetes namespace and service account +kubectl create namespace $NAMESPACE + +kubectl create serviceaccount squad-agent \ + --namespace $NAMESPACE + +# Annotate the service account with the managed identity client ID +kubectl annotate serviceaccount squad-agent \ + --namespace $NAMESPACE \ + azure.workload.identity/client-id=$IDENTITY_CLIENT_ID + +# Federated credential — links the K8s ServiceAccount to the Azure identity +az identity federated-credential create \ + --name squad-agent-fedcred \ + --identity-name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP \ + --issuer $OIDC_ISSUER \ + --subject "system:serviceaccount:${NAMESPACE}:squad-agent" \ + --audience api://AzureADTokenExchange +``` + +Reference: [Use Azure workload identity with AKS](https://learn.microsoft.com/azure/aks/workload-identity-deploy-cluster) + +--- + +## Step 3 — Key Vault and Secrets + +```bash +# Create Key Vault +az keyvault create \ + --name $KEYVAULT_NAME \ + --resource-group $RESOURCE_GROUP \ + --enable-rbac-authorization true + +KV_ID=$(az keyvault show --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP --query id -o tsv) + +# Grant managed identity read access to secrets +az role assignment create \ + --assignee-object-id $IDENTITY_OBJECT_ID \ + --assignee-principal-type ServicePrincipal \ + --role "Key Vault Secrets User" \ + --scope $KV_ID + +# Store GitHub token +az keyvault secret set \ + --vault-name $KEYVAULT_NAME \ + --name github-token \ + --value "" +``` + +### SecretProviderClass + +Create a `SecretProviderClass` to mount the Key Vault secret into the pod: + +```yaml +# squad-secret-provider.yaml +apiVersion: secrets-store.csi.x-k8s.io/v1 +kind: SecretProviderClass +metadata: + name: squad-keyvault-secrets + namespace: squad +spec: + provider: azure + parameters: + usePodIdentity: "false" + clientID: "" # from Step 2 + keyvaultName: "" + objects: | + array: + - | + objectName: github-token + objectType: secret + objectVersion: "" + tenantID: "" + # Sync as a Kubernetes Secret so it can be injected as an env var + secretObjects: + - secretName: squad-github-token + type: Opaque + data: + - objectName: github-token + key: token +``` + +```bash +kubectl apply -f squad-secret-provider.yaml +``` + +Reference: [Use the Secrets Store CSI Driver with AKS](https://learn.microsoft.com/azure/aks/csi-secrets-store-driver) + +--- + +## Step 4 — ConfigMap for Squad Config + +Bake the `.squad/` configuration into a ConfigMap so it can be volume-mounted into pods without rebuilding the image on every config change. + +```bash +# Create ConfigMap from your .squad/ directory +kubectl create configmap squad-config \ + --namespace $NAMESPACE \ + --from-file=team.md=./.squad/team.md \ + --from-file=routing.md=./.squad/routing.md \ + --from-file=config.json=./.squad/config.json +``` + +> The ConfigMap covers static config files. Dynamic state (decisions, histories, etc.) should use the orphan or two-layer state backend rather than a ConfigMap, which does not support concurrent writes. See [State Backends](/squad/docs/features/state-backends/). + +--- + +## Step 5 — Deployment YAML + +```yaml +# squad-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: squad-agent + namespace: squad + labels: + app: squad-agent +spec: + replicas: 1 # KEDA will manage this; start at 1 for warmup + selector: + matchLabels: + app: squad-agent + template: + metadata: + labels: + app: squad-agent + azure.workload.identity/use: "true" # required for workload identity + spec: + serviceAccountName: squad-agent # from Step 2 + terminationGracePeriodSeconds: 60 # allow Squad to drain in-flight work + + securityContext: + runAsNonRoot: true + runAsUser: 1001 # matches non-root user in Dockerfile + runAsGroup: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault + + containers: + - name: squad-agent + image: .azurecr.io/squad-agent:latest + imagePullPolicy: Always + + resources: + requests: + cpu: "250m" + memory: "512Mi" + limits: + cpu: "1" + memory: "1Gi" + + env: + - name: GITHUB_TOKEN + valueFrom: + secretKeyRef: + name: squad-github-token # synced by SecretProviderClass + key: token + + - name: SQUAD_DEPLOYMENT_MODE + value: "squad-per-pod" + + - name: SQUAD_POD_ID + valueFrom: + fieldRef: + fieldPath: metadata.name # unique per pod replica + + # Optional — OpenTelemetry + # - name: OTEL_EXPORTER_OTLP_ENDPOINT + # value: "http://otel-collector.monitoring:4317" + # - name: OTEL_SERVICE_NAME + # value: "squad-agent" + + # Squad exposes no HTTP health endpoints (see #1577). + # Kubernetes restarts exited containers automatically. + # If your cluster policy requires an exec probe, use Node.js (present in + # node:22-alpine) to check /proc/1/status. This is a shallow PID-1 + # existence check only — not a readiness guarantee. See #1577 for the + # tracked HTTP health endpoint implementation. + livenessProbe: + exec: + command: + - node + - -e + - | + const fs = require('fs'); + fs.accessSync('/proc/1/status'); + initialDelaySeconds: 15 + periodSeconds: 30 + failureThreshold: 3 + + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: false # Squad writes state to /app/.squad + capabilities: + drop: ["ALL"] + + volumeMounts: + # CSI secret volume (required even if using secretObjects sync) + - name: secrets-store + mountPath: "/mnt/secrets" + readOnly: true + # Squad config from ConfigMap + - name: squad-config + mountPath: /app/.squad/team.md + subPath: team.md + - name: squad-config + mountPath: /app/.squad/routing.md + subPath: routing.md + - name: squad-config + mountPath: /app/.squad/config.json + subPath: config.json + + volumes: + - name: secrets-store + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: squad-keyvault-secrets + - name: squad-config + configMap: + name: squad-config +``` + +```bash +kubectl apply -f squad-deployment.yaml +kubectl rollout status deployment/squad-agent --namespace $NAMESPACE +``` + +--- + +## Step 6 — KEDA ScaledObject + +Scale Squad based on GitHub issue queue depth using the Squad external scaler. See [KEDA Autoscaling](/squad/docs/features/keda-scaling/) for scaler configuration details. + +First, deploy the Squad external scaler: + +```bash +helm repo add kedacore https://kedacore.github.io/charts +# If using AKS KEDA add-on, skip the helm install below +# helm install keda kedacore/keda --namespace keda --create-namespace + +kubectl apply -f templates/keda/scaler-deployment.yaml # from Squad repo +``` + +Then apply the ScaledObject: + +```yaml +# squad-scaledobject.yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: squad-agents + namespace: squad +spec: + scaleTargetRef: + name: squad-agent-deployment + minReplicaCount: 0 # scale to zero when no work + maxReplicaCount: 10 + pollingInterval: 30 # seconds between queue depth checks + cooldownPeriod: 300 # seconds to wait before scaling down + triggers: + - type: external + metadata: + scalerAddress: squad-external-scaler.squad:8080 + owner: + repo: + labels: "squad:ready" + targetQueueLength: "5" # issues per replica + authenticationRef: + name: squad-scaler-auth +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: squad-scaler-auth + namespace: squad +spec: + secretTargetRef: + - parameter: personalAccessToken + name: squad-github-token + key: token +``` + +```bash +kubectl apply -f squad-scaledobject.yaml +kubectl get scaledobject -n squad +``` + +Reference: [KEDA + AKS workload identity](https://learn.microsoft.com/azure/aks/keda-workload-identity) + +> **KEDA scaler PAT rotation and asymmetry:** The `TriggerAuthentication` above references `squad-github-token` — a Kubernetes Secret holding a GitHub PAT used by KEDA to poll the issue queue. This token is **separate** from the `GITHUB_TOKEN` the Squad pod itself uses to act on issues. They require independent rotation: +> - **Scaler PAT** — rotate by updating the Kubernetes Secret and restarting the KEDA operator or re-applying the `TriggerAuthentication`. KEDA does not hot-reload secrets. +> - **Agent token** — rotate via the CSI driver (Key Vault auto-rotation picks up the new version on the next CSI mount refresh cycle, without pod restart). +> +> Minimum scaler PAT scopes: `repo` (read issues/labels). The agent token needs the full scope set documented in the [Container Image contract](/squad/docs/reference/container-image/). + +--- + +## Step 7 — Persistent Volume for State (Optional) + +For durable state across pod restarts without using the orphan Git backend, mount an Azure Disk or Azure Files PVC: + +```yaml +# squad-pvc.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: squad-state-pvc + namespace: squad +spec: + accessModes: + - ReadWriteOnce # use ReadWriteMany (Azure Files) for multi-replica + storageClassName: managed-csi-premium + resources: + requests: + storage: 5Gi +``` + +Add to `squad-deployment.yaml` volumes and volumeMounts: + +```yaml +# In spec.volumes: +- name: squad-state + persistentVolumeClaim: + claimName: squad-state-pvc + +# In containers[].volumeMounts: +- name: squad-state + mountPath: /app/.squad/state +``` + +> ⚠️ **Multi-replica write safety:** `ReadWriteOnce` (Azure Disk) only supports one writer pod at a time. For multiple replicas, use `ReadWriteMany` with an Azure Files storage class, **and** switch to the orphan or two-layer state backend. See [#1402](https://github.com/bradygaster/squad/issues/1402). + +--- + +## Upgrade Strategy + +```bash +# Build and push new image +az acr build \ + --registry $ACR_NAME \ + --image squad-agent:v \ + --file Dockerfile \ + . + +# Rolling update (zero-downtime by default with RollingUpdate strategy) +kubectl set image deployment/squad-agent \ + squad-agent=.azurecr.io/squad-agent:v \ + --namespace $NAMESPACE + +kubectl rollout status deployment/squad-agent --namespace $NAMESPACE + +# Rollback if needed +kubectl rollout undo deployment/squad-agent --namespace $NAMESPACE +``` + +--- + +## Troubleshooting + +### OOMKilled — pod restarts with exit code 137 + +**Symptom:** `kubectl describe pod -n squad` shows `OOMKilled`; `kubectl get events -n squad` shows memory limit exceeded. + +```bash +# Check current resource usage +kubectl top pods --namespace $NAMESPACE + +# Check recent events +kubectl get events --namespace $NAMESPACE --sort-by='.lastTimestamp' +``` + +**Fix:** Increase `resources.limits.memory`. Start at `1Gi`; agent runs processing large repos or many parallel tools may need `2Gi+`. + +--- + +### KEDA not firing — replicas stay at 0 + +**Symptom:** Issues are labeled `squad:ready` but `kubectl get scaledobject -n squad` shows `READY=false` or `ACTIVE=false`. + +```bash +# Check ScaledObject status +kubectl describe scaledobject squad-agents --namespace $NAMESPACE + +# Check KEDA operator logs +kubectl logs -l app=keda-operator --namespace keda --tail=100 + +# Check external scaler logs +kubectl logs -l app=squad-external-scaler --namespace $NAMESPACE --tail=50 +``` + +Common causes: +- External scaler pod is not running (`kubectl get pods -n $NAMESPACE`) +- `personalAccessToken` in `TriggerAuthentication` is invalid or rate-limited +- `owner`/`repo`/`labels` in ScaledObject metadata don't match your repository +- ScaledObject was created before the target Deployment existed + +--- + +### GitHub API 401 — unauthorized + +**Symptom:** Squad logs `401 Unauthorized` when calling the GitHub API; no issues are processed. + +```bash +# Exec into the pod to test the token +kubectl exec -it --namespace $NAMESPACE -- \ + wget -qO- --header="Authorization: Bearer $(cat /mnt/secrets/github-token)" \ + https://api.github.com/user + +# Verify the CSI secret mount +kubectl exec -it --namespace $NAMESPACE -- \ + ls /mnt/secrets/ +``` + +Common causes: +- The managed identity does not have `Key Vault Secrets User` on the vault +- Federated credential subject does not match `system:serviceaccount::` exactly +- The pod is not labeled `azure.workload.identity/use: "true"` +- The ServiceAccount annotation `azure.workload.identity/client-id` is missing or wrong + +Reference: [Troubleshoot workload identity in AKS](https://learn.microsoft.com/azure/aks/workload-identity-overview#troubleshoot) + +--- + +### State backend contention — multiple pods stepping on each other + +**Symptom:** Duplicate issue processing; corrupted state files; agent history entries missing or overwritten. + +```bash +# Check how many replicas are running +kubectl get pods --namespace $NAMESPACE --selector app=squad-agent + +# Check which state backend is configured +kubectl exec -it --namespace $NAMESPACE -- \ + cat /app/.squad/config.json | grep stateBackend +``` + +**Fix:** Switch to the `orphan` or `two-layer` state backend, which uses Git-native structures safe for concurrent access. See [State Backends](/squad/docs/features/state-backends/) and open issue [#1402](https://github.com/bradygaster/squad/issues/1402). + +The `local` (file-based) backend is **not safe** for multi-replica deployments. + +--- + +### Pod stuck in Pending — image pull failure + +**Symptom:** `kubectl describe pod ` shows `ErrImagePull` or `ImagePullBackOff`. + +```bash +kubectl describe pod --namespace $NAMESPACE | grep -A 10 Events +``` + +Common causes: +- ACR not attached to the cluster: `az aks update --attach-acr $ACR_NAME --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP` +- Image tag or name typo in the Deployment spec +- ACR is in a different subscription than the AKS cluster — requires explicit role assignment + +--- + +## State Backend Warning + +> ⚠️ The `local` state backend (default) is **not safe for concurrent writes** from multiple Squad pods. The orphan-branch state backend uses a dedicated Git branch (`squad-state`) that is safe for multi-replica deployments because commits serialize writes. The two-layer backend further separates immutable assets from mutable state for better concurrency. Use one of these backends when scaling beyond a single replica. +> +> See [State Backends](/squad/docs/features/state-backends/) and [#1402](https://github.com/bradygaster/squad/issues/1402) for details and planned improvements. + +--- + +## Current Limitations and Future Work + +| Feature | Status | +|---|---| +| Remote dispatch (`--remote` flag for webhook-triggered runs) | RFC — [#1189](https://github.com/bradygaster/squad/issues/1189) | +| External state backend safety for multi-replica writes | Design — [#1402](https://github.com/bradygaster/squad/issues/1402) | +| FSStorageProvider rootDir bug (no env-var override; set `rootDir` explicitly in `config.json`) | Open — [#1555](https://github.com/bradygaster/squad/issues/1555) | +| HTTP health/readiness endpoints | Planned — [#1577](https://github.com/bradygaster/squad/issues/1577) | +| Helm chart for Squad (community request) | Not planned in this issue — contributions welcome | + +--- + +## Further Reading + +- [AKS workload identity overview](https://learn.microsoft.com/azure/aks/workload-identity-overview) — Microsoft Learn +- [KEDA with AKS and workload identity](https://learn.microsoft.com/azure/aks/keda-workload-identity) — Microsoft Learn +- [Secrets Store CSI Driver with AKS](https://learn.microsoft.com/azure/aks/csi-secrets-store-driver) — Microsoft Learn +- [AKS security concepts](https://learn.microsoft.com/azure/aks/concepts-security) — Microsoft Learn +- *Scaling AI Agents on Kubernetes with KEDA and Workload Identity* — Tamir Dresher, March 2026. Describes a reference deployment pattern for AI agent fleets on AKS using Helm, KEDA ScaledObjects, and Azure Workload Identity federation. The Squad configuration and KEDA trigger patterns in this runbook are informed by and consistent with that approach. [tamirdresher.com/blog/2026/03/26/scaling-ai-part8-pathfinder](https://www.tamirdresher.com/blog/2026/03/26/scaling-ai-part8-pathfinder) +- [squad-on-aks reference repository](https://github.com/tamirdresher/squad-on-aks) — Tamir Dresher's Helm + KEDA + Workload Identity reference repo, demonstrating the full deployment topology this runbook is based on. +- [Container Image contract](/squad/docs/reference/container-image/) — environment variables, Dockerfile, process lifecycle +- [KEDA Autoscaling](/squad/docs/features/keda-scaling/) — Squad KEDA ScaledObject configuration +- [Dual-Mode Deployment](/squad/docs/features/dual-mode-deployment/) — `SQUAD_DEPLOYMENT_MODE` and `SQUAD_POD_ID` +- [State Backends](/squad/docs/features/state-backends/) — choosing and migrating state backends diff --git a/docs/src/content/docs/scenarios/azure-container-apps.md b/docs/src/content/docs/scenarios/azure-container-apps.md new file mode 100644 index 000000000..66444b865 --- /dev/null +++ b/docs/src/content/docs/scenarios/azure-container-apps.md @@ -0,0 +1,387 @@ +--- +title: Azure Container Apps Deployment +description: Deploy Squad as a managed container agent on Azure Container Apps — workload identity, Key Vault secrets, KEDA scaling, ACA Jobs for one-shot dispatch, and troubleshooting. +order: 20 +--- + +# Azure Container Apps Deployment + +> ⚠️ **Experimental** — Squad is alpha software. APIs, commands, and behavior may change between releases. + +Azure Container Apps (ACA) is the lowest-friction path to running Squad in production on Azure. ACA provides managed KEDA scaling, managed TLS/ingress, and native workload identity — so you get autoscaling and secretless authentication without managing a Kubernetes control plane. + +This guide takes you from zero to a running Squad agent that polls a GitHub repository and executes issues automatically. + +**Prerequisites:** +- An Azure subscription with Contributor access +- `az` CLI 2.53+ and the `containerapp` extension: `az extension add --name containerapp` +- A container image pushed to ACR or GHCR (see [Container Image reference](/squad/docs/reference/container-image/)) +- A GitHub repository with Squad initialized (`squad init`) + +--- + +## Architecture Overview + +``` +GitHub Issues + │ + ▼ +┌─────────────────────────────────────────┐ +│ Azure Container Apps Environment │ +│ │ +│ ┌────────────────────────────────────┐ │ +│ │ Squad Agent Container App │ │ +│ │ • KEDA HTTP or queue scaler │ │ +│ │ • Workload identity (no PATs) │ │ +│ │ • Key Vault secret references │ │ +│ └────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────┐ │ +│ │ ACA Jobs (optional) │ │ +│ │ • One-shot issue dispatch │ │ +│ └────────────────────────────────────┘ │ +└─────────────────────────────────────────┘ + │ + ▼ + Azure Key Vault Azure Monitor / Log Analytics +``` + +--- + +## Step 1 — Resource Setup + +```bash +# Variables — adjust to your environment +RESOURCE_GROUP="rg-squad-prod" +LOCATION="eastus2" +ACA_ENV="squad-aca-env" +LOG_WORKSPACE="squad-logs" +ACR_NAME="squadregistry" # must be globally unique +KEYVAULT_NAME="squad-kv-prod" # must be globally unique +APP_NAME="squad-agent" +IDENTITY_NAME="squad-agent-id" + +# Resource group +az group create --name $RESOURCE_GROUP --location $LOCATION + +# Log Analytics workspace (required for ACA environment) +az monitor log-analytics workspace create \ + --resource-group $RESOURCE_GROUP \ + --workspace-name $LOG_WORKSPACE + +LOG_WORKSPACE_ID=$(az monitor log-analytics workspace show \ + --resource-group $RESOURCE_GROUP \ + --workspace-name $LOG_WORKSPACE \ + --query customerId -o tsv) + +LOG_WORKSPACE_KEY=$(az monitor log-analytics workspace get-shared-keys \ + --resource-group $RESOURCE_GROUP \ + --workspace-name $LOG_WORKSPACE \ + --query primarySharedKey -o tsv) + +# Container Apps environment +az containerapp env create \ + --name $ACA_ENV \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --logs-workspace-id $LOG_WORKSPACE_ID \ + --logs-workspace-key $LOG_WORKSPACE_KEY +``` + +Reference: [Azure Container Apps environments](https://learn.microsoft.com/azure/container-apps/environment) + +--- + +## Step 2 — Managed Identity + +Using a user-assigned managed identity is the recommended authentication strategy. It replaces long-lived GitHub PATs for Azure service access and enables OIDC federation for GitHub API calls. + +```bash +# Create user-assigned managed identity +az identity create \ + --name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP + +IDENTITY_CLIENT_ID=$(az identity show \ + --name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP \ + --query clientId -o tsv) + +IDENTITY_RESOURCE_ID=$(az identity show \ + --name $IDENTITY_NAME \ + --resource-group $RESOURCE_GROUP \ + --query id -o tsv) +``` + +Reference: [Managed identities in Azure Container Apps](https://learn.microsoft.com/azure/container-apps/managed-identity) + +### ACR pull with managed identity + +Grant the identity `AcrPull` on your registry so the container can pull the Squad image without a stored password: + +```bash +ACR_ID=$(az acr show --name $ACR_NAME --resource-group $RESOURCE_GROUP --query id -o tsv) + +az role assignment create \ + --assignee $IDENTITY_CLIENT_ID \ + --role AcrPull \ + --scope $ACR_ID +``` + +--- + +## Step 3 — Key Vault and GitHub Token Secret + +Store the GitHub token (or any other secrets) in Key Vault rather than in the container environment. + +```bash +# Create Key Vault +az keyvault create \ + --name $KEYVAULT_NAME \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --enable-rbac-authorization true + +# Grant the managed identity read access to secrets +KV_ID=$(az keyvault show --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP --query id -o tsv) + +az role assignment create \ + --assignee $IDENTITY_CLIENT_ID \ + --role "Key Vault Secrets User" \ + --scope $KV_ID + +# Store the GitHub token +az keyvault secret set \ + --vault-name $KEYVAULT_NAME \ + --name github-token \ + --value "" +``` + +Reference: [Key Vault secret references in Azure Container Apps](https://learn.microsoft.com/azure/container-apps/manage-secrets?tabs=azure-portal#reference-secret-from-key-vault) + +--- + +## Step 4 — Deploy the Container App + +```bash +# Use a versionless URI so ACA automatically picks up rotated secrets. +# az keyvault secret show returns a versioned URI — strip the version segment. +KV_SECRET_URI="https://${KEYVAULT_NAME}.vault.azure.net/secrets/github-token" + +az containerapp create \ + --name $APP_NAME \ + --resource-group $RESOURCE_GROUP \ + --environment $ACA_ENV \ + --image "${ACR_NAME}.azurecr.io/squad-agent:latest" \ + --registry-server "${ACR_NAME}.azurecr.io" \ + --registry-identity $IDENTITY_RESOURCE_ID \ + --user-assigned $IDENTITY_RESOURCE_ID \ + --secrets "github-token=keyvaultref:${KV_SECRET_URI},identityref:${IDENTITY_RESOURCE_ID}" \ + --env-vars \ + "GITHUB_TOKEN=secretref:github-token" \ + "SQUAD_DEPLOYMENT_MODE=agent-per-node" \ + "OTEL_SERVICE_NAME=squad-agent" \ + --cpu 0.5 \ + --memory 1.0Gi \ + --min-replicas 0 \ + --max-replicas 5 +``` + +> **Versionless URI and rotation:** Using the versionless secret URI (`…/secrets/` with no version segment) means ACA will resolve the latest active version of the secret on each new revision. If you rotate your GitHub token in Key Vault, create a new container app revision to pick up the new value. Using a versioned URI pins the secret and will not pick up rotations automatically. + +> **No ingress required.** Squad agents poll GitHub; they do not need to receive inbound HTTP traffic. Omit `--ingress` (or use `--ingress internal`) to restrict access to the ACA environment. + +--- + +## Step 5 — KEDA Scaling + +Azure Container Apps has [built-in KEDA support](https://learn.microsoft.com/azure/container-apps/scale-app?pivots=azure-cli). You can add scaling rules after creation or inline during `az containerapp create`. + +### Option A — ACA built-in HTTP scaler + +For HTTP-triggered workflows (webhook dispatch, future `--remote` mode — see [#1189](https://github.com/bradygaster/squad/issues/1189)): + +```bash +az containerapp update \ + --name $APP_NAME \ + --resource-group $RESOURCE_GROUP \ + --scale-rule-name http-scaler \ + --scale-rule-type http \ + --scale-rule-http-concurrency 10 +``` + +### Option B — KEDA external scaler (GitHub issue queue) + +Use the Squad KEDA external scaler to scale based on GitHub issue queue depth. This matches the pattern documented in [KEDA Autoscaling](/squad/docs/features/keda-scaling/). + +The KEDA external scaler must be deployed as a separate container in the same ACA environment: + +```bash +# Deploy external scaler service +az containerapp create \ + --name squad-external-scaler \ + --resource-group $RESOURCE_GROUP \ + --environment $ACA_ENV \ + --image "ghcr.io/bradygaster/squad-external-scaler:latest" \ + --env-vars "PORT=8080" \ + --ingress internal \ + --target-port 8080 \ + --min-replicas 1 \ + --max-replicas 1 +``` + +Then add a custom KEDA scaling rule to the Squad agent app using the KEDA external scaler protocol: + +```bash +az containerapp update \ + --name $APP_NAME \ + --resource-group $RESOURCE_GROUP \ + --scale-rule-name github-queue \ + --scale-rule-type external \ + --scale-rule-metadata \ + "scalerAddress=squad-external-scaler:8080" \ + "owner=" \ + "repo=" \ + "labels=squad:ready" \ + "targetQueueLength=5" \ + --scale-rule-auth "trigger=github-token" "secretRef=github-token" +``` + +Reference: [KEDA scaling rules in Azure Container Apps](https://learn.microsoft.com/azure/container-apps/scale-app?pivots=azure-cli#custom) + +> **KEDA scaler PAT rotation:** The KEDA external scaler uses `github-token` (a GitHub PAT) to poll the issue queue — this is a separate credential from the Squad agent's own `GITHUB_TOKEN` (which can also be a PAT). Both tokens need to be rotated independently. When you rotate the scaler's PAT in Key Vault and update the ACA secret, redeploy the scaler container to pick up the new value. There is no automatic token refresh in the current scaler implementation. See [KEDA Autoscaling](/squad/docs/features/keda-scaling/) for scaler configuration details. + +### Scaling to zero — cold start tradeoffs + +Setting `--min-replicas 0` eliminates idle cost but introduces a cold start penalty (typically 15–45 seconds for a Node.js container). During a cold start, issues that arrive are queued by KEDA and picked up when the first replica becomes ready. + +For time-sensitive workloads, set `--min-replicas 1` to keep one warm replica. + +--- + +## ACA Jobs — One-Shot Issue Dispatch + +For triggered, one-shot agent runs (e.g., a single issue dispatched by a webhook), use [ACA Jobs](https://learn.microsoft.com/azure/container-apps/jobs) instead of a long-running Container App: + +```bash +az containerapp job create \ + --name squad-job \ + --resource-group $RESOURCE_GROUP \ + --environment $ACA_ENV \ + --image "${ACR_NAME}.azurecr.io/squad-agent:latest" \ + --registry-server "${ACR_NAME}.azurecr.io" \ + --registry-identity $IDENTITY_RESOURCE_ID \ + --user-assigned $IDENTITY_RESOURCE_ID \ + --trigger-type Manual \ + --replica-timeout 1800 \ + --replica-retry-limit 1 \ + --replica-completion-count 1 \ + --parallelism 1 \ + --secrets "github-token=keyvaultref:${KV_SECRET_URI},identityref:${IDENTITY_RESOURCE_ID}" \ + --env-vars \ + "GITHUB_TOKEN=secretref:github-token" \ + "SQUAD_DEPLOYMENT_MODE=agent-per-node" +``` + +Trigger a job run manually: + +```bash +az containerapp job start \ + --name squad-job \ + --resource-group $RESOURCE_GROUP +``` + +> **Future:** Remote dispatch (`squad --remote`) is being designed in [#1189](https://github.com/bradygaster/squad/issues/1189). Once available, ACA Jobs will be the recommended execution backend for remote dispatch. + +--- + +## State Persistence Caveats + +By default, Squad uses the `local` state backend — files written to `.squad/` inside the container. Container filesystem is ephemeral: **state is lost when a replica restarts.** + +For durable state: + +1. **Orphan/two-layer backend** (recommended): State is committed to an orphan Git branch; persists across pod restarts. Requires `GITHUB_TOKEN` write scope. +2. **Azure Files volume mount**: Mount an Azure Files share to `/app/.squad/` for shared file-system persistence across replicas. +3. **Single replica**: If scaling to one replica (min=max=1), ephemeral state survives in-memory but not across restarts. + +> ⚠️ **Multi-replica safety:** The `local` state backend is **not safe for concurrent writes** from multiple replicas. Use the orphan or two-layer backend, or constrain to one replica. See [State Backends](/squad/docs/features/state-backends/) and [#1402](https://github.com/bradygaster/squad/issues/1402). + +--- + +## Troubleshooting + +### Missing or invalid `GITHUB_TOKEN` + +**Symptom:** Squad starts but immediately logs `401 Unauthorized` or `GitHub API rate limit exceeded` for an invalid token. + +```bash +# Check live logs +az containerapp logs show \ + --name $APP_NAME \ + --resource-group $RESOURCE_GROUP \ + --type console \ + --follow + +# Verify the secret is set +az containerapp secret list \ + --name $APP_NAME \ + --resource-group $RESOURCE_GROUP +``` + +Ensure the Key Vault secret URI is correct and the managed identity has `Key Vault Secrets User` on the vault. Check the identity assignment: + +```bash +az containerapp identity show \ + --name $APP_NAME \ + --resource-group $RESOURCE_GROUP +``` + +### KEDA scaler not triggering + +**Symptom:** Issues are labeled `squad:ready` but no new replicas start. + +- Verify the external scaler container is running: `az containerapp show --name squad-external-scaler --resource-group $RESOURCE_GROUP --query properties.runningStatus` +- Check scaler logs for GitHub API errors (rate limit, auth failure). +- Confirm the `targetQueueLength` is below the current issue count. +- Check the ACA scaling logs in Log Analytics: `az monitor log-analytics query --workspace $LOG_WORKSPACE_ID --analytics-query "ContainerAppConsoleLogs_CL | where ContainerName_s == 'squad-agent' | take 50"` + +### Cold start delays + +**Symptom:** First issue pickup after scale-from-zero is slow (15–45 s typical). + +- This is expected behavior — Squad has no readiness endpoint yet (see [#1577](https://github.com/bradygaster/squad/issues/1577)). ACA restarts exited containers automatically; no HTTP probe is needed. +- Set `--min-replicas 1` to keep one warm replica if latency matters. +- Use `node:22-alpine` base image (not `node:22`) for a smaller, faster startup. + +### Container exits immediately + +**Symptom:** Container starts and exits with code 1 within seconds. + +- Check that `GITHUB_TOKEN` is injected correctly (test with `az containerapp exec` if available). +- Verify the image tag matches what was pushed to ACR. +- Check startup logs: `az containerapp logs show --name $APP_NAME --resource-group $RESOURCE_GROUP --type system` + +--- + +## Current Limitations and Future Work + +| Feature | Status | +|---|---| +| `--remote` dispatch (webhook-triggered issue execution) | RFC — [#1189](https://github.com/bradygaster/squad/issues/1189) | +| External state gap for multi-replica deployments | Design — [#1402](https://github.com/bradygaster/squad/issues/1402) | +| FSStorageProvider rootDir bug (no env-var override; set `rootDir` in `config.json`) | Open — [#1555](https://github.com/bradygaster/squad/issues/1555) | +| HTTP health/readiness endpoints | Planned — [#1577](https://github.com/bradygaster/squad/issues/1577) | +| ACA Dynamic Sessions / Sandbox execution | Planned — [#1564](https://github.com/bradygaster/squad/issues/1564) | + +--- + +## Further Reading + +- [Azure Container Apps documentation](https://learn.microsoft.com/azure/container-apps/) — official reference +- [Managed identities in Azure Container Apps](https://learn.microsoft.com/azure/container-apps/managed-identity) — first-party guide +- [KEDA scaling in Azure Container Apps](https://learn.microsoft.com/azure/container-apps/scale-app) — built-in and custom scalers +- [Key Vault secret references](https://learn.microsoft.com/azure/container-apps/manage-secrets?tabs=azure-portal#reference-secret-from-key-vault) — secretless secret management +- [Container Image contract](/squad/docs/reference/container-image/) — environment variables, Dockerfile reference, process lifecycle +- [KEDA Autoscaling](/squad/docs/features/keda-scaling/) — Squad-specific KEDA ScaledObject configuration +- [State Backends](/squad/docs/features/state-backends/) — choosing a backend safe for container deployments