Repository navigation
feat(backend): claude-agent-sdk and copilot-sdk execution harnesses (#2441) - #2451
Merged
Merged
Conversation
Adds the `claude-agent-sdk` harness (experimental `./backend` subpath), running admitted turns on @anthropic-ai/claude-agent-sdk, an optional peer dependency loaded lazily (no lockfile change; @github/copilot-sdk stays a hard dependency). - One query() per turn, in order per conversation, resuming the conversation's Claude session; submissions deduplicated by operationId or (conversationId, runId, seq). - canUseTool -> approval-requested with approvalId = toolUseID and a sha256 hash of the canonical tool name and input; a response with a different hash never resolves the call; approve runs the original input. - The broker holds one pending approval per turn, so parallel tool calls are queued and offered one at a time; a deny ends the turn (as the broker marks it cancelled), denying queued calls and aborting. - permissionMode 'default' and settingSources [] so no repository or user allow rule approves a call the broker never sees. Tests: 13 deterministic protocol tests replay scripted SDK exchanges through the real adapter and LocalSquadBackend; 2 live tests run the real SDK behind SQUAD_REAL_CLI_TESTS=1 and credentials (not yet run). Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… contract - `squad state-mcp --experimental-backend` registers the claude-agent-sdk harness when the optional @anthropic-ai/claude-agent-sdk peer resolves (isClaudeAgentSdkAvailable), running in the repository root; SQUAD_BACKEND_HARNESSES=none turns it off. Without the SDK nothing changes. - test/harness-contract.ts: a reusable HarnessAdapter contract (describe, event identity and epoch, redelivery, hash-bound approvals with replay, one approval at a time with deny ending the turn, idempotent cancel) through the real LocalSquadBackend, run for claude-agent-sdk. No Copilot harness exists yet, so it runs against one harness for now. - Specs, docs and help no longer say no harness ships: harness-adapters §5 and a §14 layout row, the decoupling spec, the architecture doc, the SDK reference and both state-mcp help texts. Changeset added. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tract on both - copilot-sdk (CopilotSdkHarness) on @github/copilot-sdk: one Copilot session per conversation (created, then resumed by id), send() then session.idle / session.error; only main-agent assistant.message content enters the transcript. onPermissionRequest goes to broker approvals: approve-once on approve, reject plus abort on deny; toolCallId is the approvalId when present. Cancel aborts without waiting for idle. - TurnApprovals (approval-queue.ts): the hash-bound, one-at-a-time, deny-ends-the-turn queue both harnesses now share; claude-agent-sdk moved onto it with its hash format unchanged. - The shared HarnessAdapter contract runs against copilot-sdk and claude-agent-sdk (harness-adapters §15 item 1), plus Copilot-specific tests and one live test behind SQUAD_REAL_CLI_TESTS=1. - state-mcp registers copilot-sdk only when SQUAD_BACKEND_HARNESSES lists it, since its SDK is always installed. - Specs, docs, help and the changeset describe both harnesses. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- liveRuns: copilot / copilot-sdk-harness / 2026-11-09 / pass. One plain-text turn through LocalSquadBackend on the real Copilot backend (@github/copilot-sdk 1.0.19, signed-in user, 4.4 s); no tool approval. - experimental-backend.copilot -> validated-live with the lab entry, the live test and the shared harness contract; experimental-backend.claude cites the contract and harness tests (live runs need an API key). - The matrix test allows validated-live for copilot now that it has a recorded passing run; the limitations note says so. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
🟠 Impact Analysis — PR #2451Risk tier: 🟠 HIGH 📊 Summary
🎯 Risk Factors
📦 Modules Affecteddocs (5 files)
root (2 files)
scripts (1 file)
squad-cli (3 files)
squad-sdk (7 files)
tests (7 files)
|
Contributor
🏗️ Architectural Review
Automated architectural review — informational only. |
Contributor
🛫 PR Readiness Check
PR Scope: 📦🔧 Mixed (product + infrastructure)
|
| Status | Check | Details |
|---|---|---|
| ❌ | Single commit | 13 commits — consider squashing before review |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | Changeset file found |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ✅ | Copilot threads resolved | 3 active Copilot thread(s) resolved (11 outdated skipped) |
| ❌ | CI passing | 1 check(s) still running |
Files Changed (25 files, +2576 −40)
| File | +/− |
|---|---|
.changeset/claude-agent-sdk-harness.md |
+6 −0 |
docs/specification/client-backend-decoupling-v0.2.md |
+15 −5 |
docs/specification/client-harness-adapters-v0.2.md |
+70 −4 |
docs/src/content/docs/concepts/squad-2-0-architecture.md |
+1 −1 |
docs/src/content/docs/reference/host-feature-matrix.md |
+9 −6 |
docs/src/content/docs/reference/sdk.md |
+3 −1 |
packages/squad-cli/src/cli/commands/state-mcp.ts |
+5 −2 |
packages/squad-cli/src/cli/core/backend-mcp.ts |
+75 −7 |
packages/squad-cli/src/cli/core/command-help.ts |
+3 −2 |
packages/squad-sdk/package.json |
+8 −0 |
packages/squad-sdk/src/backend/harnesses/approval-queue.ts |
+215 −0 |
packages/squad-sdk/src/backend/harnesses/claude-agent-sdk.ts |
+245 −0 |
packages/squad-sdk/src/backend/harnesses/copilot-sdk.ts |
+251 −0 |
packages/squad-sdk/src/backend/harnesses/harness-state.ts |
+78 −0 |
packages/squad-sdk/src/backend/harnesses/turn-harness.ts |
+233 −0 |
packages/squad-sdk/src/backend/index.ts |
+9 −0 |
scripts/release/test-reconciliation.mjs |
+1 −0 |
test-fixtures/spec/suite-v0.2/host-feature-matrix.json |
+45 −9 |
test/claude-agent-sdk-harness.test.ts |
+328 −0 |
test/copilot-sdk-harness.test.ts |
+301 −0 |
test/harness-contract.test.ts |
+153 −0 |
test/harness-contract.ts |
+211 −0 |
test/harness-review-round4.test.ts |
+306 −0 |
test/squad-20-host-feature-matrix.test.ts |
+2 −2 |
test/squad-20-real-clients-stdio.e2e.test.ts |
+3 −1 |
Total: +2576 −40
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…agent-sdk-harness The only conflict was the generated host-feature-matrix.md; it is regenerated from the auto-merged matrix JSON, which keeps both sides (experimental-backend.copilot validated-live with its live run, and ralph-cli-triage-watch.claude supported from #2450). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bradygaster
commented
Nov 10, 2026
bradygaster
left a comment
bradygaster
left a comment
Collaborator
Author
There was a problem hiding this comment.
SDK runtime review (sdk-runtime-dev), against head 2fb5cbcd6832630ad200c01dd908ff3f9e463928 and freshly fetched dev/merge-base 344705b0ae5e392ff03d39278728c038a85d88c9. Five actionable, nonduplicate correctness findings are inline. This is a COMMENT review, not an approval or a formal blocking verdict.
I traced admission/subscription, SDK execution, approval delivery, cancellation, conversation serialization, epoch filtering, optional loading/registration and public exports. All reproductions use the actual adapters and LocalSquadBackend with deterministic in-process SDK doubles, not live API calls. A dependency-free Node 24 runner produced 10 failing invariant cases across these five findings, with 4 passing normal-flow controls (completion/resume/epoch/sequential retry, and hash binding/replay/serialized approvals/deny, for both adapters). Source was not patched.
Validation limitations: npm test -- test/claude-agent-sdk-harness.test.ts test/copilot-sdk-harness.test.ts test/harness-contract.test.ts test/backend.test.ts test/squad-20-host-feature-matrix.test.ts could not start (vitest: command not found). npm run lint, npm run build --workspace packages/squad-sdk, and npm run build --workspace packages/squad-cli could not start (tsc: command not found). Locked restore attempts failed: official npm transport/npm exit-handler failure; offline cache missing zod-to-json-schema; configured mirror returned 404 for locked vite-8.3.6.tgz. No dependency versions or locks were weakened. Consequently I have not independently verified the claimed four broad failures against dev.
No live/charged execution was attempted. Claude live/G1–G9 evidence and restart durability remain explicitly acknowledged draft/promotion work, not new findings here. No later projection/plugin roadmap is being requested. Review was isolated; author branches/worktrees and PR body/draft state were left untouched.
Review of #2451. Both harnesses now share TurnHarness, which owns the broker-facing lifecycle: - Atomic acceptance: concurrent deliveries of one operation share one acceptance; a failed start forgets it so the broker's retry runs. - Cancellation survives startup: the run is registered before the runtime starts, and every setup await is followed by a checkpoint, so a cancel during SDK load, client start or session creation never reaches query()/send(). - Ordering: the conversation's next turn starts only after an interrupt has finished (Copilot awaits session.abort()), so late events from an aborted turn can't be attributed to the next one. cancel() acknowledges once the stop is requested, since the broker awaits it. - Approvals are scoped to their conversation (tool-call ids repeat). Copilot: events from sub-agents (agentId) no longer complete or fail the main turn; only sessions that finished creating are aborted. Approval records: TurnApprovals stores a short summary and a redacted payload (secret-looking keys and inline credentials masked, long strings truncated); the hash still covers the full action. Tests: five new shared contract checks for both harnesses (startup cancel, concurrent and retried delivery, cross-conversation approval ids, redaction) and four Copilot-specific ones (cancel during session creation, slow abort with a queued turn and late events, sub-agent idle and error). Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cords Code scanning review on #2451. Inline credential names now allow prefix segments (X-, GITHUB_, AWS_SECRET_) and cover cookie, access-key and auth-token names, so Cookie headers and AWS_SECRET_ACCESS_KEY are masked too (X-API-Key and X-Auth-Token already were). Values stop at quotes and &, so the surrounding command or URL stays intact. Look-alikes such as --author=, tokenizer.py and mkdir -p are left alone. Tests cover both. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code scanning review on #2451. redactText now also masks secrets with no name in front of them: well-known vendor token formats (GitHub, OpenAI and Anthropic, Slack, AWS, Google, GitLab, npm), JWTs, long opaque mixed-case base64-like tokens, and the password in scheme://user:pass@host. Plain hex (git SHAs, checksums), UUIDs, paths and package specs stay readable on purpose, since the approver has to see what they're approving. Tests cover both sides. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emory - Approval records carry only the id, a redacted summary and the hash; the tool input is never sent as `payload` (nothing read it), so secrets in tool arguments stay out of backend state regardless of redaction coverage. The hash still binds the decision to the full input. - copilot-sdk: optional `configDirectory` for harness sessions. The Copilot runtime can keep "approve permanently" decisions (URL prompts) in its config directory, so a session sharing the user's default could skip the broker for a URL approved in their own CLI. Default unchanged. - TurnHarness drops finished turns and idle conversation tails, and keeps the most recent 10,000 accepted delivery ids, so long-running bridges don't grow without bound; late redelivery of a finished turn stays a no-op. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects can expose credentials, conceal executable code from approvers, bypass deny rules, and repeat tool side effects after a restart.
14 open issues
What changed in this PR
Adds the first execution harnesses to the experimental SDK backend, allowing the conversation broker to dispatch turns to Claude Agent SDK or GitHub Copilot SDK.
Changes:
- Introduces both adapters with shared turn lifecycle and hash-bound approval handling.
- Adds configurable harness registration to the opt-in
state-mcpbackend. - Adds contract, regression and live tests; updates specifications and records Copilot live evidence.
| File | Description |
|---|---|
test/squad-20-real-clients-stdio.e2e.test.ts |
Updates the no-harness explanatory comment only. |
test/squad-20-host-feature-matrix.test.ts |
Allows recorded Copilot live validation. |
test/harness-contract.ts |
Defines shared adapter contract checks. |
test/harness-contract.test.ts |
Supplies scripted SDK drivers and redaction checks. |
test/copilot-sdk-harness.test.ts |
Tests Copilot execution, approvals and cancellation. |
test/claude-agent-sdk-harness.test.ts |
Tests Claude execution, permissions and registration. |
test-fixtures/spec/suite-v0.2/host-feature-matrix.json |
Records harness capabilities and live evidence. |
packages/squad-sdk/src/backend/index.ts |
Exports adapters and shared lifecycle utilities. |
packages/squad-sdk/src/backend/harnesses/turn-harness.ts |
Implements shared turn scheduling and cancellation. |
packages/squad-sdk/src/backend/harnesses/copilot-sdk.ts |
Implements Copilot session execution. |
packages/squad-sdk/src/backend/harnesses/claude-agent-sdk.ts |
Implements lazily loaded Claude execution. |
packages/squad-sdk/src/backend/harnesses/approval-queue.ts |
Implements approval queuing, hashing and redaction. |
packages/squad-sdk/package.json |
Declares the optional Claude SDK peer. |
packages/squad-cli/src/cli/core/command-help.ts |
Updates help text for Claude registration. |
packages/squad-cli/src/cli/core/backend-mcp.ts |
Wires configurable harnesses into the local backend. |
packages/squad-cli/src/cli/commands/state-mcp.ts |
Documents harness selection in command help. |
docs/src/content/docs/reference/sdk.md |
Documents adapter exports and bridge configuration. |
docs/src/content/docs/reference/host-feature-matrix.md |
Updates the generated harness validation matrix. |
docs/src/content/docs/concepts/squad-2-0-architecture.md |
Describes optional execution through the bridge. |
docs/specification/client-harness-adapters-v0.2.md |
Documents shipped adapters and shared behavior. |
docs/specification/client-backend-decoupling-v0.2.md |
Updates implementation status and restart limitations. |
.changeset/claude-agent-sdk-harness.md |
Adds SDK and CLI minor-release notes. |
Review effort: Max (auto)
- The live approval test asks for a write (`touch` in a temp directory), since Claude Code runs some read-only commands such as `echo` without calling canUseTool. It expects the broker deny to end the turn cancelled, checks the file was not created, and prints the turn, approvals and harness events on failure. - Both Claude live tests passed 2026-11-10 (Agent SDK 0.3.317, Anthropic API key): a plain turn through the broker, and a write that reached the broker, was denied and did not run. - Matrix: liveRuns claude / claude-agent-sdk-harness, and experimental-backend.claude -> validated-live. Spec §5 notes that read-only commands can bypass canUseTool. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Copilot correlation Copilot review of #2451 (round 4). Durability (HarnessState, `stateDir`): accepted delivery ids and each conversation's runtime session id survive restarts, so a redelivered turn never runs twice and the next turn resumes its session. state-mcp keeps this beside the local backend, or in a per-user cache for configured backends. Lifecycle: a failed interrupt stays retryable and tells the harness to stop reusing the session; dropping an approval the broker already holds ends the turn instead of offering the next one into the occupied slot. Claude: the real SDK starts only with an API credential (API key, auth token, Bedrock, Vertex or Foundry), never a saved claude.ai login, and state-mcp registers it only then. Deny rules from user, project and local settings are carried over as disallowedTools, since settingSources: [] would otherwise drop them. Copilot: a turn reacts only to events after its own user.message (matched to send()'s id), so late events from an aborted turn are ignored; a failed send, failed interrupt or timeout (turnTimeoutMs, default 30 min) drops the cached session; each permission request of one tool call gets its own approval id; state-mcp gives the harness its own Copilot config directory. Redaction: JSON keys, Authorization schemes, credential flags and -u user:password are masked; executable syntax is never masked. Bridge: turns run in the worktree-aware repoRoot. The stdio e2e test pins SQUAD_BACKEND_HARNESSES=none. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| const dir = (id: string) => (options.stateDir ? path.join(options.stateDir, id) : undefined); | ||
| const claudeAvailable = options.claudeSdkAvailable ?? isClaudeAgentSdkAvailable; | ||
| if (wanted.includes('claude-agent-sdk') && claudeAvailable() && claudeApiCredentialsConfigured(env)) { | ||
| registry.register(new ClaudeAgentSdkHarness({ cwd: repoRoot, env, ...(dir('claude-agent-sdk') ? { stateDir: dir('claude-agent-sdk')! } : {}) })); |
Collaborator
Author
There was a problem hiding this comment.
Not changed; leaving this open for a human reviewer. The harness gives env to the Agent SDK as Claude Code's environment, which is the same exposure as before: with no env, the SDK uses the full process environment, as any Claude Code session the user starts does. An allowlist would break the work bridge turns exist for. Approved tool calls need GITHUB_TOKEN/gh auth to open PRs, npm proxy and registry settings, PATH additions and toolchain variables. Those calls are already gated by broker approval, and settings deny rules apply as disallowedTools. Anyone embedding the harness who wants a narrower environment can pass env to ClaudeAgentSdkHarness directly.
🤖 Addressed by Claude Code
…liation "Verify exact SHA and build evidence" failed on #2451: the three live harness tests (claude-agent-sdk x2, copilot-sdk x1) skip without SQUAD_REAL_CLI_TESTS=1 and credentials, and had no reviewed rule, so they were unexplained. Add an env-gated rule scoped to the two harness test files and their "live (real ... SDK)" describe blocks only. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted config dir Two live Copilot runs on 2026-11-10 (@github/copilot-sdk 1.0.19, signed-in user), both passing: a plain turn through the broker (5.7 s), confirming the new correlation to the turn's own user.message, and the same turn with a harness-only configDirectory in an empty temp dir (3.2 s), confirming sign-in still works there. Live turns now use a 2-minute timeout so a correlation miss fails fast. Recorded in the host feature matrix. Refs #2441 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8 of 17 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the first two
HarnessAdapterimplementations to the experimental@bradygaster/squad-sdk/backendsubpath, plus a shared contract suite that runs against both. Phase 1 of epic #2438.claude-agent-sdk(ClaudeAgentSdkHarness): runs admitted turns on the Claude Agent SDK.@anthropic-ai/claude-agent-sdkis an optional peer dependency that's loaded lazily, sopackage-lock.jsondoesn't change and installs without it are unchanged.copilot-sdk(CopilotSdkHarness): runs turns on@github/copilot-sdk, which is already a dependency.TurnApprovals: one approval queue that both harnesses share.squad state-mcp --experimental-backendregistersclaude-agent-sdkwhen its SDK is installed.copilot-sdkis opt-in (SQUAD_BACKEND_HARNESSES=copilot-sdk), because its SDK is always installed.SQUAD_BACKEND_HARNESSES=noneturns both off.Why
Refs #2441. Until now the broker had the
HarnessAdapterinterface and a registry, but no implementation:squad_backend_send_messagecould only refuse. §15 of the harness-adapters spec asks for the same contract to run against Copilot and one non-Copilot host.How
Approvals. Both harnesses use
TurnApprovalsthe same way:cancelled. Queued calls are denied and the runtime is stopped.Claude.
query()per turn, in order per conversation, resuming the conversation's session.canUseToolbecomesapproval-requested, withapprovalId=toolUseID. Approve runs the original input unchanged.permissionMode: 'default'andsettingSources: [], so no repository or user allow rule can approve a call the broker never sees.Copilot.
send(), thensession.idle(completed) orsession.error(failed).assistant.messagecontent enters the transcript.onPermissionRequestbecomes approvals:approve-onceon approve,rejectplus abort on deny.Known limitation. In both harnesses a pending approval lives in process memory and is lost if the harness restarts. The specs say so: the broker's record must then be cancelled and the session resumed.
Tests
test/harness-contract.ts): 6 checks per harness through the realLocalSquadBackend, passing for both (12/12). It covers self-description, event identity and epoch, redelivery, hash-bound approvals with replay, one-at-a-time approvals with deny ending the turn, and idempotent cancel.test/claude-agent-sdk-harness.test.ts(15) andtest/copilot-sdk-harness.test.ts(8). The deterministic tests drive each real adapter with a scripted SDK; there is no network.@github/copilot-sdk1.0.19, signed-in user): one turn through the broker, 4.4 s.@anthropic-ai/claude-agent-sdk0.3.317, Anthropic API key): a plain turn, 6.7 s; and a Bashtouchthat reached the broker as an approval, was denied, ended the turncancelledand did not run, 6.1 s.experimental-backend.*→validated-live, withliveRunsentries). Claude Code runs some read-only commands (such asecho) without callingcanUseTool, so those never reach the broker; spec §5 says so.devon macOS.Open questions for reviewers
TurnHarnesslifecycle were added so the shared contract runs on Copilot plus one non-Copilot host (§15 item 1).Review rounds
Three rounds are addressed and resolved:
configDirectoryisolation, and bounded memory..changeset/claude-agent-sdk-harness.md, SDK and CLI, minor)PR Readiness Checklist
Branch & Commit
dev, up to dateBuild & Test
npm run build(SDK and CLI withtsc)npm test: 4 local failures that also fail on cleandev(see above)npm run lintnpm run lint:docsDocs
state-mcphelp texts--checkclean)Exports
./backendsubpath; no new subpathBreaking Changes
None. The harnesses are new and experimental.
copilot-sdkis opt-in on the bridge, andclaude-agent-sdkregisters only where its optional SDK is installed.Waivers
None.
🤖 Generated with Claude Code