From 790f1e460aeb80a47a5590fe666bf360f320cbf2 Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Mon, 9 Nov 2026 22:18:43 -0800 Subject: [PATCH 1/3] test(backend): run the golden trace live on the claude-agent-sdk harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit golden-trace.json (decoupling §14.4, G1–G9) previously ran only against a fake harness. A live variant now runs it on the real Claude Agent SDK behind SQUAD_REAL_CLI_TESTS=1 and an Anthropic API key: a real Bash approval survives a client disconnect and cursor reconnect, the broker approves it and the command runs (the first live approve), then a cancel ends the turn cancelled, with the fixture's backend events in order. Passed 2026-11-10 (Agent SDK 0.3.317, 8.1 s). Recorded in the host feature matrix and decoupling spec §12. Env-gated in CI by the existing reconciliation rule. Refs #2441 Co-Authored-By: Claude Opus 5.5 --- .../client-backend-decoupling-v0.2.md | 6 +- .../docs/reference/host-feature-matrix.md | 3 +- .../spec/suite-v0.2/host-feature-matrix.json | 16 +++- test/claude-agent-sdk-harness.test.ts | 74 +++++++++++++++++++ 4 files changed, 95 insertions(+), 4 deletions(-) diff --git a/docs/specification/client-backend-decoupling-v0.2.md b/docs/specification/client-backend-decoupling-v0.2.md index ac1005086..7b42d4d21 100644 --- a/docs/specification/client-backend-decoupling-v0.2.md +++ b/docs/specification/client-backend-decoupling-v0.2.md @@ -518,7 +518,11 @@ message.accepted(runId, seq) `client.*` entries are client-side steps that produce no backend event. The trace is executed end to end against the file-backed local backend and the durable-http client in `test/squad-20-all-scenarios.e2e.test.ts`, with the -same canonical history on both. +same canonical history on both. It also runs live on the real `claude-agent-sdk` harness +(`test/claude-agent-sdk-harness.test.ts`, behind `SQUAD_REAL_CLI_TESTS=1` and an +Anthropic API key): the approval is a real tool call, the approved command +runs, and the fixture's backend events appear in order among the harness's +own (verified 2026-11-10). This trace captures the core invariants of the split: stable `runId` cancellation, monotonic `seq` ordering, durable approval versioning, resumable diff --git a/docs/src/content/docs/reference/host-feature-matrix.md b/docs/src/content/docs/reference/host-feature-matrix.md index a1ccb3064..bb514cc33 100644 --- a/docs/src/content/docs/reference/host-feature-matrix.md +++ b/docs/src/content/docs/reference/host-feature-matrix.md @@ -972,7 +972,7 @@ Depends on: **c** · docs: [concepts/squad-2-0-architecture](/squad/docs/concept | Host | Capability | Validation | Mechanism | Evidence | |------|------------|------------|-----------|----------| | `copilot` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `copilot-sdk` harness (`CopilotSdkHarness`) executes turns on the Copilot SDK; opt-in on the bridge via `SQUAD_BACKEND_HARNESSES=copilot-sdk`. _Live: plain turns through the broker on the real Copilot backend (2026-11-09; again 2026-11-10 after event correlation, including with a harness-only config directory). Approvals are covered by the shared contract, not yet live._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/copilot-sdk-harness.test.ts::completes a turn through the broker`
`validated-test:test/copilot-sdk-harness.test.ts::completes a turn with a harness-only Copilot config directory (login still works)`
`validated-manual:lab:copilot-sdk-harness@2026-11-10` | -| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`
`validated-manual:lab:claude-agent-sdk-harness@2026-11-10` | +| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn; a write denied through the broker (not run); and the golden trace (G1–G9): a real approval across a client reconnect, approved and run, then cancelled. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`
`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`
`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel`
`validated-manual:lab:claude-agent-sdk-golden-trace@2026-11-10` | | `kiro` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | | `opencode` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | | `cursor` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | @@ -1055,4 +1055,5 @@ Headless real-CLI runs behind the `validated-manual:lab:` evidence ent | 2026-11-09 | `copilot` | copilot-sdk-harness | ✅ pass | — | copilot-sdk HarnessAdapter (@github/copilot-sdk 1.0.19, signed-in Copilot user) completed one turn through LocalSquadBackend: "Reply with exactly the word: pong" → completed, transcript ends in pong (4.4 s). Plain text only; no tool approval in this run. | | 2026-11-10 | `claude` | claude-agent-sdk-harness | ✅ pass | `Bash` | claude-agent-sdk HarnessAdapter (@anthropic-ai/claude-agent-sdk 0.3.317, Anthropic API key) through LocalSquadBackend: a plain turn completed (6.7 s); a Bash `touch` reached the broker as an approval, was denied, ended the turn cancelled, and the file was not created (6.1 s). A read-only `echo` ran without an approval request (Claude Code treats it as safe). | | 2026-11-10 | `copilot` | copilot-sdk-harness | ✅ pass | — | After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe. | +| 2026-11-10 | `claude` | claude-agent-sdk-golden-trace | ✅ pass | `Bash` | golden-trace.json (G1–G9) on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval survived a client disconnect and cursor reconnect, the broker approved it and the file was created (first live approve), then a cancel during `sleep 60` ended the turn cancelled; the fixture's backend events appeared in order. | diff --git a/test-fixtures/spec/suite-v0.2/host-feature-matrix.json b/test-fixtures/spec/suite-v0.2/host-feature-matrix.json index 960e8f9a0..39a08a288 100644 --- a/test-fixtures/spec/suite-v0.2/host-feature-matrix.json +++ b/test-fixtures/spec/suite-v0.2/host-feature-matrix.json @@ -4401,9 +4401,11 @@ "validated-test:test/harness-contract.test.ts::*", "validated-test:test/claude-agent-sdk-harness.test.ts::*", "validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny", - "validated-manual:lab:claude-agent-sdk-harness@2026-11-10" + "validated-manual:lab:claude-agent-sdk-harness@2026-11-10", + "validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel", + "validated-manual:lab:claude-agent-sdk-golden-trace@2026-11-10" ], - "notes": "Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker." + "notes": "Live (2026-11-10): a plain turn; a write denied through the broker (not run); and the golden trace (G1–G9): a real approval across a client reconnect, approved and run, then cancelled. Claude Code runs some read-only commands without asking, so those never reach the broker." }, "kiro": { "capability": "supported", @@ -5017,6 +5019,16 @@ "outcome": "pass", "observedTools": [], "notes": "After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe." + }, + { + "host": "claude", + "scenario": "claude-agent-sdk-golden-trace", + "date": "2026-11-10", + "outcome": "pass", + "observedTools": [ + "Bash" + ], + "notes": "golden-trace.json (G1–G9) on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval survived a client disconnect and cursor reconnect, the broker approved it and the file was created (first live approve), then a cancel during `sleep 60` ended the turn cancelled; the fixture's backend events appeared in order." } ], "expectedFailures": [ diff --git a/test/claude-agent-sdk-harness.test.ts b/test/claude-agent-sdk-harness.test.ts index ad3158c37..9566f422c 100644 --- a/test/claude-agent-sdk-harness.test.ts +++ b/test/claude-agent-sdk-harness.test.ts @@ -12,6 +12,7 @@ import { ClaudeAgentSdkHarness, DefaultHarnessRegistry, LocalSquadBackend, + MemoryStateStore, claudeApprovalHash, type ClaudeAgentSdkModule, type ClaudeQueryOptions, @@ -286,6 +287,79 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK)', () => }, 180_000); }); +describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden trace', () => { + /** + * golden-trace.json (decoupling §14.4) on the real Claude harness: accept → a real tool approval → client + * disconnect and reconnect from a cursor → approve (the approved command really runs) → cancel → cancelled. + * Ids and the hash come from the harness rather than the fixture; the fixture's backend events must appear in + * order among the real ones (a real harness adds messages and running states). + */ + it('runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel', async () => { + const fs = await import('node:fs'); + const os = await import('node:os'); + const path = await import('node:path'); + const golden = JSON.parse(fs.readFileSync(path.resolve('test-fixtures/spec/suite-v0.2/golden-trace.json'), 'utf8')) as { + conversationId: string; runId: string; expectedTerminalState: string; events: Array & { event: string }>; + }; + const ev = (name: string) => golden.events.find((e) => e.event === name)!; + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'squad-claude-golden-')); + const marker = path.join(cwd, 'squad-golden-marker.txt'); + const harness = new ClaudeAgentSdkHarness({ cwd, maxTurns: 6 }); + const registry = new DefaultHarnessRegistry(); + registry.register(harness); + const state = new MemoryStateStore(); + const connect = () => new LocalSquadBackend({ state, harnesses: registry }); + let client = connect(); + const { conversationId, runId } = golden; + const seq = ev('message.accepted')['turnSeq'] as number; + try { + await client.createConversation({ conversationId }); + const adm = await client.sendMessage(conversationId, { + runId, seq, role: 'user', harness: HARNESS, + content: 'Use the Bash tool to run these two commands, one after the other, exactly as written: touch squad-golden-marker.txt ; then: sleep 60', + }); + expect(adm).toMatchObject({ runId, seq, duplicate: false }); + + await until(async () => (await client.getApprovals(conversationId)).some((a) => a.state === 'pending'), 120_000); + const requested = ev('approval.requested'); + const [v1] = await client.getApprovals(conversationId); + expect(v1).toMatchObject({ version: requested['version'], state: requested['state'], scope: { kind: requested['scope'] } }); + expect(v1!.action).toMatch(/^Bash: .*touch/); + + // client.disconnected: remember the cursor and drop the client; client.reconnected: resume from it + const beforeDrop = await client.readEvents(conversationId); + const seen = new Map(beforeDrop.events.map((e) => [e.id, e])); + client = connect(); + const resume = await client.readEvents(conversationId, { after: beforeDrop.nextCursor }); + expect(resume.resumable).toBe(ev('client.reconnected')['resumable']); + expect(resume.lastAcceptedSeq).toBe(beforeDrop.events.at(-1)!.sequence); + for (const e of (await client.readEvents(conversationId)).events) seen.set(e.id, e); + expect(seen.size).toBeGreaterThanOrEqual(beforeDrop.events.length); + + const decided = ev('approval.resolved'); + const turn = await client.respondApproval(conversationId, { approvalId: v1!.approvalId, hash: v1!.hash, decision: decided['decision'] as 'approve' }); + expect(turn.state).toBe('running'); + // The approved command really runs: the first live approve on the real harness. + await until(() => fs.existsSync(marker), 60_000); + + const cancelled = await client.cancelRun(conversationId, runId, ev('cancellation.requested')['reason'] as string); + expect(cancelled.state).toBe(golden.expectedTerminalState); + await harness.settled(conversationId, runId, seq); + + const history = (await client.readEvents(conversationId)).events.map((e) => e.type); + const expected = golden.events.map((e) => e.event).filter((n) => !n.startsWith('client.')); + let i = 0; + for (const type of history) if (type === expected[i]) i++; + expect(i, `golden events in order within: ${history.join(', ')}`).toBe(expected.length); + const approvals = await client.getApprovals(conversationId); + expect(approvals[0]).toMatchObject({ approvalId: v1!.approvalId, version: decided['version'], state: decided['state'], decision: 'approve' }); + expect((await client.getTurn(conversationId, seq)).state).toBe(golden.expectedTerminalState); + } finally { + fs.rmSync(cwd, { recursive: true, force: true }); + } + }, 300_000); +}); + describe('claude-agent-sdk harness: state-mcp registration', () => { it('registers only when the SDK is installed, and SQUAD_BACKEND_HARNESSES=none turns it off', async () => { const { createLocalHarnesses } = await import('../packages/squad-cli/src/cli/core/backend-mcp.js'); From bc26ccf2a7b09ac4d1e66c96888b407517bcb630 Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Mon, 9 Nov 2026 23:30:02 -0800 Subject: [PATCH 2/3] test(backend): tighten live golden trace review assertions Part of #2441. Address existing feedback on PR #2452: verify replay continuity and cursor suffixes, approve only the exact foreground command, require prompt harness cancellation, and cancel/settle on failure. Qualify historical smoke evidence pending a strengthened live rerun. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23 --- .../client-backend-decoupling-v0.2.md | 12 ++-- .../docs/reference/host-feature-matrix.md | 4 +- .../spec/suite-v0.2/host-feature-matrix.json | 7 +-- test/claude-agent-sdk-harness.test.ts | 62 ++++++++++++++----- 4 files changed, 61 insertions(+), 24 deletions(-) diff --git a/docs/specification/client-backend-decoupling-v0.2.md b/docs/specification/client-backend-decoupling-v0.2.md index 7b42d4d21..a207ce919 100644 --- a/docs/specification/client-backend-decoupling-v0.2.md +++ b/docs/specification/client-backend-decoupling-v0.2.md @@ -518,11 +518,15 @@ message.accepted(runId, seq) `client.*` entries are client-side steps that produce no backend event. The trace is executed end to end against the file-backed local backend and the durable-http client in `test/squad-20-all-scenarios.e2e.test.ts`, with the -same canonical history on both. It also runs live on the real `claude-agent-sdk` harness +same canonical history on both. A live variant uses the real `claude-agent-sdk` harness (`test/claude-agent-sdk-harness.test.ts`, behind `SQUAD_REAL_CLI_TESTS=1` and an -Anthropic API key): the approval is a real tool call, the approved command -runs, and the fixture's backend events appear in order among the harness's -own (verified 2026-11-10). +Anthropic API key): it checks exact event replay and cursor suffixes, approves +only the exact foreground command, then requires cancellation to settle the +harness within 10 seconds of its start signal, well below its 60-second sleep. +The original smoke run passed on 2026-11-10, but did not prove replay continuity +or prompt harness cancellation. The strengthened live trace awaits a new +credentialed run; the earlier smoke result is not promotion evidence for those +invariants. This trace captures the core invariants of the split: stable `runId` cancellation, monotonic `seq` ordering, durable approval versioning, resumable diff --git a/docs/src/content/docs/reference/host-feature-matrix.md b/docs/src/content/docs/reference/host-feature-matrix.md index bb514cc33..867889ee1 100644 --- a/docs/src/content/docs/reference/host-feature-matrix.md +++ b/docs/src/content/docs/reference/host-feature-matrix.md @@ -972,7 +972,7 @@ Depends on: **c** · docs: [concepts/squad-2-0-architecture](/squad/docs/concept | Host | Capability | Validation | Mechanism | Evidence | |------|------------|------------|-----------|----------| | `copilot` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `copilot-sdk` harness (`CopilotSdkHarness`) executes turns on the Copilot SDK; opt-in on the bridge via `SQUAD_BACKEND_HARNESSES=copilot-sdk`. _Live: plain turns through the broker on the real Copilot backend (2026-11-09; again 2026-11-10 after event correlation, including with a harness-only config directory). Approvals are covered by the shared contract, not yet live._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/copilot-sdk-harness.test.ts::completes a turn through the broker`
`validated-test:test/copilot-sdk-harness.test.ts::completes a turn with a harness-only Copilot config directory (login still works)`
`validated-manual:lab:copilot-sdk-harness@2026-11-10` | -| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn; a write denied through the broker (not run); and the golden trace (G1–G9): a real approval across a client reconnect, approved and run, then cancelled. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`
`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`
`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel`
`validated-manual:lab:claude-agent-sdk-golden-trace@2026-11-10` | +| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command allowlist, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`
`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`
`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel` | | `kiro` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | | `opencode` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | | `cursor` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | @@ -1055,5 +1055,5 @@ Headless real-CLI runs behind the `validated-manual:lab:` evidence ent | 2026-11-09 | `copilot` | copilot-sdk-harness | ✅ pass | — | copilot-sdk HarnessAdapter (@github/copilot-sdk 1.0.19, signed-in Copilot user) completed one turn through LocalSquadBackend: "Reply with exactly the word: pong" → completed, transcript ends in pong (4.4 s). Plain text only; no tool approval in this run. | | 2026-11-10 | `claude` | claude-agent-sdk-harness | ✅ pass | `Bash` | claude-agent-sdk HarnessAdapter (@anthropic-ai/claude-agent-sdk 0.3.317, Anthropic API key) through LocalSquadBackend: a plain turn completed (6.7 s); a Bash `touch` reached the broker as an approval, was denied, ended the turn cancelled, and the file was not created (6.1 s). A read-only `echo` ran without an approval request (Claude Code treats it as safe). | | 2026-11-10 | `copilot` | copilot-sdk-harness | ✅ pass | — | After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe. | -| 2026-11-10 | `claude` | claude-agent-sdk-golden-trace | ✅ pass | `Bash` | golden-trace.json (G1–G9) on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval survived a client disconnect and cursor reconnect, the broker approved it and the file was created (first live approve), then a cancel during `sleep 60` ended the turn cancelled; the fixture's backend events appeared in order. | +| 2026-11-10 | `claude` | claude-agent-sdk-golden-trace | ✅ pass | `Bash` | Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending. | diff --git a/test-fixtures/spec/suite-v0.2/host-feature-matrix.json b/test-fixtures/spec/suite-v0.2/host-feature-matrix.json index 39a08a288..896787413 100644 --- a/test-fixtures/spec/suite-v0.2/host-feature-matrix.json +++ b/test-fixtures/spec/suite-v0.2/host-feature-matrix.json @@ -4402,10 +4402,9 @@ "validated-test:test/claude-agent-sdk-harness.test.ts::*", "validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny", "validated-manual:lab:claude-agent-sdk-harness@2026-11-10", - "validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel", - "validated-manual:lab:claude-agent-sdk-golden-trace@2026-11-10" + "validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel" ], - "notes": "Live (2026-11-10): a plain turn; a write denied through the broker (not run); and the golden trace (G1–G9): a real approval across a client reconnect, approved and run, then cancelled. Claude Code runs some read-only commands without asking, so those never reach the broker." + "notes": "Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command allowlist, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Claude Code runs some read-only commands without asking, so those never reach the broker." }, "kiro": { "capability": "supported", @@ -5028,7 +5027,7 @@ "observedTools": [ "Bash" ], - "notes": "golden-trace.json (G1–G9) on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval survived a client disconnect and cursor reconnect, the broker approved it and the file was created (first live approve), then a cancel during `sleep 60` ended the turn cancelled; the fixture's backend events appeared in order." + "notes": "Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending." } ], "expectedFailures": [ diff --git a/test/claude-agent-sdk-harness.test.ts b/test/claude-agent-sdk-harness.test.ts index 9566f422c..5c009b874 100644 --- a/test/claude-agent-sdk-harness.test.ts +++ b/test/claude-agent-sdk-harness.test.ts @@ -30,6 +30,20 @@ async function until(check: () => boolean | Promise, ms = 2000): Promis } } +async function within(work: Promise, ms: number): Promise { + let timer: ReturnType; + try { + return await Promise.race([ + work, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`harness did not settle within ${ms} ms`)), ms); + }), + ]); + } finally { + clearTimeout(timer!); + } +} + interface ToolCall { name: string; input: Record; id: string } /** @@ -228,8 +242,8 @@ describe('claude-agent-sdk harness: deterministic protocol checks', () => { }); }); -const live = process.env['SQUAD_REAL_CLI_TESTS'] === '1' && - !!(process.env['ANTHROPIC_API_KEY'] || process.env['CLAUDE_CODE_USE_BEDROCK'] || process.env['CLAUDE_CODE_USE_VERTEX']); +// Opting in without credentials or the optional SDK must fail, not silently skip. +const live = process.env['SQUAD_REAL_CLI_TESTS'] === '1'; describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK)', () => { it('completes a turn through the broker', async () => { @@ -304,6 +318,8 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t const ev = (name: string) => golden.events.find((e) => e.event === name)!; const cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'squad-claude-golden-')); const marker = path.join(cwd, 'squad-golden-marker.txt'); + const command = 'touch squad-golden-marker.txt && exec sleep 60'; + const expectedApprovedAction = `Bash: ${command}`; const harness = new ClaudeAgentSdkHarness({ cwd, maxTurns: 6 }); const registry = new DefaultHarnessRegistry(); registry.register(harness); @@ -316,7 +332,7 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t await client.createConversation({ conversationId }); const adm = await client.sendMessage(conversationId, { runId, seq, role: 'user', harness: HARNESS, - content: 'Use the Bash tool to run these two commands, one after the other, exactly as written: touch squad-golden-marker.txt ; then: sleep 60', + content: `Use the Bash tool exactly once to run this single command, exactly as written, with no background execution: ${command}`, }); expect(adm).toMatchObject({ runId, seq, duplicate: false }); @@ -324,29 +340,45 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t const requested = ev('approval.requested'); const [v1] = await client.getApprovals(conversationId); expect(v1).toMatchObject({ version: requested['version'], state: requested['state'], scope: { kind: requested['scope'] } }); - expect(v1!.action).toMatch(/^Bash: .*touch/); + expect(v1!.action).toBe(expectedApprovedAction); // client.disconnected: remember the cursor and drop the client; client.reconnected: resume from it const beforeDrop = await client.readEvents(conversationId); - const seen = new Map(beforeDrop.events.map((e) => [e.id, e])); + const cursorSequence = beforeDrop.events.at(-1)!.sequence!; client = connect(); const resume = await client.readEvents(conversationId, { after: beforeDrop.nextCursor }); expect(resume.resumable).toBe(ev('client.reconnected')['resumable']); - expect(resume.lastAcceptedSeq).toBe(beforeDrop.events.at(-1)!.sequence); - for (const e of (await client.readEvents(conversationId)).events) seen.set(e.id, e); - expect(seen.size).toBeGreaterThanOrEqual(beforeDrop.events.length); + expect(resume.lastAcceptedSeq).toBe(cursorSequence); + expect(resume.events).toEqual([]); + expect(resume.nextCursor).toBe(beforeDrop.nextCursor); + const replayed = (await client.readEvents(conversationId)).events; + expect(replayed.slice(0, beforeDrop.events.length)).toEqual(beforeDrop.events); + expect(new Set(replayed.map((e) => e.id)).size).toBe(replayed.length); + expect(await client.getApprovals(conversationId)).toEqual([v1]); const decided = ev('approval.resolved'); const turn = await client.respondApproval(conversationId, { approvalId: v1!.approvalId, hash: v1!.hash, decision: decided['decision'] as 'approve' }); expect(turn.state).toBe('running'); - // The approved command really runs: the first live approve on the real harness. - await until(() => fs.existsSync(marker), 60_000); + // One approved foreground command signals startup before its 60-second sleep. + await until(() => fs.existsSync(marker), 10_000); + expect((await client.getTurn(conversationId, seq)).state).toBe('running'); + expect(harness.activeTurns).toBe(1); - const cancelled = await client.cancelRun(conversationId, runId, ev('cancellation.requested')['reason'] as string); - expect(cancelled.state).toBe(golden.expectedTerminalState); - await harness.settled(conversationId, runId, seq); + await within((async () => { + const cancelled = await client.cancelRun(conversationId, runId, ev('cancellation.requested')['reason'] as string); + expect(cancelled.state).toBe(golden.expectedTerminalState); + await harness.settled(conversationId, runId, seq); + })(), 10_000); - const history = (await client.readEvents(conversationId)).events.map((e) => e.type); + const allEvents = (await client.readEvents(conversationId)).events; + expect(allEvents.slice(0, beforeDrop.events.length)).toEqual(beforeDrop.events); + expect(new Set(allEvents.map((e) => e.id)).size).toBe(allEvents.length); + const resumed = await client.readEvents(conversationId, { after: beforeDrop.nextCursor }); + expect(resumed.events.length).toBeGreaterThan(0); + expect(resumed.events).toEqual(allEvents.filter((e) => e.sequence! > cursorSequence)); + expect(resumed.events.every((e) => e.sequence! > cursorSequence)).toBe(true); + expect(resumed.lastAcceptedSeq).toBe(allEvents.at(-1)!.sequence); + const history = allEvents.map((e) => e.type); const expected = golden.events.map((e) => e.event).filter((n) => !n.startsWith('client.')); let i = 0; for (const type of history) if (type === expected[i]) i++; @@ -355,6 +387,8 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t expect(approvals[0]).toMatchObject({ approvalId: v1!.approvalId, version: decided['version'], state: decided['state'], decision: 'approve' }); expect((await client.getTurn(conversationId, seq)).state).toBe(golden.expectedTerminalState); } finally { + await harness.cancel(conversationId, runId, seq); + await harness.settled(conversationId, runId, seq); fs.rmSync(cwd, { recursive: true, force: true }); } }, 300_000); From 550f5955688deee117185a66dc09e9cc5d62c4ca Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Mon, 9 Nov 2026 23:45:51 -0800 Subject: [PATCH 3/3] test(backend): validate golden Bash execution options Part of #2441. Complete PR #2452's cancellation finding with test-owned full-input interception, explicit long foreground timeout and hash binding before approval, and deterministic short-timeout/background/default negatives plus an abort-honouring control. Keep strengthened live promotion evidence pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c8c234f1-ee3b-49e1-9406-11d4a2238c23 --- .../client-backend-decoupling-v0.2.md | 8 +- .../docs/reference/host-feature-matrix.md | 2 +- .../spec/suite-v0.2/host-feature-matrix.json | 2 +- test/claude-agent-sdk-harness.test.ts | 134 +++++++++++++++++- 4 files changed, 137 insertions(+), 9 deletions(-) diff --git a/docs/specification/client-backend-decoupling-v0.2.md b/docs/specification/client-backend-decoupling-v0.2.md index a207ce919..22af1c6a8 100644 --- a/docs/specification/client-backend-decoupling-v0.2.md +++ b/docs/specification/client-backend-decoupling-v0.2.md @@ -521,8 +521,12 @@ durable-http client in `test/squad-20-all-scenarios.e2e.test.ts`, with the same canonical history on both. A live variant uses the real `claude-agent-sdk` harness (`test/claude-agent-sdk-harness.test.ts`, behind `SQUAD_REAL_CLI_TESTS=1` and an Anthropic API key): it checks exact event replay and cursor suffixes, approves -only the exact foreground command, then requires cancellation to settle the -harness within 10 seconds of its start signal, well below its 60-second sleep. +only the exact command after test-owned interception verifies the full hashed +Bash input (`run_in_background: false`, explicit `timeout: 120000`), then requires +cancellation to settle the harness within 10 seconds of its start signal, well +below its 60-second sleep and 120-second tool timeout. Deterministic negative +cases reject short timeouts (including an abort-ignoring transport), background +execution and implicit timeout defaults before approval. The original smoke run passed on 2026-11-10, but did not prove replay continuity or prompt harness cancellation. The strengthened live trace awaits a new credentialed run; the earlier smoke result is not promotion evidence for those diff --git a/docs/src/content/docs/reference/host-feature-matrix.md b/docs/src/content/docs/reference/host-feature-matrix.md index 867889ee1..e1ef4c901 100644 --- a/docs/src/content/docs/reference/host-feature-matrix.md +++ b/docs/src/content/docs/reference/host-feature-matrix.md @@ -972,7 +972,7 @@ Depends on: **c** · docs: [concepts/squad-2-0-architecture](/squad/docs/concept | Host | Capability | Validation | Mechanism | Evidence | |------|------------|------------|-----------|----------| | `copilot` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `copilot-sdk` harness (`CopilotSdkHarness`) executes turns on the Copilot SDK; opt-in on the bridge via `SQUAD_BACKEND_HARNESSES=copilot-sdk`. _Live: plain turns through the broker on the real Copilot backend (2026-11-09; again 2026-11-10 after event correlation, including with a harness-only config directory). Approvals are covered by the shared contract, not yet live._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/copilot-sdk-harness.test.ts::completes a turn through the broker`
`validated-test:test/copilot-sdk-harness.test.ts::completes a turn with a harness-only Copilot config directory (login still works)`
`validated-manual:lab:copilot-sdk-harness@2026-11-10` | -| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command allowlist, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`
`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`
`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel` | +| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`
`validated-test:test/harness-contract.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::*`
`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`
`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`
`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel` | | `kiro` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | | `opencode` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | | `cursor` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`
`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` | diff --git a/test-fixtures/spec/suite-v0.2/host-feature-matrix.json b/test-fixtures/spec/suite-v0.2/host-feature-matrix.json index 896787413..a87e84f1e 100644 --- a/test-fixtures/spec/suite-v0.2/host-feature-matrix.json +++ b/test-fixtures/spec/suite-v0.2/host-feature-matrix.json @@ -4404,7 +4404,7 @@ "validated-manual:lab:claude-agent-sdk-harness@2026-11-10", "validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel" ], - "notes": "Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command allowlist, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Claude Code runs some read-only commands without asking, so those never reach the broker." + "notes": "Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker." }, "kiro": { "capability": "supported", diff --git a/test/claude-agent-sdk-harness.test.ts b/test/claude-agent-sdk-harness.test.ts index 5c009b874..3e06ee26b 100644 --- a/test/claude-agent-sdk-harness.test.ts +++ b/test/claude-agent-sdk-harness.test.ts @@ -14,6 +14,7 @@ import { LocalSquadBackend, MemoryStateStore, claudeApprovalHash, + claudeApiCredentialsConfigured, type ClaudeAgentSdkModule, type ClaudeQueryOptions, type ClaudeSdkMessage, @@ -46,6 +47,48 @@ async function within(work: Promise, ms: number): Promise { interface ToolCall { name: string; input: Record; id: string } +const GOLDEN_COMMAND = 'touch squad-golden-marker.txt && exec sleep 60'; +const GOLDEN_TOOL_TIMEOUT = 120_000; + +function goldenToolGuard() { + const requests = new Map(); + return { + wrap(sdk: ClaudeAgentSdkModule): ClaudeAgentSdkModule { + return { + query({ prompt, options = {} }) { + const canUseTool = options.canUseTool!; + return sdk.query({ + prompt, + options: { + ...options, + canUseTool(name, input, context) { + requests.set(context.toolUseID, { name, input: structuredClone(input), id: context.toolUseID }); + return canUseTool(name, input, context); + }, + }, + }); + }, + }; + }, + async approve(client: LocalSquadBackend, conversationId: string, approval: { approvalId: string; hash: string }, decision: 'approve') { + const request = requests.get(approval.approvalId); + if (request?.name !== 'Bash' || request.input['command'] !== GOLDEN_COMMAND) { + throw new Error('golden trace requires the exact Bash command input'); + } + if (request.input['run_in_background'] !== false) { + throw new Error('golden trace requires explicit foreground execution'); + } + if (request.input['timeout'] !== GOLDEN_TOOL_TIMEOUT) { + throw new Error(`golden trace requires an explicit ${GOLDEN_TOOL_TIMEOUT} ms tool timeout`); + } + if (approval.hash !== claudeApprovalHash(request.name, request.input)) { + throw new Error('golden trace approval hash does not match the captured input'); + } + return client.respondApproval(conversationId, { approvalId: approval.approvalId, hash: approval.hash, decision }); + }, + }; +} + /** * Replays one recorded exchange: optional tool calls (each through `canUseTool`, concurrently, as Claude * does for parallel calls), then the assistant text and a result event. @@ -242,6 +285,79 @@ describe('claude-agent-sdk harness: deterministic protocol checks', () => { }); }); +describe('claude-agent-sdk harness: golden execution guard', () => { + function timedTool(input: Record, ignoresAbort = false) { + const decisions: string[] = []; + let started = false; + const sdk: ClaudeAgentSdkModule = { + query({ options = {} }) { + return (async function* () { + const result = await options.canUseTool!('Bash', input, { signal: options.abortController!.signal, toolUseID: 'golden-tool' }); + decisions.push(result.behavior); + if (result.behavior === 'allow') { + started = true; + await new Promise((resolve) => { + const timeout = setTimeout(resolve, input['timeout'] as number); + if (!ignoresAbort) { + const stop = () => { clearTimeout(timeout); resolve(); }; + options.abortController!.signal.addEventListener('abort', stop, { once: true }); + if (options.abortController!.signal.aborted) stop(); + } + }); + } + yield { type: 'result', subtype: 'success', is_error: false }; + })(); + }, + }; + return { sdk, decisions, started: () => started }; + } + + it.each([ + { name: 'short timeout even when the transport ignores abort', input: { command: GOLDEN_COMMAND, timeout: 1000, run_in_background: false }, ignoresAbort: true, error: /explicit 120000 ms tool timeout/ }, + { name: 'background execution', input: { command: GOLDEN_COMMAND, timeout: GOLDEN_TOOL_TIMEOUT, run_in_background: true }, ignoresAbort: false, error: /explicit foreground/ }, + { name: 'implicit timeout defaults', input: { command: GOLDEN_COMMAND, run_in_background: false }, ignoresAbort: false, error: /explicit 120000 ms tool timeout/ }, + ])('refuses $name before approval', async ({ input, ignoresAbort, error }) => { + const transport = timedTool(input, ignoresAbort); + const guard = goldenToolGuard(); + const { harness, backend, send } = await setup(guard.wrap(transport.sdk)); + try { + await send('r1', 1); + await until(async () => (await backend.getApprovals('c')).length === 1); + const [approval] = await backend.getApprovals('c'); + expect(approval!.action).toBe(`Bash: ${GOLDEN_COMMAND}`); + await expect(guard.approve(backend, 'c', approval!, 'approve')).rejects.toThrow(error); + expect((await backend.getTurn('c', 1)).state).toBe('awaiting-approval'); + expect(transport.started()).toBe(false); + } finally { + await harness.cancel('c', 'r1', 1); + await harness.settled('c', 'r1', 1); + } + expect(transport.decisions).toEqual(['deny']); + }); + + it('approves the long foreground input and settles on cancellation, not tool timeout', async () => { + const transport = timedTool({ command: GOLDEN_COMMAND, timeout: GOLDEN_TOOL_TIMEOUT, run_in_background: false }); + const guard = goldenToolGuard(); + const { harness, backend, send } = await setup(guard.wrap(transport.sdk)); + try { + await send('r1', 1); + await until(async () => (await backend.getApprovals('c')).length === 1); + const [approval] = await backend.getApprovals('c'); + await guard.approve(backend, 'c', approval!, 'approve'); + await until(transport.started); + await within((async () => { + await backend.cancelRun('c', 'r1'); + await harness.settled('c', 'r1', 1); + })(), 2000); + expect((await backend.getTurn('c', 1)).state).toBe('cancelled'); + expect(transport.decisions).toEqual(['allow']); + } finally { + await harness.cancel('c', 'r1', 1); + await harness.settled('c', 'r1', 1); + } + }); +}); + // Opting in without credentials or the optional SDK must fail, not silently skip. const live = process.env['SQUAD_REAL_CLI_TESTS'] === '1'; @@ -318,9 +434,17 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t const ev = (name: string) => golden.events.find((e) => e.event === name)!; const cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'squad-claude-golden-')); const marker = path.join(cwd, 'squad-golden-marker.txt'); - const command = 'touch squad-golden-marker.txt && exec sleep 60'; - const expectedApprovedAction = `Bash: ${command}`; - const harness = new ClaudeAgentSdkHarness({ cwd, maxTurns: 6 }); + const expectedApprovedAction = `Bash: ${GOLDEN_COMMAND}`; + const guard = goldenToolGuard(); + const harness = new ClaudeAgentSdkHarness({ + cwd, maxTurns: 6, + loadSdk: async () => { + // A test-owned SDK wrapper must retain the real harness's API-only authentication guard. + if (!claudeApiCredentialsConfigured()) throw new Error('golden trace needs API credentials, not a claude.ai login'); + const module = '@anthropic-ai/claude-agent-sdk'; + return guard.wrap(await import(module) as ClaudeAgentSdkModule); + }, + }); const registry = new DefaultHarnessRegistry(); registry.register(harness); const state = new MemoryStateStore(); @@ -332,7 +456,7 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t await client.createConversation({ conversationId }); const adm = await client.sendMessage(conversationId, { runId, seq, role: 'user', harness: HARNESS, - content: `Use the Bash tool exactly once to run this single command, exactly as written, with no background execution: ${command}`, + content: `Use the Bash tool exactly once with timeout: ${GOLDEN_TOOL_TIMEOUT} and run_in_background: false. Run this single command exactly as written: ${GOLDEN_COMMAND}`, }); expect(adm).toMatchObject({ runId, seq, duplicate: false }); @@ -357,7 +481,7 @@ describe.skipIf(!live)('claude-agent-sdk harness: live (real Agent SDK) golden t expect(await client.getApprovals(conversationId)).toEqual([v1]); const decided = ev('approval.resolved'); - const turn = await client.respondApproval(conversationId, { approvalId: v1!.approvalId, hash: v1!.hash, decision: decided['decision'] as 'approve' }); + const turn = await guard.approve(client, conversationId, v1!, decided['decision'] as 'approve'); expect(turn.state).toBe('running'); // One approved foreground command signals startup before its 60-second sleep. await until(() => fs.existsSync(marker), 10_000);