Skip to content

test(backend): run the golden trace live on the claude-agent-sdk harness (#2441) #2452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bradygaster wants to merge 3 commits into
base: dev
Choose a base branch
from
+262 −6
Open

test(backend): run the golden trace live on the claude-agent-sdk harness (#2441) #2452

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
790f1e4
test(backend): run the golden trace live on the claude-agent-sdk harness
bradygaster Nov 10, 2026
bc26ccf
test(backend): tighten live golden trace review assertions
bradygaster Nov 10, 2026
550f595
test(backend): validate golden Bash execution options
bradygaster Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion docs/specification/client-backend-decoupling-v0.2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,19 @@ message.accepted(runId, seq)
`client.*` entries are client-side steps that produce no backend event. The
trace is executed end to end against the file-backed local backend and the
durable-http client in `test/squad-20-all-scenarios.e2e.test.ts`, with the
same canonical history on both.
same canonical history on both. A live variant uses the real `claude-agent-sdk` harness
(`test/claude-agent-sdk-harness.test.ts`, behind `SQUAD_REAL_CLI_TESTS=1` and an
Anthropic API key): it checks exact event replay and cursor suffixes, approves
only the exact command after test-owned interception verifies the full hashed
Bash input (`run_in_background: false`, explicit `timeout: 120000`), then requires
cancellation to settle the harness within 10 seconds of its start signal, well
below its 60-second sleep and 120-second tool timeout. Deterministic negative
cases reject short timeouts (including an abort-ignoring transport), background
execution and implicit timeout defaults before approval.
The original smoke run passed on 2026-11-10, but did not prove replay continuity
or prompt harness cancellation. The strengthened live trace awaits a new
credentialed run; the earlier smoke result is not promotion evidence for those
invariants.

This trace captures the core invariants of the split: stable `runId`
cancellation, monotonic `seq` ordering, durable approval versioning, resumable
Expand Down
3 changes: 2 additions & 1 deletion docs/src/content/docs/reference/host-feature-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -972,7 +972,7 @@ Depends on: **c** · docs: [concepts/squad-2-0-architecture](/squad/docs/concept
| Host | Capability | Validation | Mechanism | Evidence |
|------|------------|------------|-----------|----------|
| `copilot` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `copilot-sdk` harness (`CopilotSdkHarness`) executes turns on the Copilot SDK; opt-in on the bridge via `SQUAD_BACKEND_HARNESSES=copilot-sdk`. _Live: plain turns through the broker on the real Copilot backend (2026-11-09; again 2026-11-10 after event correlation, including with a harness-only config directory). Approvals are covered by the shared contract, not yet live._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/copilot-sdk-harness.test.ts::completes a turn through the broker`<br>`validated-test:test/copilot-sdk-harness.test.ts::completes a turn with a harness-only Copilot config directory (login still works)`<br>`validated-manual:lab:copilot-sdk-harness@2026-11-10` |
| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`<br>`validated-manual:lab:claude-agent-sdk-harness@2026-11-10` |
| `claude` | ✅ supported | 🖥️ validated-live | Opt-in tools over the same MCP server; backend semantics are host-independent. The `claude-agent-sdk` harness (`ClaudeAgentSdkHarness`) executes turns on the Claude Agent SDK when the optional peer is installed. _Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker._ | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect`<br>`validated-test:test/harness-contract.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::*`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny`<br>`validated-manual:lab:claude-agent-sdk-harness@2026-11-10`<br>`validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel` |
| `kiro` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` |
| `opencode` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` |
| `cursor` | ✅ supported | 🧪 validated-test | Opt-in tools over the same MCP server; backend semantics are host-independent. | `validated-test:test/squad-20-real-clients-stdio.e2e.test.ts::completes the MCP handshake and lists squad_state_* and all squad_backend_* tools when opted in`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Scenario 4: DurableHttpSquadBackend full HTTP lifecycle`<br>`validated-test:test/squad-20-all-scenarios.e2e.test.ts::Full non-bypass journey: host adapter → broker → harness → approval → restart/reconnect` |
Expand Down Expand Up @@ -1055,4 +1055,5 @@ Headless real-CLI runs behind the `validated-manual:lab:<scenario>` evidence ent
| 2026-11-09 | `copilot` | copilot-sdk-harness | ✅ pass | — | copilot-sdk HarnessAdapter (@github/copilot-sdk 1.0.19, signed-in Copilot user) completed one turn through LocalSquadBackend: "Reply with exactly the word: pong" → completed, transcript ends in pong (4.4 s). Plain text only; no tool approval in this run. |
| 2026-11-10 | `claude` | claude-agent-sdk-harness | ✅ pass | `Bash` | claude-agent-sdk HarnessAdapter (@anthropic-ai/claude-agent-sdk 0.3.317, Anthropic API key) through LocalSquadBackend: a plain turn completed (6.7 s); a Bash `touch` reached the broker as an approval, was denied, ended the turn cancelled, and the file was not created (6.1 s). A read-only `echo` ran without an approval request (Claude Code treats it as safe). |
| 2026-11-10 | `copilot` | copilot-sdk-harness | ✅ pass | — | After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe. |
| 2026-11-10 | `claude` | claude-agent-sdk-golden-trace | ✅ pass | `Bash` | Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending. |

15 changes: 13 additions & 2 deletions test-fixtures/spec/suite-v0.2/host-feature-matrix.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4401,9 +4401,10 @@
"validated-test:test/harness-contract.test.ts::*",
"validated-test:test/claude-agent-sdk-harness.test.ts::*",
"validated-test:test/claude-agent-sdk-harness.test.ts::routes a real tool call through broker approval and honours a deny",
"validated-manual:lab:claude-agent-sdk-harness@2026-11-10"
"validated-manual:lab:claude-agent-sdk-harness@2026-11-10",
"validated-test:test/claude-agent-sdk-harness.test.ts::runs G1–G9 on claude-agent-sdk: approve after reconnect, then cancel"
],
"notes": "Live (2026-11-10): one turn through the broker, and a write command approved-or-denied through the broker (denied: not run). Claude Code runs some read-only commands without asking, so those never reach the broker."
"notes": "Live (2026-11-10): a plain turn and a write denied through the broker (not run). The original golden-trace smoke run approved a real tool call after reconnect, but did not prove replay continuity or prompt harness cancellation. The strengthened trace (exact command and full hashed Bash input validation, explicit 120-second foreground tool timeout, cursor suffix and replay checks, 10-second cancellation deadline) awaits a new credentialed live run and is not yet promotion evidence. Deterministic negatives reject short-timeout/abort-ignoring transport, background execution and implicit timeouts before approval. Claude Code runs some read-only commands without asking, so those never reach the broker."
},
"kiro": {
"capability": "supported",
Expand Down Expand Up @@ -5017,6 +5018,16 @@
"outcome": "pass",
"observedTools": [],
"notes": "After review round 4 (event correlation to the turn's own user.message, durable state): one turn through LocalSquadBackend completed (5.7 s), and the same with a harness-only Copilot configDirectory in an empty temp dir (3.2 s): sign-in still works there, so state-mcp's isolated config directory is safe."
},
{
"host": "claude",
"scenario": "claude-agent-sdk-golden-trace",
"date": "2026-11-10",
"outcome": "pass",
"observedTools": [
"Bash"
],
"notes": "Historical smoke run of the original golden-trace test on the real claude-agent-sdk harness (Agent SDK 0.3.317, Anthropic API key), 8.1 s: a real Bash `touch` approval was approved after reconnect and the file was created; the broker recorded cancellation and the fixture's backend events appeared in order. Review found that the assertions did not prove replay continuity, cursor filtering or interruption of an executing sleep. This pass is not evidence for the strengthened trace; a new credentialed live run is pending."
}
],
"expectedFailures": [
Expand Down
Loading
Loading