[Cedric Vidal (cedricvidal)]

Original author: Cedric Vidal (@cedricvidal)

Why

Headless "Permission denied and could not request permission from user" errors were breaking benchmark runs in two distinct places. This PR fixes both. See issue #1104 (coder) and issue #1117 (judge).

Fix 1 - coder gets stuck in the build gate (#1104)

In integration, coder-acp-copilot runs failed the build gate: every execute/bash tool call (e.g. npm run build) was rejected non-interactively, so the gate exhausted all iterations, failed, and the downstream test gate was skipped.

Root cause. The Copilot CLI replaced its old bypassPermissions ACP session mode with agent / plan / autopilot. The worker only set the session mode when bypassPermissions was advertised, so the session stayed in the default agent mode, where execute/bash calls are denied non-interactively. The --yolo CLI flag does not change the ACP session mode, so it was not enough on its own.

Verified against Copilot CLI 1.0.64 by probing it directly over ACP:

• The ACP protocol does not define mode ids; they are agent-specific open strings.
• The CLI advertises three modes by canonical URL id (...#agent, ...#plan, ...#autopilot). Autopilot "enables allow-all and runs until task completion without user interaction."
• Started with --yolo, the session's currentModeId is still ...#agent.
• Calling session/set_mode with the autopilot URL id returns success.

Approach. Add an exported selectPermissionMode helper (mirroring selectModel) that switches the session into autopilot after session/new. It matches autopilot by its full canonical URL id, with an endsWith("#autopilot") safety net, warns and continues if the mode is absent, and is resilient to setSessionMode throwing. A single cold-start retry plus a formatModeError helper give clear diagnostics instead of [object Object]. The dead bypassPermissions block is removed.

Scope note: this intentionally drops backward compatibility with older Copilot CLI versions that exposed bypassPermissions; we target the current CLI's autopilot mode only.

Fix 2 - judge reports a bogus permission denial (#1117)

After Fix 1, the coder's commands ran fine, but the test gate still intermittently failed with a "Permission denied" message coming from the judge's feedback. Ground-truth captured tool-calls.jsonl proved the coder's npm test actually succeeded in every iteration - the denial was the judge's own.

Root cause. The judge's CopilotClient is created with no mode, so the SDK defaults to mode: "copilot-cli", which injects the built-in CLI tools (bash, edit, view, ...) into the session. Those built-ins are not skipPermission. When the headless judge model chose to run a build/test command itself instead of reading the coder's captured output, the runtime denied it ("could not request permission from user") and the judge mis-reported that as the coder's result - a non-deterministic failure even when the coder succeeded.

Approach. Restrict every judge session to its own custom tools via availableTools: ["custom:*"] and excludedTools: ["builtin:*", "mcp:*"] (excludedTools always wins), extracted into a testable buildSessionConfig helper. The judge keeps its 6 read-only skipPermission tools and is forced to evaluate from read_tool_outputs. Chose tool filtering over switching the client to mode: "empty" (the latter needs extra config and changes more defaults) as the surgical fix that directly closes the denial path.

Verification

Reproduced end to end against a local stack by retrying the original failing run:

• Zero "Permission denied" / "could not request permission" strings in the entire run log (previously appeared in the judge's feedback).
• The coder set autopilot mode and its execute/bash calls ran without denial.
• The judge now evaluates deterministically from readable evidence: it failed one test iteration with a legitimate reason ("no captured test run output") and passed once the coder wrote a verifiable test log - no permission error involved.

Tests and docs

• coder: apps/workers/coder-acp-copilot/src/acp-client.test.ts - 25 tests (autopilot by URL id, endsWith safety net, absent/empty modes, resilient to throw, retry-then-succeed, formatModeError serialization).
• judge: apps/judge/src/judge-strategies.test.ts - 16 tests, 5 new (restricts to custom:*, excludes builtin:*/mcp:*, never exposes bash, replace-mode prompt, exported constants).
• Builds clean (coder-acp-copilot, judge).
• Docs: docs/architecture/worker-requirements.md + worker-compliance.md (autopilot mode governs headless execution, not --yolo); docs/design/gates.md (judge tool isolation).

Closes #1104
Closes #1117

[Cedric Vidal (cedricvidal)]

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1957

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2096	2096	0	0	0	0	0	2m1s

🎉 All tests passed!

Github Test Reporter

[Cedric Vidal (cedricvidal)]

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1959

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2102	2102	0	0	0	0	0	2m8s

🎉 All tests passed!

Github Test Reporter

[Cedric Vidal (cedricvidal)]

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1962

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
2107	2107	0	0	0	0	0	1m55s

🎉 All tests passed!

Github Test Reporter