[JacobStephens2]

Adds Custom runner Arms to process Experiments (ADR 0032). Either Arm, or both, can be an outside orchestrator run from a frozen .toml manifest instead of a thirdshift build. The runner works in one Trial sandbox and returns a commit and Session traces, and both Arms are graded by the same code Graders, Grader panel and blind pairwise review.

Closes #1468

Tickets

• Hand-run comparison of the Built-in Ticket run against a Custom process, by replaying closed issues #1628 done
• Freeze Custom runner manifests and run Custom-runner Trials on process suites #2188 done
• Enclose Custom runner Trials in a multi-provider sandbox with Trial provider keys, web-search wrappers, and panel family rules #2189 done
• Support Custom-runner Arms in Console Experiment start, Trial Peek, and blind pairwise review #2190 done
• Record Custom runner sessions into OTLP traces and enforce accounting_gap and web-search Void checks #2191 landed with Record Custom runner sessions and enforce Trial accounting #2227
• Add thirdshift evals check-runner, its Console API twin, and Custom-runner documentation #2192 landed with Add evals check-runner, its Console API twin, and Custom runner documentation #2239

Summary

How a Custom-runner Trial runs:

experiment --baseline-runner A | --candidate-runner B --suite process
  evals::arms  prepare-arms
    freeze manifest + files (bundle / HOME-relative) by SHA-256
    record Git origin, HEAD, dirty (dirty stays allowed); pin Harness path + version
    Setup format 13: commit/harness null, runner + runner_digest
  per Trial
    verify every frozen hash
    append git_exclude to .git/info/exclude
    systemd-run memory scope
      one bwrap sandbox (sandbox/custom.rs)
        fresh HOME: declared logins only, [providers] Trial keys, Grok lock, OpenCode config
        HTTPS_PROXY allows only declared provider hosts; counts connections
        PATH wrappers add each Harness's no-web-search switch; count runs
        runner request.json (v1)
          record_command --role ... streams sessions live (85% quota stop)
          commit on HEAD, write result.json (v1), optional sessions.json
    classify: outer supervision > protocol failure (failed) > result.json
    collect sessions into OTLP .logs/.spans, priced from frozen prices.json
    Void for web search, usage_limit or accounting_gap
  grade committed HEAD -> code Graders, Pair evidence, panel (declared families), blind pairwise
evals check-runner <name|file.toml> [--suite --task]   static checks, or one dry-run Trial
Console: /api/v1/evals/check-runner; /api/v1/start {baseline_runner, candidate_runner}

Where the code lives:

src/evals/arms/runner.rs                    manifest, freezing, provenance, Harness pinning
src/sandbox/custom.rs                       Trial-wide sandbox, logins, provider keys, wrappers
src/proxy.rs                                provider catalogue and Accounting
src/evals/experiment/execution/custom*      request/result, record_command, sessions.json -> OTLP, Void checks
src/evals/check_runner.rs, dry_run.rs       evals check-runner
src/console/{evals,start/experiment,experiments/setup}.rs   Console twins and Arm identity

Changes from the final review, on top of the Tickets:

 /api/v1/start experiment options
+  baseline_runner / candidate_runner: a .toml path is CLI-only, as in check-runner (ADR 0026 lists it)
 Experiment Command record
-  harness: "claude"            (a placeholder for a Custom-runner Candidate)
+  harness: "custom runner"
 pairwise blinding
-  substring replace, so a runner named "review" turned reviewer into [withheld]er
-  dropped every added table with a Role column
+  withholds whole words only; drops a role table only when it names the runner
 Custom runner HOME
+  copies Muse's auth.json.lock, as native Trials do

Tests sit at the Spec's agreed seams:

• thirdshift experiment … --json, evals prepare-arms and evals check-runner --json, driven through the Experiment fixture with fake runners, fake Harnesses and a loopback provider
• Console HTTP and browser tests for /api/v1/start, /api/v1/evals/check-runner, /experiments/<id>, Trial Peek and pairwise pages
• unit seams for custom::collect, pairwise blinding and the recorder

Evidence

Execution evidence

These are the review fixes. Each test was run red before its fix and green after.

• Before: cargo test --bin thirdshift console::start::experiment::tests

  called `Result::unwrap_err()` on an `Ok` value: ["--json", "--candidate-runner", "/tmp/outside.toml", "--suite", "process"]
  test result: FAILED. 2 passed; 1 failed
  

  After: the same command

  test result: ok. 3 passed; 0 failed
  

• Before: cargo test --test experiments custom_runner_experiment_command_record_names_no_fabricated_harness

  assertion `left != right` failed: {... "baseline":{"commit":null,"harness":null,...,"runner":"fixture",...}, ... "harness":"claude", ...}
  test result: FAILED. 0 passed; 1 failed
  

  After: the same command

  test result: ok. 1 passed; 0 failed
  

• Before: cargo test --bin thirdshift evals::experiment::pairwise::tests::runner_blinding

    left: "+fn [withheld]er() {}\n"
   right: "+| Role | Permission |\n+|---|---|\n+| admin | write |\n+fn reviewer() {}\n"
  test result: FAILED. 0 passed; 1 failed
  

  After: cargo test --bin thirdshift evals::experiment::pairwise

  test result: ok. 3 passed; 0 failed
  

• Console browser tests: node --test tests/experiments.test.cjs, run from console/, gives # pass 15, # fail 0. This includes the existing "Custom runner identity in retained pair evidence stays blind" and "preserves ordinary source evidence" cases.
• Site and manual: npm run docs:check --prefix site exits 0 after the manual was regenerated. The architecture-page test in site/tests/pages.test.cjs passes.

Wider runs, and their limits on this host:

• Integration targets: cargo test --no-fail-fast --test '*' gave 2677 passed and 27 failed. All 44 custom_runner:: and check_runner:: tests passed.
  • The 27 failures come from the host, not this branch.
  • Most fail with sandbox failed to start: socat is not on PATH. The proxy relay needs socat, which this machine lacks.
  • Two memory-cap tests, a_trial_over_its_cap_dies_alone_and_is_void_while_others_carry_on and review_a_cap_killed_tool_cannot_leave_a_successful_trial, are flaky. Run alone at the pre-review commit ef1ab4d, they failed once in four runs.
• scripts/affected-tests main: it stopped at cargo test after --bin thirdshift gave 1871 passed and 20 failed. All 20 are sandbox::tests that need socat.
• What did not run here: on hosts without working bwrap namespaces or socat, the Custom-runner sandbox tests now skip through one shared guard, so here they skipped. Their sandboxed paths need CI or a host with socat.

Visual evidence

Experiment page for an Experiment whose Arms are both Custom runners, with "What was compared" opened.

Desktop, 1440×900:

Mobile, 390×844:

On main, the Console cannot read a format-13 Setup with a null commit, so the page says "Experiment is unavailable." On this branch, the page shows each Custom runner's name, digest, Git provenance and declared roles.

Merge Danger

Door: two-way

The Experiment JSON format moves to 13, with nullable commit and harness on Setup. Older thirdshift builds cannot read Custom-runner Experiments. Native Experiments keep their identity, so they resume unchanged. Credentials gain an optional, Trials-only [providers] table.

Blast Radius: Experiments

Some changes reach native Experiments too:

• A web-search tool call in a native Trial's normalized transcript now voids that Trial (Ticket Record Custom runner sessions into OTLP traces and enforce accounting_gap and web-search Void checks #2191).
• Kimi Arms and seats resolved to Fireworks hosts count as the fireworks-ai family. Legacy Bars keep their original family classification when they resume.

Factory Commands are unaffected.

Unaddressed findings

Standards

• 1. Arm/Setup as parallel Options instead of a sum type (Primitive Obsession).
  • The behaviour error it caused is fixed: the Command record now names custom runner.
  • The type refactor is declined for this PR. Spec Spec: Custom-runner Arms for process Experiments #1468 fixes the serialized Setup as nullable commit/harness plus runner/runner_digest ("Setup makes commit and harness nullable"). An internal enum would change about 60 branches across 15 files at the end of a six-Ticket Spec. That is an Architecture review's job, not a review fix.
• 2. Execution::run mixes native and runner flows (shallow module). Declined for the same reason. execution/custom.rs already isolates the runner contract. Folding the recorder, sandbox and classification into one interface is a refactor with no behaviour change.
• 3, in part. A shared per-Harness login description. The drift is fixed with one MUSE_LOGINS list. Merging the native and runner login tables is declined, because they differ on purpose: native Trials bind user rules and settings files, while runner homes require only declared logins (src/sandbox/custom.rs login).
• 4. proxy::Provider and Kimi host resolution duplication; String families. Declined as a refactor. Families stay strings because the frozen families and provider_family fields serialize them as strings (docs/experiment-results.md).
• 7, in part. Declined as minor test duplication:
  • the twice-repeated Process early return in model_grader.rs
  • the repeated argv in custom_runner.rs
  • the t.after cleanups in console/tests/experiments.test.cjs
• 13. Other test duplication. Declined: Kimi env near copies, Void-assertion loops, the suite.toml literal, the unused runner() return value, and the sessions/tests.rs table row. None changes what the tests prove.
• 14, in part. The nine sessions/tests.rs invalid-evidence cases check only is_err(). Declined: each case builds one specific invalid input, and its name says which. The refusal and conflict assertions in custom_runner.rs now check their exact messages.

Spec

• 1. A crashed or timed-out runner with unrecorded provider traffic stays a $0 failed Trial.
  • Declined, because the Spec must decide. Filed Crashed Custom runner Trials with unrecorded provider traffic count as $0 failed Trials #2241.
  • The review's test fails as described: crashed_runner_after_unrecorded_provider_traffic_is_an_accounting_gap ... FAILED.
  • But the Spec says both "Outer supervision (… timeout -> failed) keeps precedence" and that runner crashes "without outer Void evidence record that Trial as failed". The code (src/evals/experiment/execution/custom.rs collect) and its test timed_out_runner_after_provider_traffic_stays_failed follow that precedence over the accounting_gap rule.
• 3. The web-search Void applies to native Trials. Declined. Ticket Record Custom runner sessions into OTLP traces and enforce accounting_gap and web-search Void checks #2191 says to "void any Trial whose normalized session transcripts contain a web-search tool call", and User Story 15 and ADR 0032 give native and Custom-runner Trials the same web-search boundary. docs/trial.md documents it.
• 4. Native Kimi family reclassification. Declined.
  • docs/experiment-results.md defines a seat's family as its provider ("the Harness's own, the provider an OpenCode Model names, or the resolved provider of a Kimi alias"). OpenCode Models already counted by serving provider.
  • Legacy Bars keep their classification on resume, through the guard in src/evals/experiment/panel.rs check_stand_in.
• 6. A quota warning in an imported sessions.json stream voids a finished Trial. Declined.
  • The review's test fails as described, and that failure is the documented behaviour.
  • Native Trials make "a live Claude or Codex quota window at 85%" void with usage_limit (docs/trial.md:215-216). Imported streams "also retain quota stops" (docs/eval-tasks.md:563, test raw_harness_sessions_retain_normalized_web_search_and_quota_evidence).
  • So both Arms treat a Trial that crossed 85% the same way, whether or not it was stopped live.

Security

None. The security review found three boundary breaks this change introduced, and fixed each one. A proof-of-concept test for each runs a runner inside real bwrap. Each test failed before its fix, passes after it, and stays as a regression test in tests/experiments/custom_runner.rs.

• Host writes through links in the attempt directory (485bafb). The sandbox mounted the whole attempt directory read-write, so a runner could leave links in it. The host followed those links when it wrote its own records there after the runner ended. Only the repository, fresh home and output are writable now. Test: host_attempt_writes_do_not_follow_links_planted_by_the_custom_runner.
• Host runs commands from the runner's Git config (56bd0b2). The host ran git status in the runner's repository, so a core.fsmonitor or clean filter the runner set there ran on the host. Status now reads the working tree through a host-owned Git directory. Test: host_git_in_the_runner_repository_ignores_runner_configured_commands.
• Host blocks on a FIFO result.json (c0580f7). A FIFO stalled the Experiment beyond the Trial timeout. result.json now goes through the same no-follow, nonblocking, regular-file reader as sessions.json, capped at 1 MiB. Test: a_fifo_result_fails_its_trial_instead_of_holding_the_host.

Review coverage

• Read in full or by changed hunk: every changed file from the review's diff, by both reviewers.
• Excluded as generated output: site/docs/index.html. Its source, docs/guide.md, was reviewed. npm run docs:check --prefix site exits 0, and the output was spot-checked for rendering and escaping.
• Changed by the review fixes, after the reviewers read the diff:
  • src/logs.rs and src/logs/recording.rs: reviewers read them only as supporting files, before the fix.
  • docs/adr/0026-each-console-serves-an-api-that-can-do-what-the-cli-can.md: the Standards reviewer read it before the one-line addition.

CI notes

Two failures in run 43682288379 come from tests this branch does not touch:

• console-test, Working identifies live Review and Fix targets locally and across the Fleet (console/tests/working-targets.test.cjs:83, 1440px remote review-explicit: visible PR target): intermittent. The same assertion failed on main in run 43466605947 (4d24f79d) and on issue-2098 in run 43293144147. The branch changes neither this test nor the Working or Fleet rendering, and the test passes 3 of 3 times locally.
• rust-x86 / run (4), release::native_release_waits_for_leak_decisions_before_any_push (tests/experiments/release.rs:218, Experiment is already running or being rechecked): intermittent lock race. The test dismisses a leak as soon as experiments --json shows waiting_on_leaks. But execute in src/evals/experiment.rs saves that Verdict and keeps holding the Experiment lock while it runs retention::repositories and print_results, so the dismissal's try_lock sometimes loses. The branch leaves that code and this test unchanged from main, and the test passes 2 of 2 times locally. test and rust-x86 / complete failed only as aggregates of these jobs.

Built with claude 2.1.322 · claude-opus-5-5 · high

Security review outcome

No unaddressed introduced findings.