[JacobStephens2]

Summary

thirdshift evals check-runner <name-or-file.toml> [--suite <suite> --task <task>] [--json] checks a Custom runner before an Experiment spends Trials on it. Its Console twin is POST /api/v1/evals/check-runner.

evals check-runner <runner>
  Runner::freeze_reporting           # Arm preparation's own freeze, collecting every problem
    manifest | files | harnesses     # parse, sources, targets, pinned Harness and provider hosts
  sandbox::custom::missing_logins    # same login and [providers] key installs, in a throwaway home
    logins | providers
  Prices::default()                  # roles Models must be priced
    roles
  if --suite/--task and no problems
    experiment::dry_run::run         # one Candidate Trial through Execution::run,
                                     # real sandbox, temporary directory, no Experiment record
  -> version-1 report: ok, problems[{check,message}], digest (= Setup runner_digest), trial
     exit 0 only when ok; a failed check still prints its report and exits 1

 Console PAGES
+  POST /api/v1/evals/check-runner   scope run
+    JSON runner (Custom runner name; .toml file paths stay CLI-only, ADR 0026), optional suite + task
+    -> runs `thirdshift evals check-runner … --json` as a subprocess (shared `cli_twin` helper)
+    -> returns the CLI report, ok or not; parity inventory + OpenAPI updated

The dry run reports the Trial's outcome, void_cause, cause, Score with each code check, and each Session's role, Models, cost and accounting gap. It exercises everything an Experiment Trial does:

• request.json and result.json
• git_exclude
• record_command and sessions.json conversion into OTLP traces
• pricing
• the accounting_gap and web_search Void checks
• the task's offline code checks

It writes nothing under ~/.thirdshift/evals/experiments.

The Custom runner contract is documented in docs/eval-tasks.md (new "Checking a Custom runner" section), docs/trial.md, docs/experiment-results.md and the user manual. That covers:

• the manifest
• the sandbox
• the request.json, result.json, sessions.json and record_command contract
• the format-13 Setup
• the --baseline-runner and --candidate-runner flags
• [providers] Trial keys in the Credentials
• evals check-runner and its API

The manual HTML is regenerated. The hand-written homepage, how-it-works and architecture pages mention the command.

Closes #2192

Evidence

Execution evidence

• Before (origin/issue-1468): thirdshift evals check-runner fixture --json

  thirdshift: 01:31:24 usage: thirdshift evals list | approve <suite> <task> [--json] | import-replays | propose-process <repository> [--limit <1..100>] | touched [<path>...] | prepare <suite> <task> <destination> | verify <repository> <base> | prepare-arms <repository> <baseline-commit> <baseline-harness> <candidate-commit> <candidate-harness> <record.json>
  

…full usage text…
exit=2

**After:** `cargo test --test experiments check_runner`

test check_runner::a_dry_run_confines_the_runner_in_the_trial_sandbox ... ok
test check_runner::roles_naming_unpriced_models_are_reported ... ok
test check_runner::missing_sources_and_escaping_home_targets_are_each_reported ... ok
test check_runner::a_dry_run_refuses_unusable_tasks_before_checking_the_runner ... ok
test check_runner::missing_harness_logins_and_provider_keys_are_reported_then_accepted ... ok
test check_runner::a_valid_runner_passes_with_the_digest_an_experiment_records ... ok
test check_runner::static_problems_skip_the_dry_run ... ok
test check_runner::malformed_manifests_are_refused_before_any_other_check ... ok
test check_runner::a_dry_run_reports_outcome_checks_and_session_cost_without_an_experiment ... ok
test check_runner::a_dry_run_reports_its_void_cause_and_fails_the_check ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 241 filtered out; finished in 6.09s

`cargo test --test console api_checks_a_custom_runner_with_the_cli_report`

test start::api_checks_a_custom_runner_with_the_cli_report ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 331 filtered out; finished in 0.63s

`npm test --prefix site`: `# tests 70` / `# pass 70` / `# fail 0`.
`scripts/affected-tests origin/issue-1468`:

PASS npm test --prefix site (112.16s)
PASS cargo build --locked (0.65s)
PASS npm test --prefix console (223.48s)
FAIL cargo test (236.53s)

`cargo test --no-fail-fast` then ran every target: 58 failures, every one in a sandbox- or scope-dependent test. This host has bwrap but no `socat`, so the sandbox fails to start (`sandbox failed to start: socat is not on PATH` / `sandbox_unavailable`), and its journal records no OOM kills. They fall into these groups:
- 20 `sandbox::tests` unit tests;
- the `tests/trial.rs` sandboxed Trial tests;
- the sandboxed Grader and Scoring tests;
- 5 sandboxed `custom_runner` tests;
- 3 sandboxed `tests/trial/process.rs` tests;
- `review_a_cap_killed_tool_cannot_leave_a_successful_trial`.

A build of `origin/issue-1468` fails the same way:
- `sandbox::tests::a_disallowed_host_is_refused_through_the_proxy` fails with `socat is not on PATH`.
- `a_code_review_trial_reviews_the_prepared_diff_and_keeps_both_reports` fails.
- `custom_runner::declared_harness_wrappers_apply_web_restrictions_and_preserve_process_io` fails.
- `review_a_cap_killed_tool_cannot_leave_a_successful_trial` is flaky there too: it failed 2 of 5 runs on the base and 1 of 3 on this branch.

CI, which has socat, is where these get checked.
`a_dry_run_confines_the_runner_in_the_trial_sandbox` skips without socat on PATH, as on this host. The sandboxed path is otherwise exercised through the shared `Execution` and by `custom_runner_sandbox_confines_children_credentials_network_and_harness_io`.

### Visual evidence

Homepage › Experiments workflow, desktop 1440×900. The Custom runner paragraph now ends with the `evals check-runner` sentence.

![Before](https://gist.githubusercontent.com/JacobStephens2/90be6c35ab1dbd5bb0c65b4984b6ca66/raw/9bc2e143937b1d44099cce56c7fd46fb308ff8ab/before-desktop.kdD26b.png)
![After](https://gist.githubusercontent.com/JacobStephens2/90be6c35ab1dbd5bb0c65b4984b6ca66/raw/9bc2e143937b1d44099cce56c7fd46fb308ff8ab/after-desktop.K4DQEz.png)

## Review

The Standards and Spec reviews ran against `refs/remotes/origin/issue-1468`. Standards raised 7 findings (judgement calls); 6 are fixed in b05e5e5a. Spec raised 1 finding; its own test refutes it.

### Unaddressed findings

**Standards**

- *Finding 3: `freeze_reporting` returns a possibly partial `Runner`, which is passed on to the login and role checks.* Declined. The partial runner is deliberate: the static check reports every problem in one pass, so a missing source file must not hide a missing login. The two uses that need a complete freeze are guarded at `src/evals/check_runner.rs`:
- the digest: `Some(runner) if problems.is_empty() => Some(runner.digest()?)`
- the dry run: `(Some(runner), Some(task)) if problems.is_empty()`

The doc comment on `Runner::freeze_reporting` states the invariant, and the ordinary `Runner::freeze` still returns only complete runners.

**Spec**

- *Finding 1: an `accounting_gap` dry run reports `cost_usd: 0.0` instead of null.* Declined because the run refutes it. Its test, `cargo test --test experiments check_runner::a_dry_run_reports_its_void_cause_and_fails_the_check`, with the reviewer's assertions appended, failed only on `sessions.is_empty()`. The report showed `"cost_usd":null` beside a gap Session: `"sessions":[{"cost_usd":null,"gap":"accounting gap: 0 recorded sessions, …","role":"process",…}]`. The `cost_usd`-is-null assertion is kept in that test. A Trial with no Sessions at all, which therefore has nothing to price, still reports $0, and the text output now says `Sessions: none recorded`.

### Review coverage

Both reviewers read every changed file in the reviewed commits. They confirmed one exclusion: `site/docs/index.html`, generated from `docs/guide.md` by `site/generate-docs.mjs`. `npm run docs:check --prefix site` exits 0, and the site browser tests pass.

Two files changed after the review, while fixing its findings, and the review left them unread: `src/console/commands.rs` (the new `cli_twin` helper) and `docs/code-navigation.md` (the new Custom runner row).

<!-- thirdshift:security-review -->
### Security review outcome

No unaddressed introduced findings.
<!-- /thirdshift:security-review -->


## Merge Danger

**Door:** two-way

This adds a new CLI action and API route. The refactors of `Runner::freeze` and of the Custom runner sandbox's login installation keep their behaviour: the existing `tests/experiments/custom_runner.rs` suite passes.

**Blast Radius:** contained

Two existing paths could regress:

- Custom-runner Arm preparation and Trial login setup now go through the extracted `freeze_reporting`, `login` and `provider_key`.
- The `process check` API twin now runs through the shared `cli_twin` helper.

Experiments now also share the review-suite refusal and the Trial-input checks with the dry run.

Built with claude 2.1.322 · claude-opus-5-5 · high <!-- thirdshift:built-with -->