Add evals check-runner, its Console API twin, and Custom runner documentation

docs/code-navigation.md

@@ -48,6 +48,7 @@ Start with the issue's terms in `GLOSSARY.md`, then use this index to find their
 | Trial pricing, role accounting, paired cost and time | `src/trial/accounting.rs` — dated token prices and per-session accounting; `src/trial/prices.json` — bundled snapshot; `src/evals/experiment/accounting.rs` — session capture, Step attribution and paired median reporting; `src/evals/experiment/execution.rs` and `src/evals/experiment/trial.rs` — accounting-gap Void classification, including earlier attempts; `src/evals/experiment/execution/custom.rs` — Custom runner session collection and accounting-gap and web-search Void checks, with `custom/recording.rs` (live `record_command` recorder) and `custom/sessions.rs` (`output_dir/sessions.json` import into OTLP traces) | `tests/experiments/accounting.rs`, `tests/experiments/custom_runner.rs`, `src/evals/experiment/execution/custom/sessions/tests.rs` |
 | Process Trial leak check and human decisions | `src/evals/experiment/leaks.rs` — Session evidence matching, immutable hit evidence, reviewer decisions and effective Void overlay; `src/console/experiments.rs` — shared decision adapter | `tests/experiments/process.rs`, `tests/console/experiments.rs`, `console/tests/experiments.test.cjs` |
 | Trial execution, mechanical Score, execution evidence lifetime | `src/evals/experiment/execution.rs` — fixed Harness directories and one scoped execution requesting a complete Trial repository before Harness instruction fallback, review integrity, named Built-in process execution, supervision and mechanical grading; private Trace and active evidence remain alive through the completion callback | `src/evals/experiment/execution/tests.rs`; command regressions: `tests/experiments.rs`, `tests/experiments/process.rs`, `tests/experiments/code_graders.rs`, `tests/experiments/sandbox.rs` |
+| Custom runner, `evals check-runner`, dry run | `src/evals/arms/runner.rs` — `Runner::freeze_reporting`: manifest parsing, file freezing and Harness pinning, each problem reported under its `Check`; `src/sandbox/custom.rs` — the Custom runner sandbox and `missing_logins`, the same login and `[providers]` key checks in a throwaway home; `src/evals/check_runner.rs` — static checks, priced roles and the version-1 report; `src/evals/experiment/dry_run.rs` — one Trial through `Execution` in a temporary directory outside Experiment storage; `src/console/evals.rs` — `check_runner`, the API twin run as a CLI subprocess | `tests/experiments/check_runner.rs`, `tests/experiments/custom_runner.rs`, `tests/console/start.rs`; contract: `docs/eval-tasks.md` |
 | Trial results, retained history, terminal evidence, Model Score targets | `src/evals/experiment/trial_results.rs` — Trial identity and compatible filenames, shared terminal eligibility and decoding, retained captures and pending accounting, opaque Model Score application and recheck overlays; `src/evals/experiment/trial_results/files.rs` — private filesystem seam shared with retained Session discovery | `src/evals/experiment/trial_results/tests.rs`, `src/evals/experiment/reading/tests.rs`, `tests/experiments.rs`, `tests/experiments/model_graders.rs`, `tests/experiments/rechecks.rs` |
 | Console Experiments, Trial Peek, Trial Session evidence | `src/console/experiments.rs` and `src/console/experiments/render.rs` — list and live Experiment page; `src/evals/experiment/reading.rs` — one captured CLI/Console view of Trial progress, results and accounting; `src/evals/experiment/session_evidence.rs` — shared confined Session selection and content reads, independent of Trial metadata, usage and Verdict upgrades; `src/evals/experiment/store.rs` — shared retained Experiment metadata validation; `src/evals/experiment/evidence.rs` — active Trial publication | `src/evals/experiment/session_evidence/tests.rs`, `src/evals/experiment/reading/tests.rs`, `tests/console/experiments.rs`, `console/tests/experiments.test.cjs` |
 | Plan-limit warnings | `src/harness/interpretation/limits.rs` — Claude/Codex quota detection; `src/harness/codex/limits.rs` — live native reports; `src/session/limits.rs` and `src/session/recording.rs` — Eval stop policy and shared pause signal; `src/evals/experiment/execution/monitor.rs` — supervision of older Arm builds | `src/harness/interpretation_tests.rs`, `src/evals/experiment/execution/tests.rs`, `tests/trial.rs`, `tests/experiments.rs`, `tests/fresh_run.rs` |

docs/eval-tasks.md

@@ -573,6 +573,87 @@ These checks judge only an outcome the runner reports in a valid
 `result.json`: a timeout, a runner that cannot start, a crash or a missing or
 invalid `result.json` stays `failed`, as outer supervision keeps precedence.
 
+### Checking a Custom runner
+
+Check a manifest before an Experiment spends Trials on it:
+
+```sh
+thirdshift evals check-runner <name-or-file.toml> [--json]
+thirdshift evals check-runner <name-or-file.toml> --suite <suite> --task <task> [--json]
+```
+
+The static check freezes the manifest into a throwaway directory exactly as
+Arm preparation does, but reports every problem instead of stopping at the
+first. Each problem names its `check`:
+
+- `manifest`: the file cannot be resolved or parsed, has unknown fields, an
+  empty `command`, a NUL in an argument or a multi-line `git_exclude` pattern;
+- `files`: a mapped file or skill directory is missing or not a regular file or
+  directory, or its target is absolute, contains `..` or `.git`, or overlaps another;
+- `harnesses`: a declared Harness is unknown, is not on PATH, reports no
+  version, or needs Models (OpenCode, Kimi) that do not resolve to provider hosts;
+- `logins`: a declared Harness has no Trials login: Claude's
+  `claude.setup_token`, Codex, Muse or agy logins in HOME, the Trials Grok home,
+  OpenCode keys for each declared provider, or Kimi aliases and their keys;
+- `providers`: a declared direct-SDK provider has no key under `[providers]`
+  in the Credentials;
+- `roles`: a `roles` Model is missing from the embedded Trial price table.
+
+Logins are checked as a Trial would install them, in a private throwaway home;
+no secret is printed. The Grok check only looks for the Trials login, without
+waiting for a live Grok Trial's lock.
+
+`--suite` and `--task` (always together) add a dry run once every static check
+passes: one Candidate Trial of the runner on that approved or proposed `process`
+Eval task with a `[trial]` table. Review suites and other sessions are refused
+before any check runs. The Trial runs through the same execution as an
+Experiment's, inside the real Trial sandbox and memory scope: request.json,
+`git_exclude`, `record_command` recording and `sessions.json` conversion into
+OTLP traces, pricing from the embedded price table, `accounting_gap` and
+`web_search` Void checks, committed-HEAD checks and the task's offline code
+checks, with the default Bar's 30-minute timeout. It uses a private temporary
+directory, removed afterwards, and creates no Experiment record: nothing is
+written under `~/.thirdshift/evals/experiments`.
+
+`--json` prints a version-1 report:
+
+```json
+{
+  "version": 1,
+  "runner": "workflow",
+  "manifest": "/home/you/.thirdshift/runners/workflow.toml",
+  "digest": "…",
+  "ok": true,
+  "problems": [],
+  "trial": {
+    "suite": "process", "task": "example", "outcome": "delivered",
+    "void_cause": null, "cause": null, "passed": true,
+    "checks": [{"command": null, "passed": true, "reason": "Trial delivered: session completed"}],
+    "sessions": [{"role": "implement", "time_ms": 2000, "models": {"…": {}}, "cost_usd": 0.00006, "gap": null}],
+    "cost_usd": 0.00006
+  }
+}
+```
+
+`digest` is the `runner_digest` an Experiment's Setup would record; it is null
+when the manifest, files or Harnesses could not be frozen. `problems` lists
+`{"check", "message"}` objects. `trial` is null without a dry run or when a
+static check failed. `checks` holds the Trial's outcome check followed by the
+task's code checks; `passed` is its Score. Each Session gives its role, time,
+observed Models with token usage, priced cost and any accounting gap;
+`cost_usd` is null when a Session could not be priced. Without `--json` the
+same facts print as text. The command exits `0` when `ok` is true: every
+static check passed and, for a dry run, the Trial's Score passed. A failed
+check still prints its report and exits `1`. An unusable Eval task or
+argument prints only an error.
+
+The Console serves the same check at POST `/api/v1/evals/check-runner` with
+the `run` scope. Its JSON body takes `runner`, a Custom runner name from
+`~/.thirdshift/runners`, and optional `suite` and `task`; a `.toml` file path
+stays CLI-only, so the API cannot read a caller-chosen local file. It
+returns the CLI's `--json` report, whether or not `ok` is true, and refuses an
+unusable request with 422.
+
 Before its first Trial, an Experiment prepares both Arms together:
 
 ```sh

docs/experiment-results.md

@@ -27,6 +27,10 @@ Format 13 adds Custom runners: `Setup.commit` and `Setup.harness` are nullable,
 and `runner` plus `runner_digest` identify the manifest and frozen inputs.
 Prepared Custom-runner Arms have null `commit` and `harness`, an empty `binary`
 path, and `runner` holding the frozen manifest, hashes and Git provenance.
+`thirdshift evals check-runner` reports that same `runner_digest` as `digest`.
+Its dry run creates no Experiment and none of these files; it prints its own
+version-1 report, described under
+[Checking a Custom runner](eval-tasks.md#checking-a-custom-runner).
 
 The private `experiment.json` already has `format: 1`. The CLI view projects its
 metadata and retained evidence; it is not a copy of that internal storage file.

docs/guide.md

@@ -848,6 +848,7 @@ JSON bodies use `Content-Type: application/json`. GET calls take no body.
 | GET `/evals` | `read` | none | `evals list` |
 | POST `/evals/touched` | `read` | JSON `paths` array of changed path names | `evals touched <paths>...` |
 | POST `/evals/verify` | `read` | JSON `repository` (a known Launch directory's owner/name), `base` commit | `evals verify <repository> <base>` |
+| POST `/evals/check-runner` | `run` | JSON `runner` (a Custom runner name; `.toml` file paths are CLI-only), optional `suite` and `task` together | `evals check-runner <runner> [--suite <suite> --task <task>] --json`; returns the report even when the check fails |
 | GET `/other-accounts` | `read` | none | `console other-accounts --json` |
 | POST `/other-accounts` | `admin` | JSON `value` (`none`, `readable`, or account list) | `console other-accounts <value>` |
 | POST `/email-test` | `admin` | JSON with optional `address` | `email-test [<address>]` |
@@ -885,6 +886,8 @@ requires `act`. POST `/api/v1/evals/import-replays` imports the machine's privat
 replay manifests and grading index from their standard locations, like the CLI.
 It accepts no paths. Eval task approval remains an `act` judgment. Eval verification
 uses a known Launch directory through POST `/api/v1/evals/verify` with `read`.
+A Custom runner check, and its dry-run Trial, runs as its own CLI process
+through POST `/api/v1/evals/check-runner` with `run`.
 Eval preparation and Arm preparation write caller-selected local paths, so remain
 CLI-only. Updates, imports and judgments name their requester
 in the Console log.
@@ -1326,6 +1329,65 @@ Harness sessions, or a declared provider/Harness runner delivering without
 sessions voids it as `accounting_gap`; a timed-out or crashed runner stays
 failed.
 
+A runner manifest at `~/.thirdshift/runners/<name>.toml` looks like this:
+
+```toml
+command = ["python3", "{runner_dir}/runner.py"]
+git_exclude = [".scratch/"]
+providers = ["openai"]
+harnesses = { claude = [], opencode = ["anthropic/claude"] }
+[files]
+"runner.py" = "runner.py"
+"skills/example" = { home = ".agents/skills/example" }
+[roles.implement]
+model = "claude-sonnet-4-6"
+effort = "high"
+```
+
+`command` is an argv array; `{runner_dir}` expands to the Trial's frozen
+bundle and the request.json path is appended. `files` maps sources beside the
+manifest to bundle targets, or `{ home = "…" }` targets in the fresh HOME.
+`roles` describes the configuration; observed Models decide pricing. Each
+Trial's request.json gives the prepared `repository`, `issue_file`,
+`issue_url`, `base`, `branch`, `runner_dir`, `output_dir` and `record_command`.
+The runner writes `output_dir/result.json`, such as
+`{"version":1,"outcome":"delivered"}`; `stopped_to_ask`, `failed` and `void`
+(with `void_cause` `session_unavailable`, `usage_limit` or `rate_limit`) are
+the other outcomes. Put direct-SDK keys in the Credentials, reserved for Trials:
+
+```toml
+[providers]
+openai = "…"
+```
+
+Check a runner before an Experiment spends Trials on it:
+
+```sh
+thirdshift evals check-runner workflow
+thirdshift evals check-runner workflow --suite process --task example --json
+```
+
+The static check reports every malformed manifest field, missing source file
+or skill directory, invalid or overlapping target path, unresolvable declared
+Harness or provider host, missing Harness login or `[providers]` key, and
+`roles` Model missing from the Trial price table, each under its `check` name.
+With `--suite` and `--task`, once the static checks pass, it runs one dry-run
+Trial on that process Eval task inside the real Trial sandbox: request.json,
+`git_exclude`, recorded or imported sessions, pricing, `accounting_gap` and
+`web_search` Void checks, and the task's offline code checks. It reports the
+outcome, Void cause, Score, code checks and each Session's cost, writes nothing
+under Experiment storage and creates no Experiment record. It exits `0` only
+when every check passes and the dry run's Score passes; a failed check still
+prints the report and exits `1`. `--json` prints the version-1 report described
+in [Checking a Custom runner](https://github.com/JacobStephens2/thirdshift/blob/main/docs/eval-tasks.md#checking-a-custom-runner).
+Console clients with the `run` scope get the same report from POST
+`/api/v1/evals/check-runner` with JSON `runner` (a Custom runner name; a
+`.toml` file path is CLI-only) and optional `suite` and `task`.
+
+In Experiment JSON (format 13), a Custom-runner Arm's Setup has null `commit`
+and `harness`, and names `runner` and its frozen `runner_digest`, the `digest`
+`evals check-runner` reports.
+
 Console clients can start the same Experiment with `POST /api/v1/start` (also
 served at `/api/v1/runs`), putting `baseline_runner`, `candidate_runner` and
 `suite` in the request's `experiment` object. The CLI's per-Arm conflicts and
@@ -3030,7 +3092,7 @@ no Resend API key. Either:
     (a crontab line, CI secret, or a shell profile the Ticket run's shell reads)
 ```
 
-Setup validates the Credentials before asking questions on a terminal. For email, `email-test`, the Console and a Run or a Pass that asks for a notification read them only when `RESEND_API_KEY` is unset or empty. A missing file just means no key from it. The file accepts quoted `resend.key`, `claude.setup_token` and `opencode.api_key` values, and an `[opencode.keys]` table of quoted keys by OpenCode provider name. Invalid TOML, unknown sections or keys (such as `kye`), and values of other types stop the command with exit `1`, naming the file and the offending key, and nothing is sent. An OpenCode Trial's provider key goes there by hand until `setup` asks for it: one per provider under `[opencode.keys]`, such as `fireworks-ai = "…"` and `baseten = "…"`, or a single `api_key` under `[opencode]` for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with `GROK_HOME=~/.thirdshift/trials/grok grok login`; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a `warning:` line on stderr saying to `chmod 600` it. Nothing is sent to check the key itself. When Resend accepts the email, it prints `accepted by Resend; check your inbox` and exits `0`; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits `1`. It gives up after 30 seconds without an answer.
+Setup validates the Credentials before asking questions on a terminal. For email, `email-test`, the Console and a Run or a Pass that asks for a notification read them only when `RESEND_API_KEY` is unset or empty. A missing file just means no key from it. The file accepts quoted `resend.key`, `claude.setup_token` and `opencode.api_key` values, an `[opencode.keys]` table of quoted keys by OpenCode provider name, and a `[providers]` table of quoted Trials-only direct-SDK keys for Custom runners (`anthropic`, `openai`, `xai`, `meta`, `google`, `fireworks-ai`, `baseten`, `kimi`). Invalid TOML, unknown sections or keys (such as `kye`), and values of other types stop the command with exit `1`, naming the file and the offending key, and nothing is sent. An OpenCode Trial's provider key goes there by hand until `setup` asks for it: one per provider under `[opencode.keys]`, such as `fireworks-ai = "…"` and `baseten = "…"`, or a single `api_key` under `[opencode]` for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with `GROK_HOME=~/.thirdshift/trials/grok grok login`; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a `warning:` line on stderr saying to `chmod 600` it. Nothing is sent to check the key itself. When Resend accepts the email, it prints `accepted by Resend; check your inbox` and exits `0`; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits `1`. It gives up after 30 seconds without an answer.
 
 #### Run notifications
 

docs/trial.md

@@ -263,3 +263,27 @@ killed work gets no further Resume, as in a Run.
 No Command record, Command log or Activity log is created. All Trial session
 artifacts stay in `output_dir`; the Inbox, Usual ranges and Silent therefore
 ignore them. An inherited Command owner is cleared before launching the Harness.
+
+## Custom runners
+
+A **Custom runner** Arm does not use `--trial`. Each of its Trials runs the
+runner's own argv inside one Trial-wide bwrap sandbox and memory scope, with a
+fresh HOME holding its mapped files, declared Harness logins and the
+Trials-only provider keys under `[providers]` in the Credentials. The runner
+receives its own version-1 request.json (`repository`, `issue_file`,
+`issue_url`, `base`, `branch`, `runner_dir`, `output_dir`, `record_command`),
+commits its change on HEAD and writes `output_dir/result.json`. Its sessions
+stream through `record_command --role <role> -- <harness> [args...]` or import
+from `output_dir/sessions.json` into the same OTLP Session traces native Trials
+leave, priced from the Experiment's frozen price table. The outcome and
+`void_cause` it may report, and the checks that override it (`accounting_gap`,
+`web_search`, committed `git_exclude` paths, outer supervision), are in the
+[Custom runner contract](eval-tasks.md#arm-preparation); its Setup in Experiment
+results format 13 has null `commit` and `harness` and a `runner` and
+`runner_digest` instead.
+
+`thirdshift evals check-runner <name-or-file.toml> --suite <suite> --task <task>`
+runs one such Trial on a process Eval task through the same execution, outside
+Experiment storage, and reports its outcome, Score, Session costs and any Void
+cause without creating an Experiment record; see
+[Checking a Custom runner](eval-tasks.md#checking-a-custom-runner).