diff --git a/docs/code-navigation.md b/docs/code-navigation.md index 0b1d0a92..1fc693a4 100644 --- a/docs/code-navigation.md +++ b/docs/code-navigation.md @@ -48,6 +48,7 @@ Start with the issue's terms in `GLOSSARY.md`, then use this index to find their | Trial pricing, role accounting, paired cost and time | `src/trial/accounting.rs` — dated token prices and per-session accounting; `src/trial/prices.json` — bundled snapshot; `src/evals/experiment/accounting.rs` — session capture, Step attribution and paired median reporting; `src/evals/experiment/execution.rs` and `src/evals/experiment/trial.rs` — accounting-gap Void classification, including earlier attempts; `src/evals/experiment/execution/custom.rs` — Custom runner session collection and accounting-gap and web-search Void checks, with `custom/recording.rs` (live `record_command` recorder) and `custom/sessions.rs` (`output_dir/sessions.json` import into OTLP traces) | `tests/experiments/accounting.rs`, `tests/experiments/custom_runner.rs`, `src/evals/experiment/execution/custom/sessions/tests.rs` | | Process Trial leak check and human decisions | `src/evals/experiment/leaks.rs` — Session evidence matching, immutable hit evidence, reviewer decisions and effective Void overlay; `src/console/experiments.rs` — shared decision adapter | `tests/experiments/process.rs`, `tests/console/experiments.rs`, `console/tests/experiments.test.cjs` | | Trial execution, mechanical Score, execution evidence lifetime | `src/evals/experiment/execution.rs` — fixed Harness directories and one scoped execution requesting a complete Trial repository before Harness instruction fallback, review integrity, named Built-in process execution, supervision and mechanical grading; private Trace and active evidence remain alive through the completion callback | `src/evals/experiment/execution/tests.rs`; command regressions: `tests/experiments.rs`, `tests/experiments/process.rs`, `tests/experiments/code_graders.rs`, `tests/experiments/sandbox.rs` | +| Custom runner, `evals check-runner`, dry run | `src/evals/arms/runner.rs` — `Runner::freeze_reporting`: manifest parsing, file freezing and Harness pinning, each problem reported under its `Check`; `src/sandbox/custom.rs` — the Custom runner sandbox and `missing_logins`, the same login and `[providers]` key checks in a throwaway home; `src/evals/check_runner.rs` — static checks, priced roles and the version-1 report; `src/evals/experiment/dry_run.rs` — one Trial through `Execution` in a temporary directory outside Experiment storage; `src/console/evals.rs` — `check_runner`, the API twin run as a CLI subprocess | `tests/experiments/check_runner.rs`, `tests/experiments/custom_runner.rs`, `tests/console/start.rs`; contract: `docs/eval-tasks.md` | | Trial results, retained history, terminal evidence, Model Score targets | `src/evals/experiment/trial_results.rs` — Trial identity and compatible filenames, shared terminal eligibility and decoding, retained captures and pending accounting, opaque Model Score application and recheck overlays; `src/evals/experiment/trial_results/files.rs` — private filesystem seam shared with retained Session discovery | `src/evals/experiment/trial_results/tests.rs`, `src/evals/experiment/reading/tests.rs`, `tests/experiments.rs`, `tests/experiments/model_graders.rs`, `tests/experiments/rechecks.rs` | | Console Experiments, Trial Peek, Trial Session evidence | `src/console/experiments.rs` and `src/console/experiments/render.rs` — list and live Experiment page; `src/evals/experiment/reading.rs` — one captured CLI/Console view of Trial progress, results and accounting; `src/evals/experiment/session_evidence.rs` — shared confined Session selection and content reads, independent of Trial metadata, usage and Verdict upgrades; `src/evals/experiment/store.rs` — shared retained Experiment metadata validation; `src/evals/experiment/evidence.rs` — active Trial publication | `src/evals/experiment/session_evidence/tests.rs`, `src/evals/experiment/reading/tests.rs`, `tests/console/experiments.rs`, `console/tests/experiments.test.cjs` | | Plan-limit warnings | `src/harness/interpretation/limits.rs` — Claude/Codex quota detection; `src/harness/codex/limits.rs` — live native reports; `src/session/limits.rs` and `src/session/recording.rs` — Eval stop policy and shared pause signal; `src/evals/experiment/execution/monitor.rs` — supervision of older Arm builds | `src/harness/interpretation_tests.rs`, `src/evals/experiment/execution/tests.rs`, `tests/trial.rs`, `tests/experiments.rs`, `tests/fresh_run.rs` | diff --git a/docs/eval-tasks.md b/docs/eval-tasks.md index b8e6acaf..368a3766 100644 --- a/docs/eval-tasks.md +++ b/docs/eval-tasks.md @@ -573,6 +573,87 @@ These checks judge only an outcome the runner reports in a valid `result.json`: a timeout, a runner that cannot start, a crash or a missing or invalid `result.json` stays `failed`, as outer supervision keeps precedence. +### Checking a Custom runner + +Check a manifest before an Experiment spends Trials on it: + +```sh +thirdshift evals check-runner [--json] +thirdshift evals check-runner --suite --task [--json] +``` + +The static check freezes the manifest into a throwaway directory exactly as +Arm preparation does, but reports every problem instead of stopping at the +first. Each problem names its `check`: + +- `manifest`: the file cannot be resolved or parsed, has unknown fields, an + empty `command`, a NUL in an argument or a multi-line `git_exclude` pattern; +- `files`: a mapped file or skill directory is missing or not a regular file or + directory, or its target is absolute, contains `..` or `.git`, or overlaps another; +- `harnesses`: a declared Harness is unknown, is not on PATH, reports no + version, or needs Models (OpenCode, Kimi) that do not resolve to provider hosts; +- `logins`: a declared Harness has no Trials login: Claude's + `claude.setup_token`, Codex, Muse or agy logins in HOME, the Trials Grok home, + OpenCode keys for each declared provider, or Kimi aliases and their keys; +- `providers`: a declared direct-SDK provider has no key under `[providers]` + in the Credentials; +- `roles`: a `roles` Model is missing from the embedded Trial price table. + +Logins are checked as a Trial would install them, in a private throwaway home; +no secret is printed. The Grok check only looks for the Trials login, without +waiting for a live Grok Trial's lock. + +`--suite` and `--task` (always together) add a dry run once every static check +passes: one Candidate Trial of the runner on that approved or proposed `process` +Eval task with a `[trial]` table. Review suites and other sessions are refused +before any check runs. The Trial runs through the same execution as an +Experiment's, inside the real Trial sandbox and memory scope: request.json, +`git_exclude`, `record_command` recording and `sessions.json` conversion into +OTLP traces, pricing from the embedded price table, `accounting_gap` and +`web_search` Void checks, committed-HEAD checks and the task's offline code +checks, with the default Bar's 30-minute timeout. It uses a private temporary +directory, removed afterwards, and creates no Experiment record: nothing is +written under `~/.thirdshift/evals/experiments`. + +`--json` prints a version-1 report: + +```json +{ + "version": 1, + "runner": "workflow", + "manifest": "/home/you/.thirdshift/runners/workflow.toml", + "digest": "…", + "ok": true, + "problems": [], + "trial": { + "suite": "process", "task": "example", "outcome": "delivered", + "void_cause": null, "cause": null, "passed": true, + "checks": [{"command": null, "passed": true, "reason": "Trial delivered: session completed"}], + "sessions": [{"role": "implement", "time_ms": 2000, "models": {"…": {}}, "cost_usd": 0.00006, "gap": null}], + "cost_usd": 0.00006 + } +} +``` + +`digest` is the `runner_digest` an Experiment's Setup would record; it is null +when the manifest, files or Harnesses could not be frozen. `problems` lists +`{"check", "message"}` objects. `trial` is null without a dry run or when a +static check failed. `checks` holds the Trial's outcome check followed by the +task's code checks; `passed` is its Score. Each Session gives its role, time, +observed Models with token usage, priced cost and any accounting gap; +`cost_usd` is null when a Session could not be priced. Without `--json` the +same facts print as text. The command exits `0` when `ok` is true: every +static check passed and, for a dry run, the Trial's Score passed. A failed +check still prints its report and exits `1`. An unusable Eval task or +argument prints only an error. + +The Console serves the same check at POST `/api/v1/evals/check-runner` with +the `run` scope. Its JSON body takes `runner`, a Custom runner name from +`~/.thirdshift/runners`, and optional `suite` and `task`; a `.toml` file path +stays CLI-only, so the API cannot read a caller-chosen local file. It +returns the CLI's `--json` report, whether or not `ok` is true, and refuses an +unusable request with 422. + Before its first Trial, an Experiment prepares both Arms together: ```sh diff --git a/docs/experiment-results.md b/docs/experiment-results.md index d88df5db..0f303fd7 100644 --- a/docs/experiment-results.md +++ b/docs/experiment-results.md @@ -27,6 +27,10 @@ Format 13 adds Custom runners: `Setup.commit` and `Setup.harness` are nullable, and `runner` plus `runner_digest` identify the manifest and frozen inputs. Prepared Custom-runner Arms have null `commit` and `harness`, an empty `binary` path, and `runner` holding the frozen manifest, hashes and Git provenance. +`thirdshift evals check-runner` reports that same `runner_digest` as `digest`. +Its dry run creates no Experiment and none of these files; it prints its own +version-1 report, described under +[Checking a Custom runner](eval-tasks.md#checking-a-custom-runner). The private `experiment.json` already has `format: 1`. The CLI view projects its metadata and retained evidence; it is not a copy of that internal storage file. diff --git a/docs/guide.md b/docs/guide.md index 8fc66268..eec9127e 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -848,6 +848,7 @@ JSON bodies use `Content-Type: application/json`. GET calls take no body. | GET `/evals` | `read` | none | `evals list` | | POST `/evals/touched` | `read` | JSON `paths` array of changed path names | `evals touched ...` | | POST `/evals/verify` | `read` | JSON `repository` (a known Launch directory's owner/name), `base` commit | `evals verify ` | +| POST `/evals/check-runner` | `run` | JSON `runner` (a Custom runner name; `.toml` file paths are CLI-only), optional `suite` and `task` together | `evals check-runner [--suite --task ] --json`; returns the report even when the check fails | | GET `/other-accounts` | `read` | none | `console other-accounts --json` | | POST `/other-accounts` | `admin` | JSON `value` (`none`, `readable`, or account list) | `console other-accounts ` | | POST `/email-test` | `admin` | JSON with optional `address` | `email-test [
]` | @@ -885,6 +886,8 @@ requires `act`. POST `/api/v1/evals/import-replays` imports the machine's privat replay manifests and grading index from their standard locations, like the CLI. It accepts no paths. Eval task approval remains an `act` judgment. Eval verification uses a known Launch directory through POST `/api/v1/evals/verify` with `read`. +A Custom runner check, and its dry-run Trial, runs as its own CLI process +through POST `/api/v1/evals/check-runner` with `run`. Eval preparation and Arm preparation write caller-selected local paths, so remain CLI-only. Updates, imports and judgments name their requester in the Console log. @@ -1326,6 +1329,65 @@ Harness sessions, or a declared provider/Harness runner delivering without sessions voids it as `accounting_gap`; a timed-out or crashed runner stays failed. +A runner manifest at `~/.thirdshift/runners/.toml` looks like this: + +```toml +command = ["python3", "{runner_dir}/runner.py"] +git_exclude = [".scratch/"] +providers = ["openai"] +harnesses = { claude = [], opencode = ["anthropic/claude"] } +[files] +"runner.py" = "runner.py" +"skills/example" = { home = ".agents/skills/example" } +[roles.implement] +model = "claude-sonnet-4-6" +effort = "high" +``` + +`command` is an argv array; `{runner_dir}` expands to the Trial's frozen +bundle and the request.json path is appended. `files` maps sources beside the +manifest to bundle targets, or `{ home = "…" }` targets in the fresh HOME. +`roles` describes the configuration; observed Models decide pricing. Each +Trial's request.json gives the prepared `repository`, `issue_file`, +`issue_url`, `base`, `branch`, `runner_dir`, `output_dir` and `record_command`. +The runner writes `output_dir/result.json`, such as +`{"version":1,"outcome":"delivered"}`; `stopped_to_ask`, `failed` and `void` +(with `void_cause` `session_unavailable`, `usage_limit` or `rate_limit`) are +the other outcomes. Put direct-SDK keys in the Credentials, reserved for Trials: + +```toml +[providers] +openai = "…" +``` + +Check a runner before an Experiment spends Trials on it: + +```sh +thirdshift evals check-runner workflow +thirdshift evals check-runner workflow --suite process --task example --json +``` + +The static check reports every malformed manifest field, missing source file +or skill directory, invalid or overlapping target path, unresolvable declared +Harness or provider host, missing Harness login or `[providers]` key, and +`roles` Model missing from the Trial price table, each under its `check` name. +With `--suite` and `--task`, once the static checks pass, it runs one dry-run +Trial on that process Eval task inside the real Trial sandbox: request.json, +`git_exclude`, recorded or imported sessions, pricing, `accounting_gap` and +`web_search` Void checks, and the task's offline code checks. It reports the +outcome, Void cause, Score, code checks and each Session's cost, writes nothing +under Experiment storage and creates no Experiment record. It exits `0` only +when every check passes and the dry run's Score passes; a failed check still +prints the report and exits `1`. `--json` prints the version-1 report described +in [Checking a Custom runner](https://github.com/JacobStephens2/thirdshift/blob/main/docs/eval-tasks.md#checking-a-custom-runner). +Console clients with the `run` scope get the same report from POST +`/api/v1/evals/check-runner` with JSON `runner` (a Custom runner name; a +`.toml` file path is CLI-only) and optional `suite` and `task`. + +In Experiment JSON (format 13), a Custom-runner Arm's Setup has null `commit` +and `harness`, and names `runner` and its frozen `runner_digest`, the `digest` +`evals check-runner` reports. + Console clients can start the same Experiment with `POST /api/v1/start` (also served at `/api/v1/runs`), putting `baseline_runner`, `candidate_runner` and `suite` in the request's `experiment` object. The CLI's per-Arm conflicts and @@ -3030,7 +3092,7 @@ no Resend API key. Either: (a crontab line, CI secret, or a shell profile the Ticket run's shell reads) ``` -Setup validates the Credentials before asking questions on a terminal. For email, `email-test`, the Console and a Run or a Pass that asks for a notification read them only when `RESEND_API_KEY` is unset or empty. A missing file just means no key from it. The file accepts quoted `resend.key`, `claude.setup_token` and `opencode.api_key` values, and an `[opencode.keys]` table of quoted keys by OpenCode provider name. Invalid TOML, unknown sections or keys (such as `kye`), and values of other types stop the command with exit `1`, naming the file and the offending key, and nothing is sent. An OpenCode Trial's provider key goes there by hand until `setup` asks for it: one per provider under `[opencode.keys]`, such as `fireworks-ai = "…"` and `baseten = "…"`, or a single `api_key` under `[opencode]` for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with `GROK_HOME=~/.thirdshift/trials/grok grok login`; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a `warning:` line on stderr saying to `chmod 600` it. Nothing is sent to check the key itself. When Resend accepts the email, it prints `accepted by Resend; check your inbox` and exits `0`; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits `1`. It gives up after 30 seconds without an answer. +Setup validates the Credentials before asking questions on a terminal. For email, `email-test`, the Console and a Run or a Pass that asks for a notification read them only when `RESEND_API_KEY` is unset or empty. A missing file just means no key from it. The file accepts quoted `resend.key`, `claude.setup_token` and `opencode.api_key` values, an `[opencode.keys]` table of quoted keys by OpenCode provider name, and a `[providers]` table of quoted Trials-only direct-SDK keys for Custom runners (`anthropic`, `openai`, `xai`, `meta`, `google`, `fireworks-ai`, `baseten`, `kimi`). Invalid TOML, unknown sections or keys (such as `kye`), and values of other types stop the command with exit `1`, naming the file and the offending key, and nothing is sent. An OpenCode Trial's provider key goes there by hand until `setup` asks for it: one per provider under `[opencode.keys]`, such as `fireworks-ai = "…"` and `baseten = "…"`, or a single `api_key` under `[opencode]` for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with `GROK_HOME=~/.thirdshift/trials/grok grok login`; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a `warning:` line on stderr saying to `chmod 600` it. Nothing is sent to check the key itself. When Resend accepts the email, it prints `accepted by Resend; check your inbox` and exits `0`; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits `1`. It gives up after 30 seconds without an answer. #### Run notifications diff --git a/docs/trial.md b/docs/trial.md index 7fc777da..51726354 100644 --- a/docs/trial.md +++ b/docs/trial.md @@ -263,3 +263,27 @@ killed work gets no further Resume, as in a Run. No Command record, Command log or Activity log is created. All Trial session artifacts stay in `output_dir`; the Inbox, Usual ranges and Silent therefore ignore them. An inherited Command owner is cleared before launching the Harness. + +## Custom runners + +A **Custom runner** Arm does not use `--trial`. Each of its Trials runs the +runner's own argv inside one Trial-wide bwrap sandbox and memory scope, with a +fresh HOME holding its mapped files, declared Harness logins and the +Trials-only provider keys under `[providers]` in the Credentials. The runner +receives its own version-1 request.json (`repository`, `issue_file`, +`issue_url`, `base`, `branch`, `runner_dir`, `output_dir`, `record_command`), +commits its change on HEAD and writes `output_dir/result.json`. Its sessions +stream through `record_command --role -- [args...]` or import +from `output_dir/sessions.json` into the same OTLP Session traces native Trials +leave, priced from the Experiment's frozen price table. The outcome and +`void_cause` it may report, and the checks that override it (`accounting_gap`, +`web_search`, committed `git_exclude` paths, outer supervision), are in the +[Custom runner contract](eval-tasks.md#arm-preparation); its Setup in Experiment +results format 13 has null `commit` and `harness` and a `runner` and +`runner_digest` instead. + +`thirdshift evals check-runner --suite --task ` +runs one such Trial on a process Eval task through the same execution, outside +Experiment storage, and reports its outcome, Score, Session costs and any Void +cause without creating an Experiment record; see +[Checking a Custom runner](eval-tasks.md#checking-a-custom-runner). diff --git a/site/architecture/index.html b/site/architecture/index.html index 32a7a09f..d6cebac3 100644 --- a/site/architecture/index.html +++ b/site/architecture/index.html @@ -257,7 +257,7 @@

Architecture

Experiments and Scores -

A Custom-runner Arm freezes an outside workflow's argv manifest, scripts and mapped HOME skill trees, recording their SHA-256 hashes and Git origin, HEAD and dirty state. --baseline-runner and --candidate-runner require explicit process suites; two Custom runners bypass thirdshift builds. Preparation pins declared Harness paths and versions. Setup identity uses the runner digest with nullable commit and Harness. Console start forwards runner selection through the CLI parser; its OpenAPI and Experiment JSON share format 13. Experiment comparisons and Trial Peek render frozen runner provenance and declared roles, and pairwise JSON puts runner details only in the post-preference revealed object. One Trial-wide bwrap sandbox and memory scope enclose the runner and its children. Fresh HOME holds mapped files and declared Harness logins; declared provider keys come from Trials-only Credentials. One provider-only HTTPS proxy and PATH wrappers disable Harness web search. Missing credentials void the Trial as session_unavailable, and declared provider families count in Grader panel voting and stand-in checks. Each unattended Trial receives request.json, commits HEAD and returns result.json. Code Graders and Pair evidence ignore uncommitted scratch; protocol failures and committed git_exclude paths fail one Trial. Custom runner sessions stream live through record_command --role or import from output_dir/sessions.json into standard OTLP traces, priced from frozen Experiment prices and available in Trial Peek and Waterfall. Quota warnings pause the Experiment; web-search tool calls and missing session accounting void the Trial.

+

A Custom-runner Arm freezes an outside workflow's argv manifest, scripts and mapped HOME skill trees, recording their SHA-256 hashes and Git origin, HEAD and dirty state. --baseline-runner and --candidate-runner require explicit process suites; two Custom runners bypass thirdshift builds. Preparation pins declared Harness paths and versions. Setup identity uses the runner digest with nullable commit and Harness. Console start forwards runner selection through the CLI parser; its OpenAPI and Experiment JSON share format 13. Experiment comparisons and Trial Peek render frozen runner provenance and declared roles, and pairwise JSON puts runner details only in the post-preference revealed object. One Trial-wide bwrap sandbox and memory scope enclose the runner and its children. Fresh HOME holds mapped files and declared Harness logins; declared provider keys come from Trials-only Credentials. One provider-only HTTPS proxy and PATH wrappers disable Harness web search. Missing credentials void the Trial as session_unavailable, and declared provider families count in Grader panel voting and stand-in checks. Each unattended Trial receives request.json, commits HEAD and returns result.json. Code Graders and Pair evidence ignore uncommitted scratch; protocol failures and committed git_exclude paths fail one Trial. Custom runner sessions stream live through record_command --role or import from output_dir/sessions.json into standard OTLP traces, priced from frozen Experiment prices and available in Trial Peek and Waterfall. Quota warnings pause the Experiment; web-search tool calls and missing session accounting void the Trial. evals check-runner freezes the manifest in report mode, collecting every problem, checks each declared login in a throwaway home and roles against the price table, and can run one Trial through the Experiment execution in a temporary directory outside Experiment storage; its Console twin runs that CLI as a subprocess.

Custom process and home-skill content hashes enter Arm identity and cost-estimate history. Changing either starts a new Experiment; unchanged inputs resume their frozen Experiment. Process grading shares one Bar deadline across snapshot preparation and all test selectors.

An Experiment or Scoring run stopped by a plan limit records a Paused Command. Needs you and Command details name its stopping Harness, window, utilization and reset time when reported, with a resume command to copy. Starting the same work clears that pause from Needs you, keeping finished Trials and Scores. Pauses count as neither success nor failure.

diff --git a/site/docs/index.html b/site/docs/index.html index d9a0c8fa..411494c1 100644 --- a/site/docs/index.html +++ b/site/docs/index.html @@ -913,6 +913,12 @@ evals verify <repository> <base> +POST /evals/check-runner +run +JSON runner (a Custom runner name; .toml file paths are CLI-only), optional suite and task together +evals check-runner <runner> [--suite <suite> --task <task>] --json; returns the report even when the check fails + + GET /other-accounts read none @@ -990,6 +996,8 @@ replay manifests and grading index from their standard locations, like the CLI. It accepts no paths. Eval task approval remains an act judgment. Eval verification uses a known Launch directory through POST /api/v1/evals/verify with read. +A Custom runner check, and its dry-run Trial, runs as its own CLI process +through POST /api/v1/evals/check-runner with run. Eval preparation and Arm preparation write caller-selected local paths, so remain CLI-only. Updates, imports and judgments name their requester in the Console log.

@@ -1405,6 +1413,54 @@ Harness sessions, or a declared provider/Harness runner delivering without sessions voids it as accounting_gap; a timed-out or crashed runner stays failed.

+

A runner manifest at ~/.thirdshift/runners/<name>.toml looks like this:

+
command = ["python3", "{runner_dir}/runner.py"]
+git_exclude = [".scratch/"]
+providers = ["openai"]
+harnesses = { claude = [], opencode = ["anthropic/claude"] }
+[files]
+"runner.py" = "runner.py"
+"skills/example" = { home = ".agents/skills/example" }
+[roles.implement]
+model = "claude-sonnet-4-6"
+effort = "high"
+
+

command is an argv array; {runner_dir} expands to the Trial's frozen +bundle and the request.json path is appended. files maps sources beside the +manifest to bundle targets, or { home = "…" } targets in the fresh HOME. +roles describes the configuration; observed Models decide pricing. Each +Trial's request.json gives the prepared repository, issue_file, +issue_url, base, branch, runner_dir, output_dir and record_command. +The runner writes output_dir/result.json, such as +{"version":1,"outcome":"delivered"}; stopped_to_ask, failed and void +(with void_cause session_unavailable, usage_limit or rate_limit) are +the other outcomes. Put direct-SDK keys in the Credentials, reserved for Trials:

+
[providers]
+openai = "…"
+
+

Check a runner before an Experiment spends Trials on it:

+
thirdshift evals check-runner workflow
+thirdshift evals check-runner workflow --suite process --task example --json
+
+

The static check reports every malformed manifest field, missing source file +or skill directory, invalid or overlapping target path, unresolvable declared +Harness or provider host, missing Harness login or [providers] key, and +roles Model missing from the Trial price table, each under its check name. +With --suite and --task, once the static checks pass, it runs one dry-run +Trial on that process Eval task inside the real Trial sandbox: request.json, +git_exclude, recorded or imported sessions, pricing, accounting_gap and +web_search Void checks, and the task's offline code checks. It reports the +outcome, Void cause, Score, code checks and each Session's cost, writes nothing +under Experiment storage and creates no Experiment record. It exits 0 only +when every check passes and the dry run's Score passes; a failed check still +prints the report and exits 1. --json prints the version-1 report described +in Checking a Custom runner. +Console clients with the run scope get the same report from POST +/api/v1/evals/check-runner with JSON runner (a Custom runner name; a +.toml file path is CLI-only) and optional suite and task.

+

In Experiment JSON (format 13), a Custom-runner Arm's Setup has null commit +and harness, and names runner and its frozen runner_digest, the digest +evals check-runner reports.

Console clients can start the same Experiment with POST /api/v1/start (also served at /api/v1/runs), putting baseline_runner, candidate_runner and suite in the request's experiment object. The CLI's per-Arm conflicts and @@ -2962,7 +3018,7 @@ - or set RESEND_API_KEY in the environment the Ticket run starts from (a crontab line, CI secret, or a shell profile the Ticket run's shell reads) -

Setup validates the Credentials before asking questions on a terminal. For email, email-test, the Console and a Run or a Pass that asks for a notification read them only when RESEND_API_KEY is unset or empty. A missing file just means no key from it. The file accepts quoted resend.key, claude.setup_token and opencode.api_key values, and an [opencode.keys] table of quoted keys by OpenCode provider name. Invalid TOML, unknown sections or keys (such as kye), and values of other types stop the command with exit 1, naming the file and the offending key, and nothing is sent. An OpenCode Trial's provider key goes there by hand until setup asks for it: one per provider under [opencode.keys], such as fireworks-ai = "…" and baseten = "…", or a single api_key under [opencode] for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with GROK_HOME=~/.thirdshift/trials/grok grok login; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a warning: line on stderr saying to chmod 600 it. Nothing is sent to check the key itself. When Resend accepts the email, it prints accepted by Resend; check your inbox and exits 0; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits 1. It gives up after 30 seconds without an answer.

+

Setup validates the Credentials before asking questions on a terminal. For email, email-test, the Console and a Run or a Pass that asks for a notification read them only when RESEND_API_KEY is unset or empty. A missing file just means no key from it. The file accepts quoted resend.key, claude.setup_token and opencode.api_key values, an [opencode.keys] table of quoted keys by OpenCode provider name, and a [providers] table of quoted Trials-only direct-SDK keys for Custom runners (anthropic, openai, xai, meta, google, fireworks-ai, baseten, kimi). Invalid TOML, unknown sections or keys (such as kye), and values of other types stop the command with exit 1, naming the file and the offending key, and nothing is sent. An OpenCode Trial's provider key goes there by hand until setup asks for it: one per provider under [opencode.keys], such as fireworks-ai = "…" and baseten = "…", or a single api_key under [opencode] for any provider without its own. Each OpenCode session takes the key for the provider its Model names, so two OpenCode sessions on different providers, such as two Graders in a Grader panel, each get their own; without one, an OpenCode Trial fails before its session starts. A Grok Trial signs its own Trials home in once, with GROK_HOME=~/.thirdshift/trials/grok grok login; without that login, a Grok Trial fails before its session starts. One that others can read is still used, with a warning: line on stderr saying to chmod 600 it. Nothing is sent to check the key itself. When Resend accepts the email, it prints accepted by Resend; check your inbox and exits 0; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits 1. It gives up after 30 seconds without an answer.

Run notifications

A Command that waited for primary GitHub quota includes one line, Waited <duration> for the GitHub quota (<resource>), totaling its waits and its child Ticket runs' waits. A Command that never waited has no such line; secondary-limit retries add none.

diff --git a/site/how-it-works/index.html b/site/how-it-works/index.html index bab9996c..ef6b698b 100644 --- a/site/how-it-works/index.html +++ b/site/how-it-works/index.html @@ -321,7 +321,7 @@

How thirdshift works

Each process Trial's Session traces are checked for the real fix's identifiers. Pending hits hold the Verdict for your decision in the Console or with thirdshift experiments <id> --leak <hit-id> confirm|dismiss --reason '…'. Dismissals keep the Trial and remain auditable; confirmation makes it Void. Decisions retain the reviewer, time and reason.

Optional --cost-ceiling and --time-ceiling fix Candidate/Baseline ratio limits in the Bar before Trials. Both default off; a paired median above either ceiling fails the Experiment. The Verdict also reports credibly cheaper or faster Candidates, or higher cost or time, alongside the unchanged quality decision.

Every Pass skips across repositories while an Experiment of the same OS account is live on this machine, including nightly Security and a Custom process with no issue; the Activity log names the Experiment. Manually targeted Commands still run and warn that they compete with its Trials. On start or resume, an Experiment waits for other live Commands before building Arms or starting Trials. Its log, status and the Console show the wait and the remaining Commands. Stop interrupts the wait cleanly. --no-wait starts immediately and records the competing Command count beside the Verdict. Existing Runs finish and other machines carry on.

-

Either Arm can name an outside workflow with --baseline-runner or --candidate-runner, requiring --suite on process suites. Candidate runner selection omits the candidate commit and defaults a native Baseline to HEAD; two Custom runners need no thirdshift checkout or build. Preparation freezes argv, mapped files and HOME skill trees, records dirty Git provenance and pins declared Harness binaries. One Trial-wide sandbox and memory scope enclose the runner and all its children. Fresh HOME holds mapped files and declared logins, with Trials-only declared provider keys, a provider-only HTTPS proxy and PATH wrappers disabling Harness web search. Missing credentials void the Trial as session_unavailable; declared provider families count in Grader panel voting and stand-in checks. The runner receives request.json, commits HEAD and writes result.json. Uncommitted files are diagnostics; missing results, crashes, invalid Void claims or committed git_exclude paths fail the Trial. Custom runner sessions stream live through record_command --role or import from output_dir/sessions.json into standard OTLP traces, priced from frozen Experiment prices and available in Trial Peek and Waterfall. Quota warnings pause the Experiment; web-search tool calls and missing session accounting void the Trial. Custom runner commands.

+

Either Arm can name an outside workflow with --baseline-runner or --candidate-runner, requiring --suite on process suites. Candidate runner selection omits the candidate commit and defaults a native Baseline to HEAD; two Custom runners need no thirdshift checkout or build. Preparation freezes argv, mapped files and HOME skill trees, records dirty Git provenance and pins declared Harness binaries. One Trial-wide sandbox and memory scope enclose the runner and all its children. Fresh HOME holds mapped files and declared logins, with Trials-only declared provider keys, a provider-only HTTPS proxy and PATH wrappers disabling Harness web search. Missing credentials void the Trial as session_unavailable; declared provider families count in Grader panel voting and stand-in checks. The runner receives request.json, commits HEAD and writes result.json. Uncommitted files are diagnostics; missing results, crashes, invalid Void claims or committed git_exclude paths fail the Trial. Custom runner sessions stream live through record_command --role or import from output_dir/sessions.json into standard OTLP traces, priced from frozen Experiment prices and available in Trial Peek and Waterfall. Quota warnings pause the Experiment; web-search tool calls and missing session accounting void the Trial. thirdshift evals check-runner reports every manifest, file, Harness, login, [providers] key and unpriced role problem before an Experiment; with --suite and --task it runs one dry-run Trial through the same sandboxed execution, outside Experiment storage, also at POST /api/v1/evals/check-runner. Custom runner commands.

Experiment JSON identifies its documented result contract with format: 13, adding nullable commit and Harness plus Custom runner name and digest. Console OpenAPI describes those Setups, and POST /api/v1/start accepts baseline_runner and candidate_runner with the CLI's validation. Experiment comparisons and Trial Peek show runner identity, Git HEAD and dirty state, and declared roles. Blind pairwise review reveals those details only after the preference is submitted. The Experiment results format covers retained Trial files, Suspended time, leak decisions, pair evidence, the Grader panel's calls, people's blind pairwise calls, Scores, recheck history, suite Verdicts and the Bar.

Review commits, local-origin pushes and repository reports do not void a Trial. Changing its issue text, declared inputs or commits under review does. Review Trials' mechanical checks grade a private copy of the prepared repository captured before the session.

The Console's Experiments list groups Running, Needs you and Done, newest first. Open one to watch Trials fill in, then see the Verdict by suite. The page compares Arms, folds the Bar and opens a Trial or Eval task in the Peek with Grader reasons and links to Session logs or Session traces. Details stays beside the Trials grid as you scroll on desktop and opens at the bottom on phones. Rechecks stay in the CLI, with copyable commands and a filter for their Eval tasks.

diff --git a/site/index.html b/site/index.html index 5ee5d6d9..a3aae0a6 100644 --- a/site/index.html +++ b/site/index.html @@ -511,7 +511,7 @@

Seven workflows

Experiments
-

A Custom runner compares an outside agent workflow on process Eval suites. Select either Arm with --baseline-runner <name-or-file.toml> or --candidate-runner <name-or-file.toml> and name --suite. Candidate runner selection omits the candidate commit; two Custom runners need no thirdshift checkout or build. Preparation freezes the manifest, scripts and mapped HOME skill trees by SHA-256, retaining Git provenance for dirty local edits, and pins declared Harness paths and versions. One Trial-wide sandbox encloses the runner and its children with fresh HOME, declared Harness logins and Trials-only provider keys, a provider-only proxy and PATH wrappers disabling Harness web search. Missing credentials void the Trial; declared provider families count in Grader panel voting. Code Graders score committed HEAD; stopping to ask or committing a git_exclude scratch path fails the Trial. Custom runner sessions stream live through record_command --role or import from output_dir/sessions.json into standard OTLP traces, priced from frozen Experiment prices and available in Trial Peek and Waterfall. Quota warnings pause the Experiment; web-search tool calls and missing session accounting void the Trial. Custom runner commands.

+

A Custom runner compares an outside agent workflow on process Eval suites. Select either Arm with --baseline-runner <name-or-file.toml> or --candidate-runner <name-or-file.toml> and name --suite. Candidate runner selection omits the candidate commit; two Custom runners need no thirdshift checkout or build. Preparation freezes the manifest, scripts and mapped HOME skill trees by SHA-256, retaining Git provenance for dirty local edits, and pins declared Harness paths and versions. One Trial-wide sandbox encloses the runner and its children with fresh HOME, declared Harness logins and Trials-only provider keys, a provider-only proxy and PATH wrappers disabling Harness web search. Missing credentials void the Trial; declared provider families count in Grader panel voting. Code Graders score committed HEAD; stopping to ask or committing a git_exclude scratch path fails the Trial. Custom runner sessions stream live through record_command --role or import from output_dir/sessions.json into standard OTLP traces, priced from frozen Experiment prices and available in Trial Peek and Waterfall. Quota warnings pause the Experiment; web-search tool calls and missing session accounting void the Trial. thirdshift evals check-runner <name-or-file.toml> checks a manifest, its files, Harness logins, [providers] keys and priced roles; add --suite and --task for one sandboxed dry-run Trial that creates no Experiment. The Console API serves it at POST /api/v1/evals/check-runner. Custom runner commands.

While an Experiment of the same OS account is live on this machine, every Pass skips across repositories, including nightly Security and an issue-less Custom process. Its Activity log names the Experiment. Manually targeted Commands still run with a warning. Experiments wait for other live Commands before building Arms or starting Trials; status and the Console name the wait. --no-wait bypasses it and records the competing Command count beside the Verdict. Existing Runs finish, and other machines carry on.

Custom process and home-skill content identifies each Arm. Editing either starts a new Experiment with separate estimate history; unchanged inputs resume their frozen Experiment. Process grading shares one Bar timeout across snapshot preparation and all test selectors.

After each process Trial, a leak check searches its Session traces for the real fix's identifiers. Review pending hits in the Console or with thirdshift experiments <id> --leak <hit-id> confirm|dismiss --reason '…'. The Verdict waits for your decisions. Dismissals keep the Trial on an auditable false-positive list; confirmation makes it Void. Each decision names its reviewer.

diff --git a/src/console.rs b/src/console.rs index 3cfca1af..9997bd25 100644 --- a/src/console.rs +++ b/src/console.rs @@ -208,6 +208,7 @@ enum View { Evals, TouchedEvals, VerifyEval, + CheckRunner, OtherAccounts, SetOtherAccounts, EmailTest, @@ -287,6 +288,7 @@ impl PageRoute { | View::Evals | View::TouchedEvals | View::VerifyEval + | View::CheckRunner | View::OtherAccounts | View::SetOtherAccounts | View::EmailTest @@ -409,6 +411,17 @@ const PAGES: &[PageRoute] = &[ method: Method::POST, view: View::VerifyEval, }, + PageRoute { + path: "/api/v1/evals/check-runner", + api: "/api/v1/evals/check-runner", + scope: Some(Scope::Run), + cli: &[ + "thirdshift evals check-runner fixture --json", + "thirdshift evals check-runner fixture --suite process --task fixture --json", + ], + method: Method::POST, + view: View::CheckRunner, + }, PageRoute { path: "/api/v1/other-accounts", api: "/api/v1/other-accounts", @@ -1113,6 +1126,7 @@ fn router(no_sign_in: bool, changes: changes::Receiver) -> Router { (View::Evals, Method::GET) => get(commands::evals), (View::TouchedEvals, Method::POST) => post(commands::touched), (View::VerifyEval, Method::POST) => post(commands::verify), + (View::CheckRunner, Method::POST) => post(evals::check_runner), (View::OtherAccounts, Method::GET) => get(commands::other_accounts), (View::SetOtherAccounts, Method::POST) => post(commands::set_other_accounts), (View::EmailTest, Method::POST) => post(commands::email_test), diff --git a/src/console/commands.rs b/src/console/commands.rs index e86dc4bd..60a4c3af 100644 --- a/src/console/commands.rs +++ b/src/console/commands.rs @@ -49,6 +49,33 @@ pub(super) async fn process(Path(name): Path) -> Response { .await } +/// Run a CLI twin as its own process, stopping it at `timeout`. Twins that +/// run work (a Process check, a dry-run Trial) keep their interruption +/// handling and progress out of the Console this way. +pub(super) fn cli_twin( + name: &str, + args: &[String], + directory: Option<&std::path::Path>, + timeout: std::time::Duration, +) -> Result { + let mut command = std::process::Command::new(std::env::current_exe()?); + command.args(args); + if let Some(directory) = directory { + command.current_dir(directory); + } + crate::process::output_until( + &mut command, + None, + crate::process::Control { + name, + interruption: crate::process::Interruption::Ordinary, + stop: &|child| crate::process::stop(child, &[libc::SIGTERM, libc::SIGKILL]), + }, + std::time::Instant::now() + timeout, + &|| Ok(()), + ) +} + pub(super) fn directory(repository: &str) -> Result { crate::launch_directories::load()? .directories diff --git a/src/console/evals.rs b/src/console/evals.rs index d80360ea..6cfc9a6d 100644 --- a/src/console/evals.rs +++ b/src/console/evals.rs @@ -1,4 +1,7 @@ //! Machine-local Eval data changes, distinct from people's act-scoped judgments. +use std::time::Duration; + +use anyhow::{bail, ensure}; use axum::{Extension, Json, response::Response}; use serde::Deserialize; @@ -46,3 +49,50 @@ pub(super) async fn import_replays( }) .await } + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct CheckRunner { + runner: String, + suite: Option, + task: Option, +} + +/// The CLI twin runs as its own process, as `process check` does, so a dry +/// run's Trial, interruption handling and progress stay out of the Console. +/// Its `--json` report is the response, whether or not the runner passed. +pub(super) async fn check_runner(Json(request): Json) -> Response { + api::action(api::Format::Json, "/", move || { + // A caller-selected local file stays CLI-only (ADR 0026): reading any + // `.toml` path would echo its text back in the manifest problem. + ensure!( + !request.runner.ends_with(".toml"), + "the API checks Custom runners by name; a .toml file path is CLI-only" + ); + let mut args = vec![ + "evals".to_owned(), + "check-runner".to_owned(), + request.runner, + ]; + for (flag, value) in [("--suite", request.suite), ("--task", request.task)] { + if let Some(value) = value { + args.extend([flag.to_owned(), value]); + } + } + args.push("--json".into()); + crate::evals::parse(&args[1..])?; + let timeout = crate::evals::experiment::Bar::default().timeout_seconds; + // The Trial and its code checks each have the Bar's timeout. + let output = super::commands::cli_twin( + "Custom runner check", + &args, + None, + Duration::from_secs(3 * timeout), + )?; + match serde_json::from_slice::(&output.stdout) { + Ok(report) if output.status.success() || output.status.code() == Some(1) => Ok(report), + _ => bail!("{}", String::from_utf8_lossy(&output.stderr).trim()), + } + }) + .await +} diff --git a/src/console/openapi.rs b/src/console/openapi.rs index 6be2f847..1651583f 100644 --- a/src/console/openapi.rs +++ b/src/console/openapi.rs @@ -123,6 +123,22 @@ fn operation(route: &PageRoute) -> (Vec, Value) { View::Evals => array(record), View::TouchedEvals => array(text), View::VerifyEval => object(json!({"safe":{"const":true}})), + View::CheckRunner => object(json!({ + "version": version, "runner": text, + "manifest": {"type": ["string", "null"]}, "digest": {"type": ["string", "null"]}, + "ok": {"type": "boolean"}, + "problems": array(object(json!({ + "check": {"enum": crate::evals::check_runner::check_names()}, + "message": {"type": "string"} + }))), + "trial": {"oneOf": [{"type": "null"}, object(json!({ + "suite": {"type": "string"}, "task": {"type": "string"}, + "outcome": {"type": "string"}, "void_cause": {"type": ["string", "null"]}, + "cause": {"type": ["string", "null"]}, "passed": {"type": "boolean"}, + "checks": array(record.clone()), "sessions": array(record.clone()), + "cost_usd": {"type": ["number", "null"]} + }))]} + })), View::OtherAccounts | View::SetOtherAccounts => { json!({"oneOf":[{"type":"string","enum":["none","readable"]},array(text)]}) } @@ -377,6 +393,19 @@ fn action_body(view: View) -> Value { vec!["repository", "base"], ); } + View::CheckRunner => { + let mut body = json_request( + json!({ + "runner": {"type":"string","description":"A Custom runner name in ~/.thirdshift/runners; a .toml file path is CLI-only."}, + "suite": {"type":"string","description":"With task: run one dry-run Trial on this process Eval task."}, + "task": text + }), + vec!["runner"], + ); + body["content"]["application/json"]["schema"]["dependentRequired"] = + json!({"suite":["task"],"task":["suite"]}); + return body; + } View::SetOtherAccounts => return json_request(json!({"value":text}), vec!["value"]), View::EmailTest => return json_request(json!({"address":text}), vec![]), View::RenameExperiment => return json_request(json!({"name":text}), vec!["name"]), diff --git a/src/console/parity.rs b/src/console/parity.rs index 5e9329e0..a90fbc51 100644 --- a/src/console/parity.rs +++ b/src/console/parity.rs @@ -88,6 +88,7 @@ const EVALS: &[(&str, &str)] = &[ "PrepareArms", "evals prepare-arms repository 0123456789012345678901234567890123456789 codex 0123456789012345678901234567890123456789 codex record.json", ), + ("CheckRunner", "evals check-runner fixture"), ( "PrepareSelectedArms", "evals prepare-arms repository record.json --baseline-runner old --candidate-runner new --suite process", diff --git a/src/console/start.rs b/src/console/start.rs index ca10bd08..76f6d6fd 100644 --- a/src/console/start.rs +++ b/src/console/start.rs @@ -474,18 +474,11 @@ pub(super) async fn check_process(Json(mut request): Json) -> Response let (repo, mut args) = request.arguments()?; args.insert(1, "check".into()); let directory = super::commands::directory(&repo.slug())?; - let mut command = Command::new(std::env::current_exe()?); - command.args(args).current_dir(directory.path); - let output = crate::process::output_until( - &mut command, - None, - crate::process::Control { - name: "Process check", - interruption: crate::process::Interruption::Ordinary, - stop: &|child| crate::process::stop(child, &[libc::SIGTERM, libc::SIGKILL]), - }, - Instant::now() + Duration::from_secs(30), - &|| Ok(()), + let output = super::commands::cli_twin( + "Process check", + &args, + Some(&directory.path), + Duration::from_secs(30), )?; ensure!( output.status.success(), diff --git a/src/evals.rs b/src/evals.rs index 9438b2d8..20df0a8f 100644 --- a/src/evals.rs +++ b/src/evals.rs @@ -2,12 +2,13 @@ use std::path::PathBuf; -use anyhow::{Result, bail}; +use anyhow::{Context, Result, bail}; pub(crate) mod approval; mod arms; pub(crate) use arms::Runner; pub(crate) mod bug_origin; +pub mod check_runner; pub(crate) mod experiment; mod format; mod import; @@ -46,6 +47,7 @@ pub enum Action { options: Box, record: PathBuf, }, + CheckRunner(check_runner::Request), } pub fn parse(args: &[String]) -> Result { @@ -93,6 +95,9 @@ pub fn parse(args: &[String]) -> Result { destination: destination.into(), }) } + [action, runner, options @ ..] if action == "check-runner" => { + check_runner_args(runner, options) + } [action, repository, base] if action == "verify" => { format::commit_id(base)?; Ok(Action::Verify { @@ -134,11 +139,44 @@ pub fn parse(args: &[String]) -> Result { }) } _ => bail!( - "usage: thirdshift evals list | approve [--json] | import-replays | propose-process [--limit <1..100>] | touched [...] | prepare | verify | prepare-arms " + "usage: thirdshift evals list | approve [--json] | import-replays | propose-process [--limit <1..100>] | touched [...] | prepare | verify | check-runner [--suite --task ] [--json] | prepare-arms " ), } } +fn check_runner_args(runner: &str, options: &[String]) -> Result { + let (mut suite, mut task, mut json) = (None, None, false); + let mut options = options.iter(); + while let Some(flag) = options.next() { + let slot = match flag.as_str() { + "--json" if !json => { + json = true; + continue; + } + "--suite" if suite.is_none() => &mut suite, + "--task" if task.is_none() => &mut task, + _ => bail!( + "usage: thirdshift evals check-runner [--suite --task ] [--json]" + ), + }; + let value = options + .next() + .with_context(|| format!("{flag} requires a value"))?; + format::name(value)?; + *slot = Some(value.clone()); + } + let task = match (suite, task) { + (Some(suite), Some(task)) => Some((suite, task)), + (None, None) => None, + _ => bail!("a dry run needs both --suite and --task"), + }; + Ok(Action::CheckRunner(check_runner::Request { + runner: runner.into(), + task, + json, + })) +} + pub fn execute(action: Action) -> Result { let root = crate::config::home()?.join(".thirdshift/evals/suites"); match action { @@ -170,6 +208,13 @@ pub fn execute(action: Action) -> Result { &destination, )?)?) } + // The CLI prints a failed report before exiting 1; callers of this + // function get the same report as the error instead. + Action::CheckRunner(request) => { + let (report, ok) = check_runner::execute(&request)?; + anyhow::ensure!(ok, "{report}"); + Ok(report) + } Action::Verify { repository, base } => { repository::verify(&repository, &base)?; Ok(r#"{"safe":true}"#.into()) diff --git a/src/evals/arms.rs b/src/evals/arms.rs index 8e6441a6..d47370b9 100644 --- a/src/evals/arms.rs +++ b/src/evals/arms.rs @@ -16,7 +16,7 @@ use crate::process::{self, Control, Interruption}; mod frozen; pub(crate) use frozen::Process; mod runner; -pub(crate) use runner::Runner; +pub(crate) use runner::{Check, Runner}; pub(crate) struct Requested { commit: String, @@ -122,6 +122,25 @@ pub(crate) struct Prepared { pub candidate: Arm, } +impl Prepared { + /// Both Arms run one already frozen Custom runner; nothing is built. + pub(crate) fn runners(runner: Runner) -> Self { + let arm = Arm { + commit: None, + binary: PathBuf::new(), + harness: None, + process: None, + runner: Some(runner), + families: None, + }; + Self { + format: 1, + baseline: arm.clone(), + candidate: arm, + } + } +} + pub(super) fn prepare(repository: &Path, arms: [Requested; 2], record: &Path) -> Result { if record.try_exists()? { bail!("Arm record already exists: {}", record.display()); diff --git a/src/evals/arms/runner.rs b/src/evals/arms/runner.rs index efd38478..e05b90d4 100644 --- a/src/evals/arms/runner.rs +++ b/src/evals/arms/runner.rs @@ -81,9 +81,53 @@ pub(crate) struct RunnerHarness { pub families: BTreeSet, } +/// The part of a Custom runner's inputs a static problem belongs to. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum Check { + Manifest, + Files, + Harnesses, + Logins, + Providers, + Roles, +} + +impl Check { + pub const ALL: [Self; 6] = [ + Self::Manifest, + Self::Files, + Self::Harnesses, + Self::Logins, + Self::Providers, + Self::Roles, + ]; + + pub fn name(self) -> &'static str { + match self { + Self::Manifest => "manifest", + Self::Files => "files", + Self::Harnesses => "harnesses", + Self::Logins => "logins", + Self::Providers => "providers", + Self::Roles => "roles", + } + } +} + +/// Where `freeze` sends each problem: the first one ends an ordinary freeze, +/// while `evals check-runner` collects them all. +pub(crate) type ProblemSink<'a> = dyn FnMut(Check, anyhow::Error) -> Result<()> + 'a; + impl Runner { pub fn freeze(name: &str, directory: &Path) -> Result { - let path = if name.ends_with(".toml") { + Self::freeze_reporting(name, directory, &mut |_, error| Err(error))? + .context("Custom runner manifest could not be read") + } + + /// Resolve a runner name or `.toml` path to its manifest file. + pub fn manifest_path(name: &str) -> Result { + if name.ends_with(".toml") { PathBuf::from(name) } else { crate::evals::format::name(name)?; @@ -92,81 +136,88 @@ impl Runner { .join(format!("{name}.toml")) } .canonicalize() - .with_context(|| format!("cannot resolve Custom runner {name}"))?; - let bytes = fs::read(&path)?; - let manifest: Manifest = toml::from_str(std::str::from_utf8(&bytes)?)?; - ensure!( - !manifest.command.is_empty() && !manifest.command[0].is_empty(), - "Custom runner command must contain a program" - ); - ensure!( - manifest.command.iter().all(|arg| !arg.contains('\0')), - "Custom runner command contains NUL" - ); - ensure!( - manifest - .git_exclude - .iter() - .all(|pattern| !pattern.contains(['\n', '\r', '\0'])), - "git_exclude patterns must each be one line" - ); + .with_context(|| format!("cannot resolve Custom runner {name}")) + } + + /// Freeze every input that passes, reporting each one that does not. + /// Returns `None` only when no manifest could be read; the Runner it does + /// return is complete only when nothing was reported. + pub fn freeze_reporting( + name: &str, + directory: &Path, + report: &mut ProblemSink<'_>, + ) -> Result> { + let loaded = (|| -> Result<_> { + let path = Self::manifest_path(name)?; + let bytes = fs::read(&path)?; + let manifest: Manifest = toml::from_str(std::str::from_utf8(&bytes)?)?; + ensure!( + !manifest.command.is_empty() && !manifest.command[0].is_empty(), + "Custom runner command must contain a program" + ); + ensure!( + manifest.command.iter().all(|arg| !arg.contains('\0')), + "Custom runner command contains NUL" + ); + ensure!( + manifest + .git_exclude + .iter() + .all(|pattern| !pattern.contains(['\n', '\r', '\0'])), + "git_exclude patterns must each be one line" + ); + Ok((path, bytes, manifest)) + })(); + let (path, bytes, manifest) = match loaded { + Ok(loaded) => loaded, + Err(error) => { + report(Check::Manifest, error)?; + return Ok(None); + } + }; fs::create_dir_all(directory.join("bundle"))?; fs::create_dir_all(directory.join("home"))?; fs::write(directory.join("manifest.toml"), &bytes)?; let mut provenance = BTreeMap::new(); let mut harnesses = BTreeMap::new(); for (name, models) in &manifest.harnesses { - let harness = Harness::named(name) - .with_context(|| format!("unknown Custom runner Harness {name}"))?; - ensure!( - !matches!(harness, Harness::OpenCode | Harness::Kimi) || !models.is_empty(), - "Custom runner {name} requires explicit Models for provider host resolution" - ); - let binary = super::fixed_harness(harness)?; - let mut hosts = BTreeSet::new(); - let mut families = BTreeSet::new(); - let models: Vec> = if models.is_empty() { - vec![None] - } else { - models.iter().map(|model| Some(model.as_str())).collect() - }; - for model in models { - let allowed = proxy::allowed_hosts(harness, model)?; - families.insert(proxy::harness_family(harness, model, &allowed)); - hosts.extend(allowed); + match pin(name, models) { + Ok(pinned) => { + harnesses.insert(name.clone(), pinned); + } + Err(error) => report(Check::Harnesses, error)?, } - harnesses.insert( - name.clone(), - RunnerHarness { - binary, - hosts, - families, - }, - ); } if let Some(origin) = source_provenance(&path)? { provenance.insert("manifest.toml".into(), origin); } + let parent = path.parent().context("runner manifest has no parent")?; for (source, target) in &manifest.files { - let source_path = path - .parent() - .context("runner manifest has no parent")? - .join(source) - .canonicalize()?; - let destination = directory.join(target.path()?); - copy(&source_path, &destination, 0)?; - if let Some(origin) = source_provenance(&source_path)? { - provenance.insert(source.clone(), origin); + let copied = (|| -> Result<_> { + let destination = directory.join(target.path()?); + let source_path = parent + .join(source) + .canonicalize() + .with_context(|| format!("cannot find Custom runner file {source}"))?; + copy(&source_path, &destination, 0)?; + source_provenance(&source_path) + })(); + match copied { + Ok(Some(origin)) => { + provenance.insert(source.clone(), origin); + } + Ok(None) => {} + Err(error) => report(Check::Files, error.context(format!("file {source}")))?, } } - Ok(Self { + Ok(Some(Self { name: name.into(), directory: directory.to_path_buf(), manifest, files: tree_hashes(directory)?, provenance, harnesses, - }) + })) } pub fn source_digest(name: &str) -> Result { @@ -224,6 +275,34 @@ impl Runner { } } +/// Pin one declared Harness and the provider hosts its Models reach. +fn pin(name: &str, models: &[String]) -> Result { + let harness = + Harness::named(name).with_context(|| format!("unknown Custom runner Harness {name}"))?; + ensure!( + !matches!(harness, Harness::OpenCode | Harness::Kimi) || !models.is_empty(), + "Custom runner {name} requires explicit Models for provider host resolution" + ); + let binary = super::fixed_harness(harness)?; + let mut hosts = BTreeSet::new(); + let mut families = BTreeSet::new(); + let models: Vec> = if models.is_empty() { + vec![None] + } else { + models.iter().map(|model| Some(model.as_str())).collect() + }; + for model in models { + let allowed = proxy::allowed_hosts(harness, model)?; + families.insert(proxy::harness_family(harness, model, &allowed)); + hosts.extend(allowed); + } + Ok(RunnerHarness { + binary, + hosts, + families, + }) +} + fn source_provenance(path: &Path) -> Result> { let directory = if path.is_dir() { path diff --git a/src/evals/check_runner.rs b/src/evals/check_runner.rs new file mode 100644 index 00000000..f62511e6 --- /dev/null +++ b/src/evals/check_runner.rs @@ -0,0 +1,179 @@ +//! `evals check-runner`: a Custom runner's static checks and optional dry run. + +use std::path::PathBuf; + +use anyhow::Result; +use serde::Serialize; + +use super::arms::{Check, Runner}; +use super::experiment::dry_run::{self, DryRun}; + +pub struct Request { + pub runner: String, + pub task: Option<(String, String)>, + pub json: bool, +} + +#[derive(Serialize)] +pub(crate) struct Report { + pub version: u32, + pub runner: String, + pub manifest: Option, + /// The `runner_digest` an Experiment's Setup would record; absent when + /// the runner could not be frozen. + pub digest: Option, + /// Every static check passed and, for a dry run, its Trial's Score passed. + pub ok: bool, + pub problems: Vec, + /// Absent without `--suite` and `--task`, or when a static check failed. + pub trial: Option, +} + +#[derive(Serialize)] +pub(crate) struct Problem { + pub check: Check, + pub message: String, +} + +/// Every `check` a problem can name, for the API's description. +pub(crate) fn check_names() -> [&'static str; 6] { + Check::ALL.map(Check::name) +} + +/// The report, rendered as `--json` or text, and whether it is `ok`. +pub fn execute(request: &Request) -> Result<(String, bool)> { + let report = check(request)?; + let text = if request.json { + serde_json::to_string_pretty(&report)? + } else { + render(&report) + }; + Ok((text, report.ok)) +} + +fn check(request: &Request) -> Result { + // Refuse an unusable Eval task before spending time on the runner. + let task = request + .task + .as_ref() + .map(|(suite, task)| dry_run::task(suite, task)) + .transpose()?; + let frozen = tempfile::Builder::new() + .prefix("thirdshift-check-runner-") + .tempdir()?; + let mut problems = Vec::new(); + let runner = Runner::freeze_reporting(&request.runner, frozen.path(), &mut |check, error| { + problems.push(Problem { + check, + message: format!("{error:#}"), + }); + Ok(()) + })?; + let digest = match &runner { + Some(runner) if problems.is_empty() => Some(runner.digest()?), + _ => None, + }; + if let Some(runner) = &runner { + let missing = crate::sandbox::custom::missing_logins(runner, &crate::config::home()?)?; + for (check, messages) in [ + (Check::Logins, missing.harnesses), + (Check::Providers, missing.providers), + ] { + problems.extend( + messages + .into_iter() + .map(|message| Problem { check, message }), + ); + } + let prices = crate::trial::accounting::Prices::default(); + for (name, role) in &runner.manifest.roles { + if !prices.models.contains_key(&role.model) { + problems.push(Problem { + check: Check::Roles, + message: format!( + "role {name} names Model {}, which the Trial price table ({}) does not price", + role.model, prices.date + ), + }); + } + } + } + let trial = match (runner, task) { + (Some(runner), Some(task)) if problems.is_empty() => Some(dry_run::run(&runner, &task)?), + _ => None, + }; + Ok(Report { + version: 1, + runner: request.runner.clone(), + manifest: Runner::manifest_path(&request.runner).ok(), + digest, + ok: problems.is_empty() && trial.as_ref().is_none_or(|trial| trial.passed), + problems, + trial, + }) +} + +fn render(report: &Report) -> String { + let mut lines = vec![match &report.manifest { + Some(path) => format!("Custom runner {} ({})", report.runner, path.display()), + None => format!("Custom runner {}", report.runner), + }]; + if let Some(digest) = &report.digest { + lines.push(format!("Digest: {digest}")); + } + if report.problems.is_empty() { + lines.push("Static checks: passed".into()); + } else { + lines.push("Static checks: failed".into()); + for problem in &report.problems { + lines.push(format!(" {}: {}", problem.check.name(), problem.message)); + } + } + if let Some(trial) = &report.trial { + lines.push(format!( + "Dry run {}/{}: {}{}; Score {}", + trial.suite, + trial.task, + trial.outcome, + trial + .void_cause + .map(|cause| format!(" ({cause})")) + .unwrap_or_default(), + if trial.passed { "passed" } else { "failed" } + )); + if let Some(cause) = &trial.cause { + lines.push(format!(" Cause: {cause}")); + } + for check in &trial.checks { + lines.push(format!( + " {} {}", + if check.passed { "passed" } else { "failed" }, + check.reason + )); + } + for session in &trial.sessions { + let models = session + .models + .keys() + .cloned() + .collect::>() + .join(", "); + lines.push(match (&session.cost_usd, &session.gap) { + (Some(cost), _) => format!(" Session {}: ${cost:.6} ({models})", session.role), + (None, Some(gap)) => format!(" Session {}: unpriced: {gap}", session.role), + (None, None) => format!(" Session {}: unpriced", session.role), + }); + } + lines.push(match (trial.sessions.is_empty(), trial.cost_usd) { + (true, _) => " Sessions: none recorded".into(), + (false, Some(cost)) => format!(" Cost: ${cost:.6}"), + (false, None) => " Cost: incomplete".into(), + }); + } + lines.push(if report.ok { + "Custom runner check passed.".into() + } else { + "Custom runner check failed.".into() + }); + lines.join("\n") +} diff --git a/src/evals/experiment.rs b/src/evals/experiment.rs index f6386bcf..3766c0f9 100644 --- a/src/evals/experiment.rs +++ b/src/evals/experiment.rs @@ -6,6 +6,7 @@ mod coordination; mod cost; pub(crate) mod deciders; mod difficulty; +pub(crate) mod dry_run; mod efficiency; mod evidence; mod execution; @@ -259,10 +260,7 @@ fn validate_runner_suites(options: &Options, root: &Path) -> Result<()> { return Ok(()); } for name in &options.suites { - ensure!( - !matches!(name.as_str(), "code-review" | "spec-review"), - "Custom runners require process suites: {name}" - ); + runner_suite(name)?; suite::Manifest::load(&root.join(name))?; for (_, directory) in suite::directories(&root.join(name).join("tasks"))? { ensure!( @@ -274,6 +272,26 @@ fn validate_runner_suites(options: &Options, root: &Path) -> Result<()> { Ok(()) } +/// Custom runners carry out a process; the review suites have none. +fn runner_suite(name: &str) -> Result<()> { + ensure!( + !matches!(name, "code-review" | "spec-review"), + "Custom runners require process suites: {name}" + ); + Ok(()) +} + +/// The Trial inputs every Trial needs before its repository is prepared. +/// A proposed Eval task has no pull request yet; an Experiment needs one. +fn trial_inputs(inputs: &crate::trial::Inputs) -> Result<()> { + crate::issue::IssueUrl::parse(&inputs.issue_url)?; + ensure!( + !inputs.base.is_empty() && !inputs.branch.is_empty(), + "Trial inputs must be nonempty" + ); + Ok(()) +} + fn process_setups(setups: [&mut Setup; 2]) { for setup in setups { if setup.runner.is_none() { @@ -809,11 +827,8 @@ fn tasks(root: &Path, suites: &[String]) -> Result> { missing.push(format!("{name}/{task_name}")); continue; }; - crate::issue::IssueUrl::parse(&inputs.issue_url)?; - ensure!( - !inputs.base.is_empty() && !inputs.branch.is_empty() && !inputs.pr_url.is_empty(), - "Trial inputs must be nonempty" - ); + trial_inputs(inputs)?; + ensure!(!inputs.pr_url.is_empty(), "Trial inputs must be nonempty"); tasks.push(Task { suite: name.clone(), name: task_name, diff --git a/src/evals/experiment/dry_run.rs b/src/evals/experiment/dry_run.rs new file mode 100644 index 00000000..df991348 --- /dev/null +++ b/src/evals/experiment/dry_run.rs @@ -0,0 +1,144 @@ +//! One throwaway Custom-runner Trial, outside Experiment storage. +//! +//! The Trial runs through the same execution as an Experiment's: the real +//! sandbox, Session evidence, pricing, Void checks and code checks. Nothing +//! is retained: its directory is removed when the dry run ends. + +use std::collections::BTreeMap; +use std::time::Duration; + +use anyhow::{Context, Result, ensure}; +use serde::Serialize; + +use super::{Arm, Bar, Setup, Task, arms, execution, format, suite, trial_results}; +use crate::sandbox::code_grader::Check; +use crate::session::usage::Measurement; +use crate::trial::{Outcome, VoidCause}; + +#[derive(Serialize)] +pub(crate) struct DryRun { + pub suite: String, + pub task: String, + pub outcome: Outcome, + pub void_cause: Option, + pub cause: Option, + /// The Trial's Score: its outcome check and the task's code checks. + pub passed: bool, + pub checks: Vec, + pub sessions: Vec, + /// Absent when any Session could not be priced. + pub cost_usd: Option, +} + +#[derive(Serialize)] +pub(crate) struct Session { + pub role: String, + pub time_ms: u128, + pub models: BTreeMap, + pub cost_usd: Option, + pub gap: Option, +} + +/// The process Eval task a dry run would use, refused before any check runs. +pub(crate) fn task(suite: &str, task: &str) -> Result { + super::runner_suite(suite)?; + let root = crate::config::home()?.join(".thirdshift/evals/suites"); + let directory = root.join(suite); + suite::Manifest::load(&directory)?; + let directory = directory.join("tasks").join(task); + let manifest = format::Task::load(&directory) + .with_context(|| format!("cannot load Eval task {suite}/{task}"))?; + ensure!( + manifest.session == format::Session::Process, + "Custom runners require process Eval tasks: {suite}/{task}" + ); + super::trial_inputs( + manifest + .trial + .as_ref() + .with_context(|| format!("Eval task {suite}/{task} needs a [trial] table"))?, + )?; + Ok(TaskInput(Task { + suite: suite.into(), + name: task.into(), + directory, + manifest, + })) +} + +pub(crate) struct TaskInput(Task); + +/// Run one Candidate Trial of `runner` on `task` in a private directory. +pub(crate) fn run(runner: &arms::Runner, task: &TaskInput) -> Result { + let task = &task.0; + crate::interrupt::install()?; + let directory = tempfile::Builder::new() + .prefix("thirdshift-dry-run-") + .tempdir()?; + let root = directory.path().canonicalize()?; + let prepared = arms::Prepared::runners(runner.clone()); + let setup = Setup { + commit: None, + harness: None, + model: None, + effort: None, + process: None, + process_digest: None, + runner: Some(runner.name.clone()), + runner_digest: Some(runner.digest()?), + }; + let signal = root.join("plan-limit"); + let prices = crate::trial::accounting::Prices::default(); + let execution = execution::Execution::prepare( + &root, + &prepared, + [&setup, &setup], + Duration::from_secs(Bar::default().timeout_seconds), + &signal, + )? + .with_pricing(Some(&prices)); + let trace = crate::logs::trace::Context::of_session(&root); + execution.run( + execution::Request { + task, + identity: trial_results::Identity { + suite: &task.suite, + task: &task.name, + arm: Arm::Candidate, + trial: 0, + }, + attempt: 0, + trace: &trace, + }, + |_, output, score| { + let sessions: Vec = output + .accounting + .map(|accounting| accounting.sessions) + .unwrap_or_default() + .into_iter() + .map(|session| Session { + role: session.role, + time_ms: session.time_ms, + models: session.models, + cost_usd: session.cost_usd, + gap: session.gap, + }) + .collect(); + let cost_usd = sessions + .iter() + .map(|session| session.cost_usd) + .sum::>(); + Ok(DryRun { + suite: task.suite.clone(), + task: task.name.clone(), + outcome: output.outcome, + void_cause: output.void_cause, + cause: output.cause, + passed: score.passed, + checks: score.checks, + sessions, + cost_usd, + }) + }, + ) +} diff --git a/src/main.rs b/src/main.rs index 5b8baf8a..7a8bb8b8 100644 --- a/src/main.rs +++ b/src/main.rs @@ -196,7 +196,7 @@ usage: thirdshift Run the factory on the iss thirdshift bug-origins [] [--json] Show origins; --confirm / --correct / --none settles one thirdshift email-test [
] Send a test email through Resend thirdshift setup Choose defaults and write the User config - thirdshift evals Manage Eval data and Arms, including import-replays; see the manual + thirdshift evals Manage Eval data and Arms, including import-replays and check-runner; see the manual thirdshift experiment [ | --candidate-runner ] [--baseline-runner ] [options] Compare Arms; Custom runners require --suite on process suites; --no-wait skips the host wait; --release uses the last tag thirdshift experiments [] [--json] List Experiments or show one; --name renames, --record-difficulty records A/A Difficulty, --leak decides Session evidence hits, --pair reads a pair blind and records your calls thirdshift recheck |--pending|--batch ... Review or record Score rechecks; see the manual @@ -279,6 +279,20 @@ fn main() -> ExitCode { }) => { return printed(scoring::show(pull_request.as_deref(), mine, json)); } + // A finished check prints its report even when the runner fails it. + Ok(Command::Evals(evals::Action::CheckRunner(request))) => { + return match evals::check_runner::execute(&request) { + Ok((report, ok)) => { + println!("{report}"); + if ok { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + } + } + Err(error) => failure(&error), + }; + } Ok(Command::Evals(action)) => return printed(evals::execute(action)), Ok(Command::Experiment(options)) => { return match evals::experiment::run(*options) { diff --git a/src/sandbox.rs b/src/sandbox.rs index 1f6af9cf..fc9575af 100644 --- a/src/sandbox.rs +++ b/src/sandbox.rs @@ -1154,15 +1154,7 @@ fn trials_home(home: &Path) -> PathBuf { fn grok_into(request: &Request<'_>) -> Result { let home = trials_home(request.home); let guard = GrokGuard::acquire(request.home)?; - if !home.join("auth.json").is_file() { - return Err(anyhow::Error::new(NoLogin { - diagnostic: format!( - "Grok Trial is not logged in; sign the Trials home in once with \ - GROK_HOME={} grok login", - home.display() - ), - })); - } + grok_login(request.home)?; let dest = request.fresh_home.join(GROK_HOME_NAME); reject_fresh_symlink(request.fresh_home, &dest)?; fs::create_dir_all(&dest).with_context(|| { @@ -1182,6 +1174,21 @@ fn grok_into(request: &Request<'_>) -> Result { Ok(guard) } +/// The Trials Grok home must hold a login before any Grok Trial starts. +fn grok_login(home: &Path) -> Result<()> { + let home = trials_home(home); + if !home.join("auth.json").is_file() { + return Err(anyhow::Error::new(NoLogin { + diagnostic: format!( + "Grok Trial is not logged in; sign the Trials home in once with \ + GROK_HOME={} grok login", + home.display() + ), + })); + } + Ok(()) +} + /// One Grok Trial at a time: the lock is a directory beside the Trials home, /// held until the guard drops. A lock whose owner is gone is stale and taken. struct GrokGuard { diff --git a/src/sandbox/custom.rs b/src/sandbox/custom.rs index 82e15c5e..02b362c4 100644 --- a/src/sandbox/custom.rs +++ b/src/sandbox/custom.rs @@ -59,37 +59,8 @@ pub(crate) fn prepare( secrets: Secrets::from(&credentials), adapter_env: vec![], }; - match harness { - Harness::Claude => { - let token = credentials - .claude_setup_token() - .ok_or_else(|| no_login("declared claude has no Trials setup token"))?; - env.set_str(CLAUDE_TOKEN_ENV, &token); - } - Harness::Codex => required_login( - &request, - &mut env, - Some("CODEX_HOME"), - ".codex", - "auth.json", - )?, - Harness::Muse => required_login( - &request, - &mut env, - Some("XDG_CONFIG_HOME"), - ".config", - "muse/auth.json", - )?, - Harness::Agy => required_login( - &request, - &mut env, - None, - ".gemini/antigravity-cli", - "antigravity-oauth-token", - )?, - Harness::Grok => grok = Some(grok_into(&request)?), - Harness::OpenCode => opencode(runner, &credentials, &mut env)?, - Harness::Kimi => kimi(runner, &request, &mut env, &declared.hosts)?, + if let Some(guard) = login(runner, &request, &credentials, &mut env, &declared.hosts)? { + grok = Some(guard); } for (key, value) in harness.adapter().environment() { env.set_str(key, value); @@ -99,10 +70,10 @@ pub(crate) fn prepare( binaries.push((name, harness, binary)); } for provider in &runner.manifest.providers { - let key = credentials.provider_key(*provider).ok_or_else(|| { - no_login(format!("missing [providers].{} Trial key", provider.name())) - })?; - env.set_str(provider.environment(), key); + env.set_str( + provider.environment(), + provider_key(&credentials, *provider)?, + ); allowed.extend(provider.hosts().iter().map(|host| host.to_string())); } // Preserve missing-login classification even on hosts that cannot start @@ -139,6 +110,114 @@ pub(crate) fn prepare( ) } +/// Install the login one declared Harness needs into the fresh home. +fn login( + runner: &Runner, + request: &Request<'_>, + credentials: &Credentials, + env: &mut Environment, + hosts: &BTreeSet, +) -> Result> { + match request.harness { + Harness::Claude => { + let token = credentials + .claude_setup_token() + .ok_or_else(|| no_login("declared claude has no Trials setup token"))?; + env.set_str(CLAUDE_TOKEN_ENV, &token); + } + Harness::Codex => { + required_login(request, env, Some("CODEX_HOME"), ".codex", "auth.json")?; + } + Harness::Muse => required_login( + request, + env, + Some("XDG_CONFIG_HOME"), + ".config", + "muse/auth.json", + )?, + Harness::Agy => required_login( + request, + env, + None, + ".gemini/antigravity-cli", + "antigravity-oauth-token", + )?, + Harness::Grok => return grok_into(request).map(Some), + Harness::OpenCode => opencode(runner, credentials, env)?, + Harness::Kimi => kimi(runner, request, env, hosts)?, + } + Ok(None) +} + +fn provider_key(credentials: &Credentials, provider: proxy::Provider) -> Result<&str> { + credentials + .provider_key(provider) + .ok_or_else(|| no_login(format!("missing [providers].{} Trial key", provider.name()))) +} + +/// What `prepare` would refuse for want of a login or a `[providers]` key, +/// each declaration checked on its own in a throwaway fresh home. Grok's +/// login is only looked for, without waiting on a live Grok Trial's lock. +pub(crate) fn missing_logins(runner: &Runner, home: &Path) -> Result { + let mut missing = MissingLogins::default(); + let credentials = match Credentials::read() { + Ok(credentials) => credentials, + Err(error) => { + missing.harnesses.push(format!("{error:#}")); + return Ok(missing); + } + }; + let scratch = tempfile::Builder::new() + .prefix("check-runner-") + .permissions(fs::Permissions::from_mode(0o700)) + .tempdir()?; + let fresh_home = scratch.path().join("home"); + fs::create_dir_all(&fresh_home)?; + for (name, declared) in &runner.harnesses { + let harness = Harness::named(name).context("unknown frozen Harness")?; + let request = Request { + harness, + model: None, + repository: scratch.path(), + skills_dir: scratch.path(), + harness_binary: &declared.binary.path, + home, + fresh_home: &fresh_home, + secrets: Secrets::from(&credentials), + adapter_env: vec![], + }; + let result = if harness == Harness::Grok { + grok_login(home) + } else { + let mut env = Environment::default(); + login(runner, &request, &credentials, &mut env, &declared.hosts).map(drop) + }; + if let Err(error) = result { + missing.harnesses.push(diagnostic(&error)); + } + } + for provider in &runner.manifest.providers { + if let Err(error) = provider_key(&credentials, *provider) { + missing.providers.push(diagnostic(&error)); + } + } + Ok(missing) +} + +/// A missing login's own words, without the Trial-time prefix. +fn diagnostic(error: &anyhow::Error) -> String { + match error.downcast_ref::() { + Some(missing) => missing.diagnostic.clone(), + None => format!("{error:#}"), + } +} + +#[derive(Default)] +pub(crate) struct MissingLogins { + pub harnesses: Vec, + pub providers: Vec, +} + fn no_login(message: impl Into) -> anyhow::Error { NoLogin { diagnostic: message.into(), diff --git a/tests/console/start.rs b/tests/console/start.rs index 783d9ce1..cde78131 100644 --- a/tests/console/start.rs +++ b/tests/console/start.rs @@ -326,6 +326,89 @@ fn api_checks_the_same_named_process_as_the_cli() { assert_eq!(api::json(response), json!(expected.stdout)); } +#[test] +fn api_checks_a_custom_runner_with_the_cli_report() { + let scenario = fixture(); + let directory = scenario.path("home/.thirdshift/runners"); + fs::create_dir_all(&directory).unwrap(); + fs::write(directory.join("runner.py"), "print('runner')\n").unwrap(); + let manifest = directory.join("fixture.toml"); + fs::write( + &manifest, + "command = [\"python3\", \"{runner_dir}/runner.py\"]\n[files]\n\"runner.py\" = \"runner.py\"\n", + ) + .unwrap(); + let runner = tokens::create(&scenario.path("home"), "runner", &["--scopes", "run"]); + let reader = tokens::create(&scenario.path("home"), "reader", &[]); + let console = Console::start_command(scenario.command(&["console", "--no-sign-in"])); + let client = reqwest::blocking::Client::new(); + let post = |token: &str, body: Value| { + client + .post(format!("{}/api/v1/evals/check-runner", console.address)) + .bearer_auth(token) + .json(&body) + .send() + .unwrap() + }; + for (manifest_text, code) in [ + (None, Some(0)), + (Some("\"missing.py\" = \"missing.py\"\n"), Some(1)), + ] { + if let Some(text) = manifest_text { + let current = fs::read_to_string(&manifest).unwrap(); + fs::write(&manifest, current + text).unwrap(); + } + let expected = scenario.run(&["evals", "check-runner", "fixture", "--json"]); + assert_eq!(expected.code, code, "{}", expected.stderr); + let expected: Value = serde_json::from_str(&expected.stdout).unwrap(); + assert_eq!( + api::json(post(&runner, json!({"runner":"fixture"}))), + expected + ); + } + assert_eq!(post(&reader, json!({"runner":"fixture"})).status(), 403); + for (body, evidence) in [ + (json!({"runner":"fixture.toml"}), "CLI-only"), + ( + json!({"runner":"fixture","suite":"process"}), + "both --suite and --task", + ), + ( + json!({"runner":"fixture","suite":"process","task":"absent"}), + "suites/process/suite.toml", + ), + ] { + let refused = post(&runner, body.clone()); + assert_eq!(refused.status(), 422, "{body}"); + let message = refused.json::().unwrap()["error"].to_string(); + assert!(message.contains(evidence), "{body}: {message}"); + } +} + +/// A `run` token cannot make the Console read a caller-chosen local `.toml` +/// file and echo its text back in a manifest problem (ADR 0026). +#[test] +fn api_check_runner_does_not_read_caller_selected_files() { + let scenario = fixture(); + let secret = "dummy-check-runner-secret-6d1f"; + let private = scenario.path("home/private"); + fs::create_dir_all(&private).unwrap(); + let file = private.join("tool.toml"); + fs::write(&file, format!("api_key = \"{secret}\"\n")).unwrap(); + let runner = tokens::create(&scenario.path("home"), "runner", &["--scopes", "run"]); + let console = Console::start_command(scenario.command(&["console", "--no-sign-in"])); + let response = reqwest::blocking::Client::new() + .post(format!("{}/api/v1/evals/check-runner", console.address)) + .bearer_auth(&runner) + .json(&json!({"runner": file.to_str().unwrap()})) + .send() + .unwrap(); + let status = response.status(); + let body = response.text().unwrap(); + assert!(!body.contains(secret), "{status}: {body}"); + assert_eq!(status, 422, "{body}"); +} + #[test] fn api_test_email_uses_the_cli_sender_and_is_audited() { let scenario = fixture(); diff --git a/tests/experiments.rs b/tests/experiments.rs index d61ec925..3cf51a35 100644 --- a/tests/experiments.rs +++ b/tests/experiments.rs @@ -66,6 +66,9 @@ mod process; #[path = "experiments/custom_runner.rs"] mod custom_runner; +#[path = "experiments/check_runner.rs"] +mod check_runner; + #[path = "experiments/pairs.rs"] mod pairs; diff --git a/tests/experiments/check_runner.rs b/tests/experiments/check_runner.rs new file mode 100644 index 00000000..d370dd9d --- /dev/null +++ b/tests/experiments/check_runner.rs @@ -0,0 +1,417 @@ +use super::custom_runner::{DELIVER, declarations, harness_script, runner}; +use super::*; + +/// The report on stdout, whatever the exit status, with that status. +fn check(fixture: &Fixture, args: &[&str]) -> (Option, Value) { + let mut all = vec!["evals", "check-runner"]; + all.extend_from_slice(args); + all.push("--json"); + let result = fixture.scenario.run_with_env(&all, &NO_SANDBOX); + let report = serde_json::from_str(&result.stdout).unwrap_or_else(|error| { + panic!( + "{error}: stdout {:?}, stderr {:?}", + result.stdout, result.stderr + ) + }); + (result.code, report) +} + +fn problems(report: &Value, check: &str) -> Vec { + report["problems"] + .as_array() + .unwrap() + .iter() + .filter(|problem| problem["check"] == check) + .map(|problem| problem["message"].as_str().unwrap().to_owned()) + .collect() +} + +fn manifest(fixture: &Fixture) -> PathBuf { + fixture + .scenario + .path("home/.thirdshift/runners/fixture.toml") +} + +fn append(fixture: &Fixture, text: &str) { + let path = manifest(fixture); + let manifest = fs::read_to_string(&path).unwrap(); + fs::write(path, format!("{manifest}\n{text}\n")).unwrap(); +} + +/// A Claude stand-in reporting one priced Model turn. +fn priced_claude(fixture: &Fixture) { + harness_script( + fixture, + "claude", + r#"echo '{"type":"assistant","message":{"id":"m1","model":"claude-sonnet-4-6","usage":{"input_tokens":10,"output_tokens":2},"content":[{"type":"text","text":"done"}]}}' +echo '{"type":"result","subtype":"success","result":"done"}'"#, + ); +} + +#[test] +fn a_valid_runner_passes_with_the_digest_an_experiment_records() { + let fixture = process::fixture(); + runner(&fixture, DELIVER); + let (code, report) = check(&fixture, &["fixture"]); + assert_eq!(code, Some(0), "{report}"); + assert_eq!(report["ok"], true, "{report}"); + assert_eq!(report["problems"], json!([]), "{report}"); + assert!(report["trial"].is_null(), "{report}"); + assert_eq!( + PathBuf::from(report["manifest"].as_str().unwrap()), + manifest(&fixture).canonicalize().unwrap() + ); + let digest = report["digest"].as_str().unwrap(); + assert_eq!(digest.len(), 64, "{report}"); + let path = manifest(&fixture); + let (code, by_path) = check(&fixture, &[path.to_str().unwrap()]); + assert_eq!(code, Some(0), "{by_path}"); + assert_eq!(by_path["digest"], digest); + let text = fixture + .scenario + .run_with_env(&["evals", "check-runner", "fixture"], &NO_SANDBOX); + assert_eq!(text.code, Some(0), "{}", text.stderr); + assert!( + text.stdout.contains("Static checks: passed"), + "{}", + text.stdout + ); + assert!(text.stdout.contains(digest), "{}", text.stdout); + assert!( + text.stdout.contains("Custom runner check passed."), + "{}", + text.stdout + ); +} + +#[test] +fn malformed_manifests_are_refused_before_any_other_check() { + for (text, evidence) in [ + ("command = []\n", "must contain a program"), + ("command = [\"run\"]\nunknown = 1\n", "unknown field"), + ("command = \"run\"\n", "invalid type"), + ( + "command = [\"run\"]\ngit_exclude = [\"a\\nb\"]\n", + "must each be one line", + ), + ] { + let fixture = process::fixture(); + runner(&fixture, DELIVER); + fs::write(manifest(&fixture), text).unwrap(); + let (code, report) = check(&fixture, &["fixture"]); + assert_eq!(code, Some(1), "{report}"); + assert_eq!(report["ok"], false, "{report}"); + assert_eq!(report["problems"].as_array().unwrap().len(), 1, "{report}"); + let messages = problems(&report, "manifest"); + assert!(messages[0].contains(evidence), "{text}: {report}"); + assert!(report["digest"].is_null(), "{report}"); + } + let fixture = process::fixture(); + let (code, report) = check(&fixture, &["absent"]); + assert_eq!(code, Some(1), "{report}"); + assert!( + problems(&report, "manifest")[0].contains("cannot resolve Custom runner absent"), + "{report}" + ); +} + +#[test] +fn missing_sources_and_escaping_home_targets_are_each_reported() { + let fixture = process::fixture(); + runner(&fixture, DELIVER); + append( + &fixture, + "\"missing.py\" = \"missing.py\"\n\"./runner.py\" = { home = \"../outside\" }", + ); + let (code, report) = check(&fixture, &["fixture"]); + assert_eq!(code, Some(1), "{report}"); + let messages = problems(&report, "files"); + assert_eq!(messages.len(), 2, "{report}"); + assert!( + messages + .iter() + .any(|message| message.contains("cannot find Custom runner file missing.py")), + "{report}" + ); + assert!( + messages + .iter() + .any(|message| message.contains("confined relative Eval path: ../outside")), + "{report}" + ); + assert!(report["digest"].is_null(), "{report}"); +} + +#[test] +fn missing_harness_logins_and_provider_keys_are_reported_then_accepted() { + let fixture = process::fixture(); + runner(&fixture, DELIVER); + harness_script(&fixture, "claude", "exit 0"); + harness_script(&fixture, "codex", "exit 0"); + declarations( + &fixture, + "providers = [\"openai\"]\nharnesses = { claude = [], codex = [], unknown = [] }", + ); + let (code, report) = check(&fixture, &["fixture"]); + assert_eq!(code, Some(1), "{report}"); + assert!( + problems(&report, "harnesses")[0].contains("unknown Custom runner Harness unknown"), + "{report}" + ); + let logins = problems(&report, "logins"); + assert_eq!(logins.len(), 2, "{report}"); + assert!( + logins + .iter() + .any(|message| message.contains("declared claude has no Trials setup token")), + "{report}" + ); + assert!( + logins + .iter() + .any(|message| message.contains("declared codex has no login")), + "{report}" + ); + assert_eq!( + problems(&report, "providers"), + ["missing [providers].openai Trial key"], + "{report}" + ); + + let text = fs::read_to_string(manifest(&fixture)) + .unwrap() + .replace(", unknown = []", ""); + fs::write(manifest(&fixture), text).unwrap(); + fixture.scenario.credentials_are( + "[claude]\nsetup_token = \"trial-login\"\n[providers]\nopenai = \"trial-key\"\n", + ); + fs::create_dir_all(fixture.scenario.path("home/.codex")).unwrap(); + fs::write(fixture.scenario.path("home/.codex/auth.json"), "{}").unwrap(); + let (code, report) = check(&fixture, &["fixture"]); + assert_eq!(code, Some(0), "{report}"); + assert_eq!(report["problems"], json!([]), "{report}"); +} + +#[test] +fn roles_naming_unpriced_models_are_reported() { + let fixture = process::fixture(); + runner(&fixture, DELIVER); + append( + &fixture, + "[roles.implement]\nmodel = \"claude-sonnet-4-6\"\n[roles.review]\nmodel = \"unpriced-model\"\neffort = \"high\"", + ); + let (code, report) = check(&fixture, &["fixture"]); + assert_eq!(code, Some(1), "{report}"); + let roles = problems(&report, "roles"); + assert_eq!(roles.len(), 1, "{report}"); + assert!( + roles[0].contains("role review names Model unpriced-model"), + "{report}" + ); + // A pricing problem does not stop the runner from freezing. + assert_eq!(report["digest"].as_str().unwrap().len(), 64, "{report}"); +} + +#[test] +fn a_dry_run_reports_outcome_checks_and_session_cost_without_an_experiment() { + let fixture = process::fixture(); + priced_claude(&fixture); + runner( + &fixture, + &(String::from( + "subprocess.run(request['record_command'] + ['--role','implement','--','claude','-p','test'], check=True)\n", + ) + DELIVER), + ); + declarations(&fixture, "harnesses = { claude = [] }"); + fixture + .scenario + .credentials_are("[claude]\nsetup_token = \"trial-login\"\n"); + let (code, report) = check( + &fixture, + &["fixture", "--suite", "process", "--task", "fixture"], + ); + assert_eq!(code, Some(0), "{report}"); + assert_eq!(report["ok"], true, "{report}"); + let trial = &report["trial"]; + assert_eq!(trial["suite"], "process", "{report}"); + assert_eq!(trial["task"], "fixture", "{report}"); + assert_eq!(trial["outcome"], "delivered", "{report}"); + assert!(trial["void_cause"].is_null(), "{report}"); + assert!( + trial["cause"] + .as_str() + .unwrap() + .contains("uncommitted working-tree files ignored by grading"), + "{report}" + ); + assert_eq!(trial["passed"], true, "{report}"); + let checks = trial["checks"].as_array().unwrap(); + assert!(checks.len() > 1, "{report}"); + assert!( + checks.iter().all(|check| check["passed"] == true), + "{report}" + ); + assert!( + checks.iter().any(|check| check["command"].is_array()), + "the task's code checks ran: {report}" + ); + let sessions = trial["sessions"].as_array().unwrap(); + assert_eq!(sessions.len(), 1, "{report}"); + assert_eq!(sessions[0]["role"], "implement", "{report}"); + assert_eq!(sessions[0]["cost_usd"], 0.000060, "{report}"); + assert_eq!(trial["cost_usd"], 0.000060, "{report}"); + assert!( + !fixture + .scenario + .path("home/.thirdshift/evals/experiments") + .exists(), + "a dry run writes no Experiment storage" + ); +} + +#[test] +fn a_dry_run_reports_its_void_cause_and_fails_the_check() { + let fixture = process::fixture(); + harness_script(&fixture, "claude", "exit 0"); + runner(&fixture, DELIVER); + declarations(&fixture, "harnesses = { claude = [] }"); + fixture + .scenario + .credentials_are("[claude]\nsetup_token = \"trial-login\"\n"); + let (code, report) = check( + &fixture, + &["fixture", "--suite", "process", "--task", "fixture"], + ); + assert_eq!(code, Some(1), "{report}"); + assert_eq!(report["ok"], false, "{report}"); + assert_eq!(report["problems"], json!([]), "{report}"); + assert_eq!(report["trial"]["outcome"], "void", "{report}"); + assert_eq!(report["trial"]["void_cause"], "accounting_gap", "{report}"); + assert_eq!(report["trial"]["passed"], false, "{report}"); + let text = fixture.scenario.run_with_env( + &[ + "evals", + "check-runner", + "fixture", + "--suite", + "process", + "--task", + "fixture", + ], + &NO_SANDBOX, + ); + assert_eq!(text.code, Some(1), "{}", text.stderr); + assert!( + text.stdout + .contains("Dry run process/fixture: void (accounting_gap); Score failed"), + "{}", + text.stdout + ); + assert!( + text.stdout.contains("Custom runner check failed."), + "{}", + text.stdout + ); + assert!( + report["trial"]["cost_usd"].is_null(), + "an accounting gap has no cost: {report}" + ); + assert!(text.stdout.contains("Cost: incomplete"), "{}", text.stdout); +} + +#[test] +fn static_problems_skip_the_dry_run() { + let fixture = process::fixture(); + let started = fixture.scenario.path("runner-started"); + runner( + &fixture, + &format!( + "pathlib.Path({:?}).write_text('started')\n{DELIVER}", + started + ), + ); + harness_script(&fixture, "claude", "exit 0"); + declarations(&fixture, "harnesses = { claude = [] }"); + let (code, report) = check( + &fixture, + &["fixture", "--suite", "process", "--task", "fixture"], + ); + assert_eq!(code, Some(1), "{report}"); + assert_eq!(problems(&report, "logins").len(), 1, "{report}"); + assert!(report["trial"].is_null(), "{report}"); + assert!(!started.exists(), "the runner must not start"); +} + +#[test] +fn a_dry_run_refuses_unusable_tasks_before_checking_the_runner() { + let fixture = process::fixture(); + runner(&fixture, DELIVER); + for (args, evidence) in [ + ( + vec!["--suite", "process", "--task", "absent"], + "cannot load Eval task process/absent", + ), + ( + vec!["--suite", "code-review", "--task", "fixture"], + "Custom runners require process suites: code-review", + ), + (vec!["--suite", "process"], "needs both --suite and --task"), + ( + vec!["--task", "../fixture", "--suite", "process"], + "invalid Eval suite or task name: ../fixture", + ), + ] { + let mut all = vec!["evals", "check-runner", "fixture"]; + all.extend(args); + all.push("--json"); + let result = fixture.scenario.run_with_env(&all, &NO_SANDBOX); + assert_ne!(result.code, Some(0), "{all:?}"); + assert!(result.stdout.is_empty(), "{all:?}: {}", result.stdout); + assert!( + result.stderr.contains(evidence), + "{all:?}: {}", + result.stderr + ); + } +} + +#[test] +fn a_dry_run_confines_the_runner_in_the_trial_sandbox() { + let available = |program: &str, flag: &str| { + std::process::Command::new(program) + .arg(flag) + .output() + .is_ok_and(|output| output.status.success()) + }; + if !available("bwrap", "--version") || !available("socat", "-V") { + println!("SKIPPED (dry-run sandbox): bwrap or socat is not on PATH"); + return; + } + let fixture = process::fixture(); + let hidden = fixture.scenario.path("operator-private"); + fs::write(&hidden, "operator secret").unwrap(); + runner( + &fixture, + &format!( + "assert not pathlib.Path({hidden:?}).exists(), 'unbound host file exposed'\nassert 'THIRDSHIFT_OPERATOR_TOKEN' not in os.environ\n{DELIVER}" + ), + ); + let result = fixture.scenario.run_with_env( + &[ + "evals", + "check-runner", + "fixture", + "--suite", + "process", + "--task", + "fixture", + "--json", + ], + &[("THIRDSHIFT_OPERATOR_TOKEN", "operator-token")], + ); + let report: Value = serde_json::from_str(&result.stdout) + .unwrap_or_else(|error| panic!("{error}: {}", result.stderr)); + assert_eq!(report["trial"]["outcome"], "delivered", "{report}"); + assert_eq!(report["trial"]["passed"], true, "{report}"); + assert_eq!(result.code, Some(0), "{report}"); +} diff --git a/tests/experiments/custom_runner.rs b/tests/experiments/custom_runner.rs index f991f2b2..17f846eb 100644 --- a/tests/experiments/custom_runner.rs +++ b/tests/experiments/custom_runner.rs @@ -1,6 +1,6 @@ use super::*; -fn declarations(fixture: &Fixture, text: &str) { +pub(super) fn declarations(fixture: &Fixture, text: &str) { let path = fixture .scenario .path("home/.thirdshift/runners/fixture.toml"); @@ -8,7 +8,7 @@ fn declarations(fixture: &Fixture, text: &str) { fs::write(path, format!("{text}\n{manifest}")).unwrap(); } -fn harness_script(fixture: &Fixture, name: &str, body: &str) { +pub(super) fn harness_script(fixture: &Fixture, name: &str, body: &str) { let path = fixture.scenario.path(&format!("bin/{name}")); let _ = fs::remove_file(&path); fs::write(&path, format!("#!/bin/sh\nif [ \"$1\" = --version ]; then echo '{name} fixture 1'; exit 0; fi\n{body}\n")).unwrap(); @@ -876,7 +876,7 @@ fn nonzero_exit_without_outer_void_evidence_fails_despite_a_runner_void_claim() } } -fn runner(fixture: &Fixture, body: &str) -> String { +pub(super) fn runner(fixture: &Fixture, body: &str) -> String { let directory = fixture.scenario.path("home/.thirdshift/runners"); fs::create_dir_all(&directory).unwrap(); fs::write( @@ -909,7 +909,7 @@ git_exclude = [".scratch/"] "fixture".into() } -const DELIVER: &str = r#" +pub(super) const DELIVER: &str = r#" subprocess.run(['git', 'config', 'user.name', 'Eval'], check=True) subprocess.run(['git', 'config', 'user.email', 'eval@example.invalid'], check=True) pathlib.Path('feature.txt').write_text('implemented\n')