Freeze Custom runner manifests and run Custom-runner Trials on process suites #2188

This was generated by AI during a grill-with-docs session.

Parent

#1468

What to build

End-to-end Custom runner Arm preparation and Trial execution on process Eval suites (#1468 Q1.1–Q1.3, Q1.5, Q1.7, Q2.1–Q2.3, Q2.5, Q3.2; ADR 0032).

• Symmetric CLI selection and validation: evals prepare-arms and experiment accept --baseline-runner <name-or-file.toml> and --candidate-runner <name-or-file.toml> (resolved from ~/.thirdshift/runners/<name>.toml or an explicit .toml path). Whenever either Arm uses a Custom runner, --suite is required and every named suite must be a process suite; code-review suites, spec-review suites, --release, and combining --*-runner with a commit, --*-process, --*-harness, --*-model, or --*-effort on the same Arm are refused before dispatch. When --candidate-runner is set, the positional <candidate> commit is omitted; when Baseline is a thirdshift build and Candidate is a Custom runner, --baseline defaults to HEAD; when both Arms are Custom runners, neither a thirdshift checkout nor a Cargo build is needed.
• Manifest freezing and provenance: prepare-arms freezes the manifest (command = [...] argv with no shell splitting, mapped files, optional roles map, and optional git_exclude patterns) and every file or folder in files by SHA-256—placing each either in the frozen bundle (runner_dir) or at a validated relative path inside fresh HOME (rejecting .. and absolute targets). When a source path sits inside a Git worktree, record its repository origin/path, HEAD commit, and dirty boolean as provenance without refusing uncommitted local edits. Verify every frozen file's SHA-256 before each Trial starts.
• Setup format 13: make commit and harness nullable (null on a Custom-runner Arm) and add runner and runner_digest to Setup, bumping the Experiment result format to 13 so Experiment identity, resume checks, and per-Arm cost history key accurately off the Custom runner.
• Trial execution, git_exclude, and grading: before launching the Custom runner in the prepared repository, append git_exclude patterns to .git/info/exclude alongside .thirdshift-issue.md. Pass version-1 request.json (version, repository, issue_file, issue_url, base, branch, runner_dir, output_dir, record_command). Run strictly unattended with no simulated user. Read committed HEAD and version-1 output_dir/result.json (delivered, stopped_to_ask, failed, or void restricted to session_unavailable, usage_limit, rate_limit; outer supervision keeps precedence). Record missing/unreadable result.json, unexplained nonzero exit, claiming an outer void_cause, or committing a path matching git_exclude on HEAD as a failed Trial (with exit status and stderr in cause) rather than aborting the Experiment. Ignore uncommitted working-tree files during grading (noting them in Trial diagnostics), grade HEAD through code Graders and Pair evidence, and count stopped_to_ask as a failed Score tallied under outcomes.stopped_to_ask.

Acceptance criteria

• If not already on the base branch, GLOSSARY.md defines Custom runner (and broadens Arm, Trial, Experiment, and Credentials) and docs/adr/0032-custom-runners-run-inside-one-trial-sandbox.md records the Custom runner boundary.
• An Experiment can pit a thirdshift Process Arm against a Custom runner, a Custom runner against a thirdshift Process Arm, or two Custom runners against each other (with no thirdshift checkout or Cargo build when both Arms are Custom runners), grading committed HEAD with code Graders and Pair evidence.
• code-review suites, spec-review suites, --release, missing --suite, and conflicting per-Arm flags/commit arguments with --*-runner fail validation before dispatch.
• Frozen bundle files and fresh-HOME mapped skill trees are verified by SHA-256 before every Trial (aborting if tampered), reject .. and absolute target paths, and record Git repository/commit/dirty provenance without refusing dirty local worktrees.
• Format-13 Setup stores nullable commit/harness plus runner and runner_digest, preserving Experiment resume identity and per-Arm cost-estimate lookup across commits.
• git_exclude keeps runner scratch files out of git add ., and a Custom runner that force-commits a git_exclude path, exits nonzero without outer Void evidence, omits result.json, or claims an outer void_cause records that Trial as failed without aborting the Experiment.

Blocked by

• None (can start immediately).