[JacobStephens2]

Summary

Console Experiments accept Custom runners and show their frozen identity without assuming a thirdshift commit or a single Harness.

POST /api/v1/start (also /api/v1/runs)
  shared CLI validation: baseline_runner / candidate_runner
  format-13 Setup → Experiment comparison and Trial Peek
  blind pair → sealed merge calls → preference → runner details in revealed

OpenAPI describes nullable commit/Harness fields and optional runner identity. Pair evidence withholds runner attribution and role tables while preserving ordinary source declarations. The manual and hand-written site pages describe the behavior.

Closes #2190

Evidence

Execution evidence

• Before: node --test --test-name-pattern='identity in retained|preserves ordinary' console/tests/experiments.test.cjs failed both tests: leaked inspect; runner: bool and role = load_user_role() disappeared from the returned diff.
  After: the same command passes: 2 passed; 0 failed. Both regressions remain in the suite.
• cargo test --test console api_starts_custom_runner_experiments_and_shares_cli_refusals: passes all three runner selections, native HEAD defaults, repeat CLI parity and invalid combinations.
• All five reviewer reproduction commands were run unchanged, failed as reported before correction, and now pass. They cover high_watermark, highest_score, runner: bool, role = ..., and Markdown role-table disclosure.
• Custom-runner list/detail JSON matches CLI JSON and validates against the served OpenAPI schemas with Draft202012Validator.
• cargo clippy --all-targets -- -D warnings: passes.
• RUST_TEST_THREADS=4 scripts/affected-tests refs/remotes/origin/issue-1468: PASS. Site checks passed (69 tests, 72.73s), Console checks passed (209 tests, 152.02s), and the full Rust suite passed via cargo test (2143.58s / 35m44s, compilation included). Updated Custom-runner browser checks also pass separately (4 tests). Nextest was unavailable, so the documented fallback ran with four test threads.

Visual evidence

Matched Chromium captures cover the Arm comparison, Trial Peek and post-preference reveal at 1440×900 desktop and 390×844 mobile. All six final states fit the viewport. Gist publication failed because the supplied GitHub token lacks the gist scope (HTTP 404 from POST /gists); captures remain at these exact paths:

Viewport / flow	Before	After
Desktop / comparison	/tmp/before-desktop.M7MJEq.png	/tmp/after-desktop.0YMcXZ.png
Desktop / Trial Peek	/tmp/before-desktop.U4Q9sJ.png	/tmp/after-desktop.jv6M6t.png
Desktop / reveal	/tmp/before-desktop.gqANqn.png	/tmp/after-desktop.V6G9jK.png
Mobile / comparison	/tmp/before-mobile.7MqDIB.png	/tmp/after-mobile.GNiLhO.png
Mobile / Trial Peek	/tmp/before-mobile.wPoO4F.png	/tmp/after-mobile.ttqgH9.png
Mobile / reveal	/tmp/before-mobile.JQVIW6.png	/tmp/after-mobile.31gZjL.png

Test seams

• Structured Experiment options at Console start admission, using the shared CLI parser.
• Console HTTP start, Experiment list/detail JSON and OpenAPI, and blind pair calls.
• Rendered Experiment Arm comparison and Trial Peek in Chromium at desktop and mobile viewports.

Review coverage

Independent Standards and Spec reviews used refs/remotes/origin/issue-1468 (d6b4f99b). Each read all 19 changed files, including the generated manual hunk. No changed file was left unread and no generated output was excluded. Reports and final files-read lists are retained in .thirdshift-review-3ia6E7/standards.md and spec.md; summary.json records Standards 1 / Spec 3, including findings subsequently fixed.

Unaddressed findings

• Standards: None. The evidence-corruption finding and its follow-up variant were reproduced, fixed and retained as regressions.
• Spec: None. All three findings were reproduced, fixed and retained as regressions.
• Security: None. Reviewed git diff d6b4f99bde5c24f595dfa339d090ad14d076fac3...HEAD at 51776790d3f1c0954d445bc4bac90161bc316331 and supporting code in one guidance-mode session. No confirmed introduced Security findings. Checked the new start route's run scope and CSRF enforcement, shared CLI argument validation, escaped runner identity rendering, and the pairwise reveal gate. The cached unit-test executable passed console::start::experiment::tests (2), console::filters::tests (3), and evals::experiment::pairwise::tests (2), with no network, an empty allowlisted environment, read-only binaries, bounded scratch storage and resource limits. No source changes.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Adds an API alias and presentation/schema support; no stored-format migration.

Blast Radius: Experiments

Experiment admission, Arm presentation and blind pair evidence. Existing CLI validation and native Arm behavior remain covered.

Built with codex codex-cli 0.167.2 · gpt-6.1-sol · xhigh