[JacobStephens2]

Summary

Closes #2189

Arm preparation
  freeze runner inputs, declared Harness paths/versions, provider hosts and families
Trial memory scope
  one bwrap sandbox → Custom runner and every child
    fresh HOME → mapped files and declared Harness logins
    declared [providers] → Trials-only SDK keys
    PATH wrappers → pinned Harnesses with native Trial web restrictions
    HTTPS_PROXY → union of declared provider API hosts
Grader panel
  freeze provider families → withhold same-family votes → check stand-in breadth

Missing declared logins or SDK keys produce session_unavailable Void Trials. Grok holds its Trials-home lock through the runner; OpenCode gets a multi-provider configuration with web tools denied. Kimi accepts provider-env API keys and endpoints, and saved provider-family identities survive reports and resume, including legacy Bars. The manual and all three hand-written site pages describe the shipped sandbox.

Merged the current issue-1468 base (2ff812fc) in 463650ed, preserving its Console Custom-runner behavior and regenerating the manual from the combined source.

Evidence

Execution evidence

Original implementation evidence retained from the existing branch:

• Before: cargo test --test experiments custom_runner_preparation_pins_declared_harnesses_and_provider_hosts -- --nocapture failed with unknown field harnesses, expected one of command, files, roles, git_exclude.
  After: the same command passed (1 passed; 0 failed).
• Before: cargo test --test experiments custom_runner_sandbox_confines_children_credentials_network_and_harness_io -- --nocapture failed with AssertionError: unbound host file exposed.
  After: the same command passed (1 passed; 0 failed), exercising local bwrap, child process I/O/status, declared credentials and denied hosts.

Each behavioral review command was run before deciding the finding, failed as reported, and passed after correction. All regressions are retained:

Exact command	Actual failing-before evidence	Actual passing-after evidence
cargo test --test experiments custom_runner::custom_kimi_accepts_declared_provider_env_login_without_managed_oauth -- --exact --nocapture	left: String("void"), right: "delivered"; missing unrelated credentials/kimi-code.json	1 passed; 0 failed
cargo test --test experiments custom_runner::custom_kimi_accepts_declared_provider_env_endpoint_without_managed_oauth -- --exact --nocapture	left: String("void"), right: "delivered"; Kimi provider hosts changed since Arm preparation	1 passed; 0 failed
cargo test --test experiments custom_runner::a_kimi_panel_seat_with_the_custom_runners_fireworks_alias_withholds_its_vote -- --exact --nocapture	expected pre-Trial family refusal absent; output showed Trials: 2 remaining and later panel Grader left no calls	1 passed; 0 failed
cargo test --test experiments custom_runner::a_legacy_kimi_panel_bar_resumes_with_its_original_family_classification -- --exact --nocapture	the same command must resume the legacy Bar; got: thirdshift: Bar cannot change after the first Trial	1 passed; 0 failed

The Standards assertion below failed with AssertionError: Architecture still describes the #2189 Trial sandbox as future work, then passed with exit 0. It is retained in site/tests/pages.test.cjs.

python3 - <<'CHECK'
from pathlib import Path
page = Path('site/architecture/index.html').read_text()
stale = 'The Trial-wide provider sandbox and session recorder follow in #2189 and #2191.'
assert stale not in page, 'Architecture still describes the #2189 Trial sandbox as future work'
CHECK

Focused validation before the base merge: cargo test --test experiments custom_runner:: -- --nocapture passed 25 tests; cargo test --test eval_arms passed 12; cargo test --bin thirdshift evals::experiment::panel::tests passed 3. The additional endpoint and legacy-resume regressions passed separately afterward.

Final local validation on 463650ed: NEXTEST_TEST_THREADS=3 /usr/bin/time -p scripts/affected-tests refs/remotes/origin/issue-1468 passed (exit 0). Site: 70 passed; 0 failed, 89.58s. Rust: 4557 passed; 201 skipped across 107 binaries, 4297.73s including compilation (3m31s) with a warm dependency cache and three test threads. Total selector wall time: 4387.98s. All CI checks passed on the same head, including the four Rust partitions and coverage gate, Console/site browser tests, builds and release smoke checks.

Visual evidence

Matched Chromium captures use base 2ff812fc for Before and merged head 463650ed for After. Each pair uses the same flow, viewport and open-section state. Media is hosted in a secret Gist; exact mktemp paths are retained below.

Homepage / Experiments fold — 1440×900

Before: /tmp/before-desktop.a76ZmY.png. After: /tmp/after-desktop.YoH5JZ.png.

Homepage / Experiments fold — 390×844

Before: /tmp/before-mobile.mW9JRV.png. After: /tmp/after-mobile.IiUmxH.png.

How it works / Custom runner description — 1440×900

Before: /tmp/before-desktop.eF6hDL.png. After: /tmp/after-desktop.SFhsi4.png.

How it works / Custom runner description — 390×844

Before: /tmp/before-mobile.49HlaJ.png. After: /tmp/after-mobile.ENHQPx.png.

Manual / Experiments section — 1440×900

Before: /tmp/before-desktop.gnYf4O.png. After: /tmp/after-desktop.TfalWV.png.

Manual / Experiments section — 390×844

Before: /tmp/before-mobile.hLGLDP.png. After: /tmp/after-mobile.NPBEN6.png.

Architecture / Experiments and Scores fold — 1440×900

Before: /tmp/before-desktop.4nIG31.png. After: /tmp/after-desktop.3iVaOs.png.

Architecture / Experiments and Scores fold — 390×844

Before: /tmp/before-mobile.8b3pjJ.png. After: /tmp/after-mobile.jYUz7y.png.

Test seams

• thirdshift evals prepare-arms and thirdshift experiment ... --json: frozen declarations, pinned executables, credential Void causes, runner child I/O, provider configuration and Grader panel rules.
• Sandbox launch: existing local bwrap and proxy fixtures, including native Trial behavior.
• thirdshift experiments ... --json and repeat Experiment commands: retained provider identity and legacy-Bar resume.
• Static site output and Chromium rendering at desktop/mobile widths.

Review coverage

Independent Standards and Spec reviews used git diff refs/remotes/origin/issue-1468...HEAD with fixed reference 2ff812fcf310255602a839431b11f51e9e5ababe, plus one coverage follow-up per axis for the fixes. The initial comparison had merge-base d6b4f99b; the final branch includes the base merge and has 28 changed files.

Standards raised 1 finding; Spec raised 3, including the resume finding from the follow-up. All were reproduced, fixed and retained as regressions. Reports and each axis's final files-read list are saved in .thirdshift-review-dNeEQo/standards.md and spec.md; summary.json is {"standards":1,"spec":3}.

• Standards unread changed files: src/evals/experiment.rs and src/evals/experiment/store.rs. These resume-compatibility edits followed that axis's one permitted coverage follow-up; Spec and the implementing agent read them.
• Standards partially reviewed final files: later endpoint/resume hunks in src/sandbox/custom.rs, src/evals/experiment/panel.rs and tests/experiments/custom_runner.rs followed its coverage follow-up. Standards read earlier hunks in those files; Spec read the final edits.
• Spec unread changed files: none.
• Generated-output exclusions: none; both reviewers read the generated manual hunks directly. Merge resolution regenerated the manual from docs/guide.md; the implementing agent inspected the merged output and preserved the base's surrounding Console documentation.

Unaddressed findings

Standards

None. S1 (the architecture page's stale #2189 statement) was fixed and retained as a site regression.

Spec

None. All three findings were fixed; no finding was declined or skipped, and no Day shift decision was deferred.

Security

None. The review covered the Custom runner sandbox's binds, environment allowlist, declared Harness logins, [providers] keys, the Kimi config copy, the allowlisting proxy, the PATH wrappers and the Void Trial path. It found no vulnerability introduced by this change.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

No data migration is required. A rollback to an older reader can require adapting newly saved manifest/family fields. Existing Bars retain their original family classification.

Blast Radius: Experiments

Custom-runner Trial launch, declared credentials and provider access, and panel family voting. The test/emergency THIRDSHIFT_TRIAL_NO_SANDBOX escape remains available.

Built with codex codex-cli 0.167.1 · gpt-6.1-sol · xhigh

[JacobStephens2]

Delivery completed on agent-authored or reviewed head 7a03d84.