[JacobStephens2]

Summary

Custom runner sessions now become retained, priced Session traces, and Trials whose evidence breaks the sandbox's accounting become Void Trials. Closes #2191.

Custom runner
  record_command --role → protected live OTLP traces
  sessions.json         → raw Harness / SDK conversion → same OTLP traces
  traces                → frozen prices, roles, leak checks, Trial Peek, Waterfall, export
  quota ≥85%            → pause Experiment (usage_limit)
any Trial
  web-search tool call  → Void (web_search); undecodable events from another build are skipped
runner-reported outcome (valid result.json)
  missing sessions      → Void (accounting_gap), using host proxy and PATH-wrapper evidence
timeout / could not start / crash / bad result.json
  → stays failed (outer supervision keeps precedence, #1468)

Evidence

Execution evidence

Local Rust builds were unavailable in the final session: the host has no C compiler, and the proposed zig cc shim was denied. So review findings were checked through CI's full suite on each pushed head.

• Before: CI run 43649302716 on 44a0cb99 (new tests, no fix). evals::experiment::execution::custom::tests::timed_out_runner_after_provider_traffic_stays_failed panicked: Some("accounting gap: 0 recorded sessions, 0 Harness sessions, 1 provider connections, 1 Harness runs across the proxy (codex)"). …::web_search_scan_skips_events_from_other_arm_builds panicked: unknown variant `event_from_a_newer_arm_build` . The two new accounting-gap trigger tests passed.
  After: CI run 43650177112 on fe577acb: all four Rust shards, strict Clippy/tooling, Console and site tests passed. That green run on the final head is the full-suite validation; scripts/affected-tests could not be run locally.
• Before: cargo test --test experiments custom_runner_records_live_harness_sessions_with_roles -- --nocapture failed with assert isinstance(request['record_command'], list) / AssertionError; the Trial outcome was failed instead of delivered.
  After: Same command: 1 passed; 0 failed.
• Before: cargo test --test experiments native_trials_with_normalized_web_search_calls_are_void -- --nocapture failed with left: String("delivered"), right: "void".
  After: Same command: 1 passed; 0 failed.
• Before: cargo test --test experiments experiment_trace_contains_trials_and_graders_and_reports_arm_usage -- --exact --nocapture failed with invalid type: map, expected f64.
  After: Same command: 1 passed; 0 failed.
• Before: custom_runner::missing_declared_credentials_void_custom_trials_without_using_ambient_keys, run with a PATH excluding socat, failed with sandbox_unavailable instead of session_unavailable, matching CI.
  After: Same test: 1 passed; 0 failed.
• Real-sandbox recording and deletion protection, SDK pricing and leak-check regressions passed. Importer regressions cover raw streams, cache-inclusive pricing, quota and web-search detection, and confined file reads.
• npm run docs:check --prefix site: exit 0 after regenerating the manual.
• Security review (c2a8474d): sandboxed code can reach the host's record_command socket, and the host process that runs concurrent Trials sits outside the Trial memory cap. Before: four regression tests in custom::recording::tests failed on fe577acb. An endless frame left 67218496 bytes buffered on the host. A flood of frames held one poll for 6.11s, pausing timeout and quota checks. 128 closed connections kept 128 host descriptors. 65 open connections were all accepted. After: cargo test --bin thirdshift custom::: 15 passed; 0 failed. Unfinished frames are capped at 64 MiB in total and open connections at 64, closed connections are released, and each poll returns within 50 ms. cargo test --test experiments custom_runner: 27 passed, including live recording and quota pauses. The 6 failures all report socat is not on PATH on this host, as do 20 sandbox unit tests in scripts/affected-tests.

Visual evidence

The manual's Void list and Custom runner paragraph (this session's changes). Before is the previously published head adfe4909; the changed paragraph is outlined.

Manual: Void causes — 1440×900

Manual: Void causes — 390×844

Manual: Custom runner sessions — 1440×900

Manual: Custom runner sessions — 390×844

Earlier captures of the branch's site changes, before against issue-1468:

Homepage: Experiments workflow — 1440×900 / 390×844

How it works: Experiments — 1440×900 / 390×844

Architecture: Experiments and Scores — 1440×900 / 390×844

Manual: Custom runner contract — 1440×900 / 390×844

Test seams

• thirdshift experiment: Custom runner recording, quota pauses, Void classification, role accounting and leak decisions, with retained evidence read through experiments --session and OTLP export.
• custom::collect: real Recorder and sessions.json import with fixed proxy evidence, covering each accounting-gap trigger and the precedence of supervised failures.
• custom::web_search: real OTLP log files, including events from another Arm's build.
• custom::sessions::import: real input files converted to standard OTLP and read through the existing usage and OTLP readers, including filename confinement.
• Recorder::start/poll/finish: real Unix sockets plus native Trial-home evidence verify live retained usage.
• Sandbox proxy and PATH wrappers: real local sockets verify provider counts, Harness attribution, probe exclusion and preserved stdio.

Review coverage

The final review pinned refs/remotes/origin/issue-1468...adfe4909. Both Standards and Spec read all 22 changed files. Standards excluded the generated site/docs/index.html after reviewing docs/guide.md and site/generate-docs.mjs, with npm run docs:check --prefix site exiting 0. Spec read that output's changed hunk. No follow-up pass was needed.

Not reviewed: the files changed by the review fixes. Those are docs/code-navigation.md, docs/trial.md and the new src/evals/experiment/execution/custom/tests.rs, plus fix edits to these files:

• src/evals/experiment/execution/custom.rs, src/evals/experiment/execution.rs, src/evals/experiment/execution/custom/recording.rs (test-module rename)
• src/evals/experiment/execution/custom/sessions/tests.rs (test rename)
• docs/guide.md, docs/eval-tasks.md, docs/experiment-results.md and site/docs/index.html (regenerated)

Spec findings, 3 actionable, all addressed:

• The timeout/crash accounting-gap precedence and the strict web-search decoding each failed in CI as described before the fix.
• The untested accounting-gap triggers now have tests.

Standards findings, 12 actionable: fixed the native web-search Void docs, the stale docs/code-navigation.md pointers and the non-selectable test-module name. The rest are below.

Unaddressed findings

Standards

• 1 (follow-up only): ask whether web search should be a failed Score rather than a Void Trial (GLOSSARY.md:433). Declined. Record Custom runner sessions into OTLP traces and enforce accounting_gap and web-search Void checks #2191 requires voiding "any Trial whose normalized session transcripts contain a web-search tool call", and Spec: Custom-runner Arms for process Experiments #1468 Q2.6 makes the same decision. Evidence-integrity Voids (accounting_gap, fix_leaked) already sit under the same definition.
• 4: duplicated web-search Voiding; detectors live in custom. Declined (judgement call). The collect branch (src/evals/experiment/execution/custom.rs:42-45) lets web_search take precedence over the accounting-gap check that follows it. The post-collection check in src/evals/experiment/execution.rs covers native Trials. Moving modules would not change behaviour.
• 5 and 6: collect reads Recorder fields; repeated flag-and-first-limit merging. Declined (judgement call). The merge happens in one function, and Recorder is private to execution::custom (recording.rs:15).
• 7: four matches on Source in import. Declined (judgement call). It is a two-variant enum matched within one function (sessions.rs:51-91).
• 8: Source::Sdk => Harness::Codex and with_harness clearing version. Declined. Codex is chosen because input_excludes_cache() is false for it (src/harness.rs:76-78), which is the SDK records' cache-inclusive input after conversion. The trace's Harness label is then overwritten as sdk.
• 9: arm.runner and recorder as paired Options. Declined (judgement call). recorder is built from arm.runner (src/evals/experiment/execution.rs:202-207), so the two cannot diverge.
• 10: attempt layout strings. Declined (judgement call). The paths follow the existing attempt layout (session, output, request.json) that the version-1 runner contract in docs/eval-tasks.md publishes.
• 11: rename proxy::Accounting to Traffic. Declined (naming judgement). Its doc comment (src/proxy.rs:319-320) says what it holds.
• 12: repeated test assertions. Declined (judgement call). Each test asserts its own scenario's view inline, as the neighbouring Experiment tests do.

Spec

None.

Security

None. The one introduced finding, unbounded host-side reads from the record_command socket, is fixed in c2a8474d with regression tests.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Blast Radius: Experiments

Changes Custom runner evidence collection and Trial Void classification. Existing OTLP consumers use the same representation; runner protocol mistakes remain local to one Trial.

Built with claude 2.1.322 · claude-opus-5-5 · high

[JacobStephens2]

Delivery completed on agent-authored or reviewed head adfe490.