Skip to content

Install pinned Rust toolchains before Trial sessions - #1372

Merged
JacobStephens2 merged 3 commits into
mainfrom
issue-1369
Oct 26, 2026
Merged

JacobStephens2 merged 3 commits into
mainfrom
issue-1369

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Summary

Trials prepare their repository's Rust pin on the host before starting a session. Both workers share preparation, and a failed install produces a Void Trial with session_unavailable.

prepare historical repository
  rustup toolchain install --no-self-update
    share result for the same pin and listed components
  expose read-only Rust proxies to current and historical Arms
  start session with the repository's pinned tools

The guide and hand-written site pages describe the behavior. The Trial proxy allowlist is unchanged.

Closes #1369

Evidence

  • Before: cargo test --bin thirdshift trial_installs_its_historical_rust_pin_before_the_session -- --nocapture failed with pinned Rust 1.99.0 download refused by Trial proxy.
    After: the same regression passes for two concurrent Arms, with one host installation. It also clears RUSTUP_HOME before invoking Cargo to cover historical Arms.
  • Before: the real-rustup sandbox fixture could not find its installed pin. Spec review also reproduced Cargo 1.99.1 overriding a 1.99.0 pin when the host PATH named that toolchain directly.
    After: cargo test --bin thirdshift cargo_uses_the_repository_pin_with_a_fresh_sandbox_home -- --nocapture passes with that explicit PATH. The reviewer's direct-host-PATH probe now reports Cargo 1.99.0 and exits 0.
  • cargo test --bin thirdshift evals::experiment::execution::tests -- --nocapture: 21 passed, including failed installation producing session_unavailable without launching the Arm or running code checks.
  • cargo clippy --bin thirdshift -- -D warnings: passed.
  • The real historical Arm Experiment probe now exits 0 and observes Cargo 1.99.0 in both Arms. Its fixture deliberately stops the later model Grader; its assertions cover the two Trial toolchain observations.
  • npm run docs:check --prefix site and all 50 site tests: passed.
  • NEXTEST_TEST_THREADS=8 /usr/bin/time -p scripts/affected-tests main: site checks passed; Rust stopped after 764 passes at the existing OTLP timing test (a_stalled_receiver_cannot_delay_a_session_by_its_span_count). Under parallel load it measured 4.584s baseline and 6.808s stalled, exceeding its 2s margin. Wall time was 635.10s including site checks and compilation (26.33s Rust compilation).
  • The exact isolated rerun, cargo test --test otlp_endpoint a_stalled_receiver_cannot_delay_a_session_by_its_span_count -- --nocapture, passed. No test threshold was changed.
  • Full Linux CI passed on final commit 0ace3de2, including the complete Rust suite, formatting, clippy across all targets, and release-generator checks. Site, Console, macOS gate and release artifact checks also passed. Reused that successful exact-head full-suite result instead of repeating it locally.

Test seams: complete Trial execution through its completion callback, external rustup/Cargo commands, and the real bwrap sandbox launch interface with real rustup dispatch and isolated Rust executables.

Review coverage

Standards and Spec reviewers read every changed file, including the generated documentation hunk. Standards raised 0 findings; Spec raised 2, both addressed by runner-owned proxies and corrected PATH priority. No generated-output exclusions.

After the scoped follow-up, the final compatibility adjustment and stronger regression changed src/evals/experiment/execution.rs, src/evals/experiment/execution/tests.rs, and src/evals/experiment/execution/toolchains.rs. Both reviewers read earlier revisions of these files; their final amendments remain unread by the reviewers. The implementer verified the adjustment with the retained regression and the real historical Arm Experiment probe.

Unaddressed findings

  • Standards: none.
  • Spec: none.
  • Security: none. Guidance-mode review of the be154d7f..HEAD diff found no introduced finding: proxy wrappers quote paths with the POSIX single-quote helper (verified against hostile inputs), host installs consume only the operator-selected task pin via official rustup dist with --no-self-update and the proxy allowlist unchanged, the sandbox gains only read-only binds plus existing credential concealment, and each attempt uses a fresh prepared checkout so a session cannot influence another attempt's host-side read. No fix, no regression test, no pre-existing entries.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Reverting restores the previous preparation behavior. Successfully installed host toolchains remain available.

Blast Radius: Experiments

Host preparation can download pinned Rust toolchains and components. Failed provisioning voids affected Trials; sandbox network access is unchanged.

Built with codex codex-cli 0.164.0 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 8fcee47 into main Oct 26, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1369 branch October 26, 2026 12:45
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trials can't use the Rust toolchain their task's commit pins

1 participant