Repository navigation
Install pinned Rust toolchains before Trial sessions - #1372
Merged
Merged
Conversation
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Trials prepare their repository's Rust pin on the host before starting a session. Both workers share preparation, and a failed install produces a Void Trial with
session_unavailable.The guide and hand-written site pages describe the behavior. The Trial proxy allowlist is unchanged.
Closes #1369
Evidence
cargo test --bin thirdshift trial_installs_its_historical_rust_pin_before_the_session -- --nocapturefailed withpinned Rust 1.99.0 download refused by Trial proxy.After: the same regression passes for two concurrent Arms, with one host installation. It also clears
RUSTUP_HOMEbefore invoking Cargo to cover historical Arms.After:
cargo test --bin thirdshift cargo_uses_the_repository_pin_with_a_fresh_sandbox_home -- --nocapturepasses with that explicit PATH. The reviewer's direct-host-PATH probe now reports Cargo 1.99.0 and exits 0.cargo test --bin thirdshift evals::experiment::execution::tests -- --nocapture: 21 passed, including failed installation producingsession_unavailablewithout launching the Arm or running code checks.cargo clippy --bin thirdshift -- -D warnings: passed.npm run docs:check --prefix siteand all 50 site tests: passed.NEXTEST_TEST_THREADS=8 /usr/bin/time -p scripts/affected-tests main: site checks passed; Rust stopped after 764 passes at the existing OTLP timing test (a_stalled_receiver_cannot_delay_a_session_by_its_span_count). Under parallel load it measured 4.584s baseline and 6.808s stalled, exceeding its 2s margin. Wall time was 635.10s including site checks and compilation (26.33s Rust compilation).cargo test --test otlp_endpoint a_stalled_receiver_cannot_delay_a_session_by_its_span_count -- --nocapture, passed. No test threshold was changed.0ace3de2, including the complete Rust suite, formatting, clippy across all targets, and release-generator checks. Site, Console, macOS gate and release artifact checks also passed. Reused that successful exact-head full-suite result instead of repeating it locally.Test seams: complete Trial execution through its completion callback, external rustup/Cargo commands, and the real bwrap sandbox launch interface with real rustup dispatch and isolated Rust executables.
Review coverage
Standards and Spec reviewers read every changed file, including the generated documentation hunk. Standards raised 0 findings; Spec raised 2, both addressed by runner-owned proxies and corrected PATH priority. No generated-output exclusions.
After the scoped follow-up, the final compatibility adjustment and stronger regression changed
src/evals/experiment/execution.rs,src/evals/experiment/execution/tests.rs, andsrc/evals/experiment/execution/toolchains.rs. Both reviewers read earlier revisions of these files; their final amendments remain unread by the reviewers. The implementer verified the adjustment with the retained regression and the real historical Arm Experiment probe.Unaddressed findings
be154d7f..HEAD diff found no introduced finding: proxy wrappers quote paths with the POSIX single-quote helper (verified against hostile inputs), host installs consume only the operator-selected task pin via official rustup dist with--no-self-updateand the proxy allowlist unchanged, the sandbox gains only read-only binds plus existing credential concealment, and each attempt uses a fresh prepared checkout so a session cannot influence another attempt's host-side read. No fix, no regression test, no pre-existing entries.Security review outcome
No unaddressed introduced findings.
Merge Danger
Door: two-way
Reverting restores the previous preparation behavior. Successfully installed host toolchains remain available.
Blast Radius: Experiments
Host preparation can download pinned Rust toolchains and components. Failed provisioning voids affected Trials; sandbox network access is unchanged.
Built with codex codex-cli 0.164.0 · gpt-6.1-sol · xhigh