Ride out crates.io blips in required cargo e2e legs

[Mikola Lysenko (mikolalysenko)]

Problem

The cargo 1.93.1 lock-v4 (ubuntu-latest) leg (cargo-vex-matrix, which runs on PRs and in the merge queue) failed on PR #1449, run 38307581268 attempt 1. It passed when the same SHA was re-run. The failing test was e2e_vendor_cargo_build::cargo_rollback_after_agent_to_vendored_restores_cache_behind_cargo_vendor_dir, in its baseline fixture build:

baseline `cargo build` failed (crates.io unreachable?):
error: failed to download from `https://static.crates.io/crates/cfg-if/1.0.5/download`
  [7] Could not connect to server (Failed to connect to static.crates.io port 443 after 0 ms ...)
 (and SOCKET_PATCH_CARGO_E2E_REQUIRED is set, so this leg must not skip)

Each of the ~120 tests in these suites does a fresh-CARGO_HOME fetch from crates.io. A short network blip fails the leg, and in the merge queue it would evict the entry and cost a full rebuild.

Root cause

Cargo treats a connect failure as spurious and retries it, but the default net.retry = 3 gives up after about 12s. The test took 13s. A blip longer than that fails the build. With REQUIRED=1 set, the harness turns that failed build into a hard failure.

Fix

cargo_e2e_matrix::cargo_command is the one helper every real-cargo suite uses (e2e_redirect_cargo_build, e2e_redirect_cargo_shapes, e2e_vendor_cargo_build, mode_migration_cargo, e2e_safety_cargo_build). When the leg is required, it now sets CARGO_NET_RETRY=10 on its children, which gives about 80s of cargo's own backoff. An ambient CARGO_NET_RETRY still wins. Optional local runs keep the default, so an offline machine still skips quickly. Only errors cargo itself classifies as transient are retried. Checksum, resolver and --locked failures still fail immediately.

The one test that needs a fast network failure (the old-cargo 127.0.0.1:9 proxy check in e2e_vendor_cargo_build.rs) builds its own Command with CARGO_NET_RETRY=0, so it is unaffected.

Proof

• Retry window, with cargo fetch of cfg-if through a refused proxy (the same [7] Could not connect ... after 0 ms error): retry=3 fails after 12s (3 spurious warnings). retry=10 fails after 80s.
• Simulated 20s blip (a local forwarder that starts accepting connections after 20s): retry=3 fails after 12s with the CI error. retry=10 succeeds after 21s (4 spurious retries, then Downloaded cfg-if v1.0.5).
• SOCKET_PATCH_CARGO_E2E_REQUIRED=1 cargo test -p socket-patch-cli --test e2e_redirect_cargo_build --test e2e_redirect_cargo_shapes --test e2e_vendor_cargo_build --test mode_migration_cargo: 22 + 31 + 39 + 29 passed.
• rustfmt --check is clean on the touched file. Clippy (-D warnings) reports nothing in the touched file. The same targets already fail clippy on unmodified main in tests/common/mod.rs and tests/prebuilt_common/mod.rs (11 errors either way).

No tests removed or moved.

Not covered: a 2-minute runner DNS outage (main push run 38182319501, rustup toolchain install stable failed 4 times on Temporary failure in name resolution). That is runner infrastructure, and no retry window here is meant to absorb it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GTcGpX2RVXH9BXATkDJLdm

---

Note

Low Risk
Test-harness-only change to env vars on spawned cargo processes; no production runtime or security impact.

Overview
Required cargo e2e legs now set CARGO_NET_RETRY=10 on child cargo invocations (via shared cargo_e2e_matrix::cargo_command) so transient crates.io connect failures get ~80s of Cargo’s built-in backoff instead of the default ~12s, which was flaking CI when SOCKET_PATCH_CARGO_E2E_REQUIRED forbids skips.

An explicit ambient CARGO_NET_RETRY still wins; optional local runs keep Cargo’s default for fast offline skips. A new REQUIRED_NET_RETRY constant and expanded helper docs describe the behavior.

^{Reviewed by Cursor Bugbot for commit 641708e. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 641708e. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Ready for review at 641708e77e334acc2e2f0e97e4667d280db65bfb.

• CI: all green on this head (35 pass + 19 skipped), no failures.
• Bugbot: reviewed 641708e, no new issues; no unresolved review threads.
• Mergeable, no CHANGELOG.md changes.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does: On required cargo e2e legs (SOCKET_PATCH_CARGO_E2E_REQUIRED), every cargo child spawned through the shared cargo_command() helper now gets CARGO_NET_RETRY=10, unless the environment already sets a non-empty CARGO_NET_RETRY. Cargo's default of 3 retries gave up after about 11s on a crates.io connect blip and failed the leg. With 10 it rides out about 80s. Test-only, one file.

Risk: low. Test harness only, and gated on required legs. It adds one env var and removes nothing, so hermetic scrubbing and the run_socket invocations are unchanged. The negative tests still fail as before: cargo doesn't retry a checksum mismatch or the wiremock 404 fallback, and no mock serves 5xx.

Look here:

• cargo_e2e_matrix/mod.rs:89: REQUIRED_NET_RETRY and the backoff rationale.
• cargo_e2e_matrix/mod.rs:105: the gated cmd.env. An explicit ambient value wins.

Verified:

• Read the full diff and every caller of cargo_command: e2e_redirect_cargo_build, e2e_redirect_cargo_shapes, e2e_vendor_cargo_build, mode_migration_cargo and e2e_safety_cargo_build. They all go through this helper. The one direct cargo_run, in e2e_safety, runs --offline --frozen and needs no network.
• CI on 641708e: 35 success, 19 skipped of 54; ci-ok and clippy green. Cursor Bugbot passed. No unresolved review threads. CHANGELOG.md untouched.

Changes I made: none.

Open questions (non-blocking): a silently dropped connection could now take several minutes per transfer (11 tries x the 30s http.timeout) before it fails, so a real outage would show up as a job timeout instead of a clear "Could not connect". Also setting CARGO_HTTP_TIMEOUT lower on required legs would bound that, but it's optional.

Auto-merge (squash) is armed, so approving sends it straight to the merge queue.

---

Generated by Claude Code