Repository navigation
CI perf: yarn-classic 1.22.22 — one failed corepack yarn@1.22.22 fetch fails the leg and evicts a merge-queue entry (~30 MQ-min, ~94 job-min per hit) #1671
Description
Measurement
Window: 2026-10-21 20:16 → 2026-10-22 20:16 UTC. I fetched jobs for all 49 merge_group CI runs in that window.
- Merge-group run 39840895419 (Fix VEX product purl using raw PEP 440 version (#1589) #1653's group, created 19:31:50) failed in
yarn-classic 1.22.22. 38 of the 40 tests ine2e_redirect_yarn_classic_buildpassed. The other 2 panicked atcrates/socket-patch-cli/tests/common/yarn_classic_vex.rs:106:Ninety seconds earlier, the same job's rust-cache restore hitSKIP e2e_redirect_yarn_classic_build (empty-range) (yarn@1.22.22): `corepack yarn@1.22.22` unavailable (corepack absent or not fetchable) (SOCKET_PATCH_YARN_E2E_REQUIRED=1 forbids skipping) SKIP e2e_redirect_yarn_classic_build (other-name file:) (yarn@1.22.22): ... samelookup cache.depot.dev on 9.9.9.9:53: ... i/o timeout. So this was a DNS/network blip on the Depot host, not a product failure. The same code passed on the re-enqueue (39843511768, green). - Cost of this one hit:
- Fix VEX product purl using raw PEP 440 version (#1589) #1653 was evicted at 19:41:56, re-enqueued at 19:49:38 and merged at 20:02:05: about 20 min added.
- CI: gem variant apply tests assert JSON codes (#1089) #1642's group 39840901117 had already finished green (45.7 job-min). It was rebuilt as 39842369042 and merged at 19:53:28 instead of about 19:42: about 12 min added.
- Job-minutes: 48.6 for the failed run plus 45.7 for the discarded green run, about 94 Linux job-min.
- This is the only yarn-classic failure among the 49 merge-group runs and 12 PR runs in the window. Frequency is therefore n=1 so far. Depot DNS/network blips are recurring, though: see CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day) #1512, plus the crates.io and PHP-download retries that just landed (Ride out crates.io blips in coverage's cargo e2e #1654, Use the runner image's PHP in composer e2e legs #1659).
Where the time goes
yarn-classic 1.22.22 p50 2.15 / p90 3.34 min (n=55). The time isn't spent in this job. It's spent on the eviction: the failed group, the green groups built on top of it that get rebuilt, and the PR's re-enqueue.
Root cause
require_yarn_classic (crates/socket-patch-cli/tests/common/yarn_classic_vex.rs:115) probes corepack yarn@<release> --version once, with no retry. scripts/yarn-classic-vex-matrix.sh sets SOCKET_PATCH_YARN_E2E_REQUIRED=1, which turns a failed probe into a panic. That's the right call for catching a missing yarn. The problem is that one transient fetch failure from corepack, on a host whose DNS is briefly timing out, becomes a hard test failure and evicts the entry from the merge queue.
Proposed fix
- In
require_yarn_classic, retry the probe up to 3 times with backoff (2 s, 5 s) before callingskip. Panic only if every attempt fails, and print the last attempt's stderr, which the probe currently discards withStdio::null(). - In
scripts/yarn-classic-vex-matrix.sh, add one warm-upcorepack yarn@$release --versioninto the tests' sandboxCOREPACK_HOMEbefore the suites run, with the same retry. Later probes then read the cached release instead of fetching it again.
The same pattern probably exists in the yarn-berry, pnpm and rush corepack probes. Check them while you're in there, but keep the PR small.
Expected saving
- With 1 hit per ~50 merge-group runs (n=1 so far), and ~25 merges on a busy day, about 0.5 hits/day. That's ~15 merge-queue min/day amortized, on the queue's critical path, for the evicted PR and every green group behind it, plus about ~50 Linux job-min/day.
- No change to PR feedback latency or macOS.
Coverage and risk
- Every test still runs. A yarn release that really can't be fetched still fails after the retries, because
REQUIRED=1semantics are kept. - Worst case it adds ~7 s to a leg that is already failing.
- Required checks
ci-okandclippyare unchanged.
Effort
S (one helper and one script line).
ROI
(0.05k job-min + 15 MQ-min) × confidence 0.4 (single observation) ÷ effort 1 ≈ 6.0
Activity
mikolalysenko commentedon Oct 22, 2026
Working on this in the CI perf implementer.
Generated by Claude Code
mikolalysenko commentedon Oct 22, 2026
[agent] Triaged: p3 (CI-only). Open PR #1673 (
ci-perf/1671-yarn-classic-probe-retry) addresses this, so leaving it open for that PR.Generated by Claude Code