CI perf: PR CI — 23% of full PR runs re-test unchanged diffs after a merge-main push (~20–30k Linux job-min/day, feeds the ubuntu-latest backlog) #1373

Measurement

Window: 2026-10-09 16:16–20:16 UTC. Data: all 299 CI pull_request runs in the window (3 listing pages), plus job detail for 22 of them.

• 202 of the 299 runs were full runs: about 172 jobs, still running or lasting more than 6 min. The other 97 were draft or skip runs of 30 jobs, mostly skipped.
• 46 of the 202 full runs (23%) ran on a head commit that only merges main into the PR branch. The head commit's first line was Merge remote-tracking branch 'origin/main' into …, Merge branch 'main' into … or Merge main …. Two more were cancelled early.
• By hour: 32 of 95 PR runs created 18:00–19:00 were merge-main heads, and 14 of 41 between 19:00 and 20:00. This followed Repin live minimist@1.2.2 suites to republished patch 642d7f02 (#1293) #1301 landing on main at 18:02, when about 80 open agent branches re-synced.
• Examples:
  • 37983012523 (agent/v5-nuget-mapping)
  • 37982493866 (agent/v5-nuget-lock-reader)
  • 37981753120 (ci/depot-linux-runners)
  • 37979308752 (agent/v5-cargo-stale-rlib)
• A full PR run costs ~353 Linux + ~65 Windows job-min (median of the sampled full PR runs, n=12).

Where the time goes

• 46 × 353 ≈ 16,000 Linux + 3,000 Windows job-min in this 4-hour window, spent re-testing PR diffs that had not changed. Only main moved under them.
• They landed in the 18:00–20:00 burst that put ubuntu-latest queue wait at p50 16.6 / p90 18.9 min at 19:00. Merge-group runs took 77–81 min instead of ~22 (see CI perf: ci-ok — waits 9–18 min for an ubuntu-latest runner while ubuntu-22.04 starts in <1 min (merge-queue critical path, backlog hours) #1372 and CI perf (settings): merge queue builds 5 entries at a time — a 13-PR burst took 100 min to drain on a 21-min CI run (~21 min per PR past the 5th) #1248).

Root cause

• With a merge queue, a PR doesn't need to be up to date with main. The queue re-tests every entry on top of the latest main before it merges.
• Agent sessions still merge main into their branches after each landing, usually with no conflict. Each such push triggers the full PR tier again.
• concurrency (cancel-in-progress on PRs) cancels only the superseded run on the same branch. It does nothing about the burst across about 80 branches.

Proposed fix

Two parts. Either one helps on its own.

1. CI side (M), in ci.yml's preflight / dispatch-tests job:
   • If the PR head is a 2-parent merge whose second parent is on origin/main, check whether the PR's own diff is unchanged. Compare git diff $(git merge-base origin/main HEAD) HEAD with git diff $(git merge-base origin/main HEAD^1) HEAD^1 by git patch-id --stable.
   • If it is unchanged, look up the newest CI run for HEAD^1. That is one REST read: actions/runs?head_sha=<HEAD^1>&event=pull_request.
   • If that run concluded success, emit an output (for example reuse=true) that makes the heavy jobs skip (if: needs.dispatch-tests.outputs.reuse != 'true'), the same way the draft tier skips today.
   • Otherwise run the full tier. That covers conflict resolutions, a failed or missing previous run, and API errors.
   • clippy keeps running, so a compile break from main is still caught on the PR.
   • This mirrors the identical-SHA reuse ci: reuse the merge-queue verdict on main and trim coverage/sbt work #1355 proposes for push-to-main.
2. Process side (S): in AGENTS.md, tell agent sessions not to merge main into a PR branch unless it has a conflict or needs a change from main. The merge queue already tests against the latest main.

Expected saving

• About 23% of full PR runs during a post-landing burst. That is ~16k Linux + ~3k Windows job-min per 4 h at today's burst level.
• Conservatively ~20–30k Linux + ~4–6k Windows job-min/day, since bursts follow each batch of landings.
• Indirectly, it shrinks the ubuntu-latest backlog behind merge_group runs, which was the cause of the 77–81 min groups today.
• PR feedback latency on a merge-main push drops from ~22 min (unloaded) to about 2 min (clippy plus preflight).

Coverage and risk

• Every test still runs on the PR diff, on the previous head, and again in the merge queue against current main before merging. The skip applies only to a clean merge whose PR diff is byte-identical to a head that already passed.
• ci-ok and clippy keep their names. Skipped needed jobs already count as passing in ci-ok.
• Risk: a semantic conflict between main and the PR shows up only in the merge queue, about 20 min later, not on the PR. That is the same guarantee the queue provides today.
• Risk: patch-id equality on a merge with conflict resolution. Any resolved hunk changes the PR diff, so the patch-id differs and the full tier runs.

Effort

M for the CI side. S for the process side.

ROI

Saving ≈ (25k Linux ×1 + 5k Windows ×2) / 1000 = 35 × confidence 0.5 ÷ effort 2 = 8.8

Generated by Claude Code