Run e2e suites that share a toolchain in one leg

[Mikola Lysenko (mikolalysenko)]

Fixes #1178 (first slice: rows that already share one toolchain setup; see "Not in this PR").

Problem

#1178: a CI run spawns ~134 e2e legs, 145 of 157 distinct legs have p50 < 2.5 min, and ~64% of a small leg is per-job overhead (Set up job, checkout, downloading and unpacking the e2e binaries, toolchain setup). Billing rounds every job up to a whole minute. The jobs also take up Linux runner slots, which pushes the Linux queue p90 to 12.8 min.

Many e2e rows differ only in suite. They install the same composer line, Ruby + bundler era, bun release or Maven line, so each pays that overhead two or three times.

Change

.github/workflows/ci.yml only. The shared "Run e2e tests" step already loops for suite in $E2E_SUITE and fails any suite that runs zero tests. So rows whose keys match except for suite are merged into one row with a space-separated suite:

family	job	legs before → after
composer vendor + redirect (1, 2.2, 2)	e2e	6 → 3
gem redirect + vendor (7 bundler eras)	e2e	14 → 7
bun redirect + vendor + mode_migration (1.1.45, 1.2.23, 1.3.14, 1.4.2)	e2e	12 → 4
Maven redirect + vendor (3.6.3, 3.8.9, 3.9.16, 4.0.0-rc-6; 3.9.3/3.9.4 stay redirect-only)	e2e	10 → 6
bun redirect + vendor + mode_migration (1.4.2)	e2e-windows	3 → 1
gem redirect + vendor (bundler 2.7.2)	e2e-full	2 → 1

Each of these families' setup steps keys on the toolchain key (composer, bundler, bun, jvm_tool), not on suite. e2e_bun_lockb keeps its own legs because the SOCKET_PATCH_BUN_LOCKB_* env gates check matrix.suite == 'e2e_bun_lockb'. e2e_safety_pnpm + e2e_redirect_rush_sim also stay separate, because the pnpm/npm setup steps key on the suite name. e2e-macos is left alone (#1176).

Where each test runs

Nothing moves. A script expanded every row into (job, suite, toolchain keys) cells. That gives 207 cells before and 207 identical cells after, in the same jobs and on the same events. Per-suite failure reporting is unchanged: the loop runs every suite, tees each suite's log, and emits ::error:: naming the suite that ran no tests.

Expected saving

Baseline is merge_group run 37876502927 (Linux + Windows legs of these families):

• Before: 45 legs, 25.5 job-min, 45 billed min. Only 11.8 min of that is the "Run e2e tests" step. Longest leg: 0.95 min.
• Expected after: 21 legs, ~18 job-min, ~21 billed min. Merged legs stay ~1–1.5 min, so they don't come near the critical path.
• That is 24 fewer jobs and Linux/Windows runner slots per run, ~7 actual and ~14–24 billed job-min per run. At ~260 runs/day that is ~1,800 actual and ~3,600–6,000 billed job-min/day, plus less Linux queue pressure. This is about a fifth of the CI perf: ci.yml e2e — ~120 sub-2.5-min Linux/Windows legs per run, 64% setup overhead (~12,000 job-min/day) #1178 target. The rest needs per-version loops (below).

Measured result

This PR's CI run is 37894197827 (pull_request, green). The baseline is merge_group run 37876502927. Both cover the Linux and Windows legs of the merged families:

	before	after
legs	45 (42 Linux + 3 Windows)	21 (20 + 1)
job-min	25.5	18.4 (-28%)
billed min (rounded up per job)	45	31 (-31%)
"Run e2e tests" min	11.8	11.5 (same tests)
longest merged leg	0.95	1.47 min

So each run starts 24 fewer jobs, saving ~7 actual and ~14 billed job-min. Every merged leg passed, and the loop's per-suite 0 passed check confirms that each listed suite ran tests. Total jobs in the run went from 213 to 161, but that number also reflects PR-vs-merge_group differences elsewhere in the workflow. The profiler will verify the merge_group effect after merge.

Validation

• python3 -m unittest discover -s scripts/tests: 281 tests OK. This covers test_ci_e2e_tiers: every row is a test target, e2e-full only adds releases to PR suites, and the bundle covers every OS.
• actionlint 1.7.7 and zizmor (offline): same findings as main, nothing new.
• Merges cleanly with the other open PRs that touch ci.yml (Cut release-test wait time and recover Maven downloads #1166, Fix macOS Composer legs red on main via setup-php 2.40.0 #1215). The hunks don't overlap.

Not in this PR

The rest of #1178 means merging rows that differ by toolchain version (uv/poetry/pdm/hatch VEX rows, vlt eras, dotnet). That needs a per-version setup loop inside a leg, so it is left for a follow-up.

Risk

Low. One failing suite now marks its merged leg red, and a re-run repeats 2–3 suites (each ~0.1–0.7 min). The job title lists every suite in the leg. Required checks ci-ok / clippy are unchanged, and ci-ok depends on the e2e jobs as a whole, not on leg names.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bp6XLBUDPLHroKWxwtXi25

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1cb87af. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: labeled Ready for review at head 1cb87af1f407ec94de1b6b11de2dee3c3d31c92f.

• CI: all checks green on this head (success/skipped only).
• Bugbot: reviewed 1cb87af1, no new issues; no unresolved review threads.
• Mergeable against main, no CHANGELOG.md change.
• Slack announcement not sent this run (Slack send unavailable); the next run will retry.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Dequeued on a failure this PR does not cause. In merge_group run 37904060236, hosted-e2e failed at its Install uv step, before any test ran. That job is untouched by this diff, which only changes e2e / e2e-windows / e2e-full matrix rows. Every merged e2e leg in that run passed or was cancelled by the queue reset. The earlier queue run 37902416871 was cancelled by a queue reset, with no failures of its own. I found no fix to port; it looks like a transient toolchain download failure. The head (1cb87af) is unchanged and green on its PR run, so it can be re-queued as is. I did not re-run or re-queue anything.

---

Generated by Claude Code