Repository navigation
ci(windows): run each Windows test group on its own runner - #4468
Conversation
jrusso1020
left a comment
There was a problem hiding this comment.
Approve at fd751627. No test group is dropped, and the required checks still report.
Coverage. The old studio-core lane is now studio-1 plus core, and the old studio-engine-cli lane is now studio-2 plus engine-cli (windows-render.yml:400-410). The Studio and package steps are gated on matrix.studio_shard and matrix.packages (:470, :475). The 4 Windows file checks moved to engine-cli (:482-497), and engine-cli is the only lane that sets runtime_contract (:502).
- The vitest summary lines add up the same: 8 = 7 + 1 and 4 = 1 + 3.
- Summed "Tests N passed" is 15,994 in all 3 PR runs. The controls give 15,994, 15,993 and 15,994. The one-test gap is inside
@hyperframes/clion a control run, so it's noise in the control, not something this PR causes.
Required checks. Ruleset main (14211637) requires Render on windows-latest and Tests on windows-latest.
Render on windows-latest(:81) is untouched.Tests on windows-latestis still the aggregator (:508-540, needs[changes, test-windows-lane]). Its logic is unchanged.- The per-lane names were never required, so renaming them can't strand a required check. Nothing in
.github/,scripts/ordocs/references the old lane names. - A path-filter skip still reports the aggregator as skipped under its required name, which satisfies the rule, same as before. A preflight failure still fails the aggregator through
always().
Timing. The paired runs measure 529/587/558 s for this change against 758/677/719 s for the control on the parent a95cb96a, which matches the body. The slowest lane drops from 589-677 s to 450-503 s.
Cost, FYI. Runner time goes from 1174-1251 s to 1544-1723 s per run, about 7.6 minutes more, because install and build now run 4 times. That's free on a public repo. The real cost is 5 concurrent Windows jobs per run instead of 3 against the org's shared Windows limit, which may add queueing on busy days. fail-fast: false is unchanged, and no caches or artifacts are shared between lanes.
Nit: the step name Run Studio shard ${{ matrix.studio_shard }} (:469) renders as "Run Studio shard " on core and engine-cli. The step is skipped on those lanes, so it only affects how the name reads.
— Rames
What
The Windows tests are the last required check to finish on most PRs: they were last on 28 of the last 40 merged PRs.
They ran on two runners, and each runner did its work in series:
Each group now gets its own runner: studio-1, studio-2, core, and engine-cli. They are the same tests, with the same worker caps, on four runners instead of two. The required check,
Tests on windows-latest, keeps its name and still fails when any lane does.The Studio halves and the engine and CLI suites still never share a runner. #3579 found that sharing one runner is what caused load timeouts.
Before and after, from real CI runs
The Windows workflow was dispatched on this change and on an unchanged copy of main at the same moment, three times, so runner queueing hit both alike.