Skip to content

ci(windows): run each Windows test group on its own runner - #4468

Merged
miguel-heygen merged 1 commit into
mainfrom
ci/windows-four-test-lanes
Sep 25, 2026
Merged

miguel-heygen merged 1 commit into
mainfrom
ci/windows-four-test-lanes

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

The Windows tests are the last required check to finish on most PRs: they were last on 28 of the last 40 merged PRs.

They ran on two runners, and each runner did its work in series:

  • First runner: a Studio shard, then the core, player and shader-transitions tests.
  • Second runner: the other Studio shard, then the engine and CLI tests, the runtime contract, and the Windows-only file checks.

Each group now gets its own runner: studio-1, studio-2, core, and engine-cli. They are the same tests, with the same worker caps, on four runners instead of two. The required check, Tests on windows-latest, keeps its name and still fails when any lane does.

The Studio halves and the engine and CLI suites still never share a runner. #3579 found that sharing one runner is what caused load timeouts.

Before and after, from real CI runs

The Windows workflow was dispatched on this change and on an unchanged copy of main at the same moment, three times, so runner queueing hit both alike.

Pair This change Main Change
1 529 s 758 s -229 s
2 586 s 676 s -90 s
3 558 s 718 s -160 s
Median 558 s 718 s -22%
  • Same tests: every run of this change passed 15,994 vitest tests on Windows. Two of the main runs passed the same number and one passed 15,993.
  • Cost: about 7 more Windows runner-minutes per run, from the two extra runners' setup. That is accepted for the shorter wait.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at fd751627. No test group is dropped, and the required checks still report.

Coverage. The old studio-core lane is now studio-1 plus core, and the old studio-engine-cli lane is now studio-2 plus engine-cli (windows-render.yml:400-410). The Studio and package steps are gated on matrix.studio_shard and matrix.packages (:470, :475). The 4 Windows file checks moved to engine-cli (:482-497), and engine-cli is the only lane that sets runtime_contract (:502).

  • The vitest summary lines add up the same: 8 = 7 + 1 and 4 = 1 + 3.
  • Summed "Tests N passed" is 15,994 in all 3 PR runs. The controls give 15,994, 15,993 and 15,994. The one-test gap is inside @hyperframes/cli on a control run, so it's noise in the control, not something this PR causes.

Required checks. Ruleset main (14211637) requires Render on windows-latest and Tests on windows-latest.

  • Render on windows-latest (:81) is untouched.
  • Tests on windows-latest is still the aggregator (:508-540, needs [changes, test-windows-lane]). Its logic is unchanged.
  • The per-lane names were never required, so renaming them can't strand a required check. Nothing in .github/, scripts/ or docs/ references the old lane names.
  • A path-filter skip still reports the aggregator as skipped under its required name, which satisfies the rule, same as before. A preflight failure still fails the aggregator through always().

Timing. The paired runs measure 529/587/558 s for this change against 758/677/719 s for the control on the parent a95cb96a, which matches the body. The slowest lane drops from 589-677 s to 450-503 s.

Cost, FYI. Runner time goes from 1174-1251 s to 1544-1723 s per run, about 7.6 minutes more, because install and build now run 4 times. That's free on a public repo. The real cost is 5 concurrent Windows jobs per run instead of 3 against the org's shared Windows limit, which may add queueing on busy days. fail-fast: false is unchanged, and no caches or artifacts are shared between lanes.

Nit: the step name Run Studio shard ${{ matrix.studio_shard }} (:469) renders as "Run Studio shard " on core and engine-cli. The step is skipped on those lanes, so it only affects how the name reads.

— Rames

@miguel-heygen
miguel-heygen added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 165707b Sep 25, 2026
58 checks passed
@miguel-heygen
miguel-heygen deleted the ci/windows-four-test-lanes branch September 25, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants