[Mikola Lysenko (mikolalysenko)]

Self-picked from the CI performance dashboard (#1182): no ci-perf issue was eligible this run (#1745 is covered by #1747; #1248 and #1512 are settings issues).

Problem

coverage is the last job before ci-ok in PR and merge-queue runs (the dashboard has coverage (2) last in 7 of 9 merge_group runs). Its two shards use the count split from scripts/ci-test-shard.py: shard 1 gets the unit tests plus a third of the integration targets, shard 2 gets the other two thirds. Shard 2 is consistently the slow one.

Last 8 green CI runs (10-28 02:30–04:27 UTC):

run	coverage (1) job / test step	coverage (2) job / test step
40631937407	3.17 min / 167 s	4.80 / 266 s
40631937411	3.20 / 170	4.88 / 270
40630907173	3.15 / 164	4.50 / 246
40630758413	3.40 / 179	6.23 / 351
40627940732	3.10 / 165	4.33 / 239
40626941926	3.07 / 160	4.48 / 246
40625835496	3.12 / 164	4.37 / 236
40623064826	3.20 / 168	4.35 / 239

Shard 2 runs 1.2–1.4 min longer (p50 4.49 vs 3.16 min). In run 40630907173 the whole run took 4.8 min and coverage (2) (4.50) set its end; the next-longest job was 3.12 min.

Change

scripts/ci-test-shard.py: when CI_TEST_COVERAGE=1 and CI_TEST_TIMINGS is unset, partition on the checked-in runtime sample scripts/ci-test-durations.json (the longest-first split the Windows test shards already use) instead of the count split. An explicit CI_TEST_TIMINGS still wins, and an empty one selects the count split. Every integration target still runs in exactly one shard. New test test_coverage_shards_default_to_the_timing_sample checks both shards in all three modes.

No workflow file changes: #1747 is open on ci.yml. The ci.yml comments on coverage (line ~615) and test (~372) don't mention that coverage now uses the sample; I'll update them once #1747 merges.

Expected saving

The sample was measured on Windows. A two-point fit of the Linux test steps to it (167 s and 246 s ≈ 33 s + 0.27 × Windows seconds) predicts ~205–210 s per shard, so the slower shard's job drops from ~4.5 to ~3.8 min: ~0.6 min per PR run and per merge-queue run on the critical path. Job-minutes are about the same (the same work, split differently). The fit has two points for two parameters, so it's a guess until this PR's own run measures it (below).

Measured result

No saving. Closing this PR. Run 40633036954 (head 43902c2). I re-ran each coverage shard once to get a second sample:

	baseline (count split, 8 runs)	this PR, attempt 1	this PR, re-run
coverage (1) job / test step	3.07–3.40 min / 160–179 s (p50 3.16 / 166)	3.77 / 203 s	3.52 / 187 s
coverage (2) job / test step	4.33–6.23 min / 236–351 s (p50 4.49 / 246)	4.72 / 258 s	4.40 / 238 s

The timed split moves 38 targets (156 → 118) to shard 1. That made shard 1 20–37 s slower, but shard 2 stayed in its baseline range, so the slower shard (shard 2 both times) didn't get any faster. The Windows sample's per-target costs don't match instrumented Linux: what makes shard 2 slow on Linux is a few targets that the sample rates as cheap. A real fix needs a Linux coverage timing sample, which means per-target times from coverage job logs. This routine can't download those (the Actions log host is blocked from its sandbox).

Where each test runs

No test moves between events. Both shards still run every integration target once, in the same coverage job, on the same events.

Risk

Low. If the Windows sample's proportions don't hold on Linux, the imbalance moves to shard 1 instead; this PR's run will show it. Revert, or set CI_TEST_TIMINGS: '' on the coverage step, to get the count split back. Targets not in the sample (10 today) are costed at the sample's default.

Validation: python3 -m unittest discover -s scripts/tests (375 tests, OK).

Code review (/code-review high)

Fixed in 43902c2: an empty CI_TEST_TIMINGS now means the count split (it fell back to the sample before); the new test uses the sample's real target names and checks both shards; the docstring no longer hardcodes measurements.
Not fixed: (a) "set CI_TEST_TIMINGS in ci.yml instead of a script default" is cleaner, but #1747 has ci.yml open, and the fleet keeps one PR per workflow file; (b) the stale ci.yml comments, same reason, follow-up after #1747; (c) the Windows sample may not match Linux. That's the open question, and the measured result decides it; (d) test_coverage_mode_comes_from_the_environment puts its single target on shard 2 under either split, so nothing changes there.

https://claude.ai/code/session_011f9aePpyeqhTUQcsKL5coJ

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 43902c2. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Closing: two samples of this PR's coverage shards show no saving. coverage (2) ran 4.72 and 4.40 min against a 4.49 p50 baseline, while coverage (1) got 20–37 s slower. The Windows timing sample doesn't model instrumented Linux targets. Details are in the PR body under Measured result. Retry this with a Linux coverage timing sample.

---

Generated by Claude Code