[JacobStephens2]

Summary

non-nightly Rust CI (rust-tests.yml): four independent cached workers per required architecture
  -> per-test JUnit timings + full/partition nextest inventories (14-day artifacts)
  -> complete: exact once-only coverage + execution check, partition balance
  -> stable test / test-linux-arm gates (always(); only `success` passes)
nightly: one complete contributor `cargo test` worker per architecture
dispatch: benchmark=true  -> one/two/four workers on both runners + benchmark-report
          probe=failure   -> partition 1 of 4 fails; probe=skipped -> all skipped

Per-test timings are kept for 14 days, with summaries of per-binary totals, the slowest tests and partition balance. An opt-in benchmark compares one, two and four workers at the same revision. Event policy, ARM opt-in, cancellation, tooling checks and Release dependencies are unchanged. No test exclusions, retries or deadline changes are added.

Closes #2031

Evidence

Execution evidence

• Before: the benchmark-report pipeline, run against warm benchmark run 43060539830, failed with unknown flag: --slurp, then ci-timings: Expecting value: line 1 column 1 (char 0). Newer gh also rejects --slurp together with --jq.
  After: gh api --paginate ".../runs/43060539830/jobs?per_page=100" --jq '.jobs[]' | jq -s . > jobs.json; python3 scripts/ci-timings.py benchmark <dir> prints the documented warm table, for example benchmark (ubuntu-24.04, [1,2,3,4]) | 514.0 | 31.05 | success and benchmark (ubuntu-24.04-arm, [1]) | 1342.0 | 22.22.
• Before: the only hosted failure probes ran the superseded one-partition workflow at deb46450.
  After: probe=failure 43131195970 ran at d8e6b3cb. run (1) failed on each architecture while runs 2–4 succeeded, and complete, test and test-linux-arm all failed. probe=skipped 43131198843 also failed all three gates.
• Before: temporarily renaming the Compile tests step makes the new test_benchmark_phases_follow_the_actual_workflow_step_names fail (phase='compilation').
  After: with the real step names, python3 -m unittest discover -s scripts -p 'test_*.py' passes all 64 tests.
• The coverage verifier passes all 28 retained benchmark worker reports. Every test is assigned and executed exactly once: 4,233 x86 tests and 4,232 ARM tests, with 97 ignored on each. The verifier has no fixed suite count built in.
• Final-head PR CI 43131649638 on 13290404 passed. That includes all four partitions and the complete coverage check on both architectures (ARM opted in), the test and test-linux-arm gates, tooling, and the PR Release checks.
• scripts/affected-tests main selected the full suite and fell back to cargo test. One test failed under load: triage_pass::recommendation_only_pass_reserves_without_claiming_or_publishing_and_notifies ("the agent never started"). It passes when re-run alone with cargo test --test triage_pass recommendation_only_pass_reserves_without_claiming_or_publishing_and_notifies -- --exact. This branch changes no Rust source or tests.

The cold and warm benchmarks use the same revision, deb46450, on hosted ubuntu-24.04 and ubuntu-24.04-arm.

Architecture	One-worker elapsed s	Four-worker elapsed s	One-worker runner min	Four-worker runner min
x86_64	1418–1608	514	23.48–26.68	31.05–31.53
ARM	1342–1349	490–514	22.22–22.30	28.65–29.88

Four workers took 62–68% less elapsed time than one, at 16–34% more runner minutes. Compared with two workers, they took 36–44% less time at 9–17% more runner minutes. The differences exceed the observed sample variance, but two samples don't promise a typical speedup. docs/ci-performance.md breaks out start delay, setup, compilation, cache and report transfer, and execution. It also explains why the workers build independently: no build archive was introduced or benchmarked.

Review coverage

Reviewed against refs/remotes/origin/main (merge-base acb42c1f). Standards raised 11 findings: 2 doc drifts and 9 judgement-call smells. Spec raised 6. Both reviewers read all eight changed files. Unread changed files: none. Generated-output exclusions: none.

Fixed:

• Stale job-limit table and ARM paragraph (Standards 1–2, Spec 4).
• Hard-coded benchmark phase names, now tied to the workflow by a test (Standards 7).
• Double JUnit parse in verify (Standards 8).
• Test that ran its subprocess twice (Standards 11, part).
• benchmark-report pagination (Spec 2).
• Single-partition hosted failure probe (Spec 1).
• --locked removed (Spec 5).

Unaddressed findings

Standards

• 3: ARM opt-in condition duplicated between rust-arm and test-linux-arm (.github/workflows/ci.yml). GitHub Actions can't share a named expression across jobs. The gate needs always() plus the same condition, so it is skipped rather than failed when ARM isn't opted in. This mirrors the existing ARM policy from Release an ARM Linux build so the installer and thirdshift update work on aarch64 Linux #1663.
• 4: partitions and nightly expressions repeated per architecture (ci.yml, rust-x86 and rust-arm). They are inputs to a reusable workflow, and a combined input would hide nightly's cargo test choice inside a partition list.
• 5: repeated probe/nightly step conditions (.github/workflows/rust-tests.yml). Splitting them into separate jobs would add runners and duplicate the toolchain setup. This is a judgement call.
• 6: repeated report path and partition count (rust-tests.yml, env: blocks and complete). strategy.job-total isn't visible to the complete job, which is why PARTITIONS is counted there.
• 9: subparser variable names (scripts/ci-timings.py, main). Cosmetic, in a 170-line CLI.
• 10: one module covers summary, coverage and benchmark (scripts/ci-timings.py). The three commands share the JUnit and inventory parsing, and the module is small.
• 11 (rest): chained scenarios and local imports in the coverage test (scripts/test_ci_timings.py). Each scenario restores the state it changed. The import stays local because scripts/ is only importable under unittest discovery.

Spec

• 3: nightly and main-push paths not run on hosted CI. schedule and push-to-main runs use main's workflow file, so this PR branch can't exercise them before merge. The gates use the same needs.*.result == success command that test_actual_workflow_gates_reject_failed_cancelled_skipped_or_missing_work runs for success, failure, cancelled, skipped and missing results. The event-policy expressions for push and unlabeled PRs are unchanged from main.
• 6: nextest pinned to 0.9.151 (rust-tests.yml, taiki-e/install-action@v2). ci-timings.py verify parses the JSON from nextest list --message-format json. The pin keeps that format, and the benchmark revision's comparisons, stable. docs/ci-performance.md records the version.
• 1 (remainder): JUnit upload after a real test failure hasn't run on hosted CI, because the probe fails before tests run. That upload step uses if: always() and if-no-files-found: error. This gap is stated in docs/ci-performance.md.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

The workflow and partition count can be reverted.

Blast Radius: CI

Four workers trade more runner minutes for less elapsed time, and scarce runner capacity can add queue delay. The stable architecture gate can pass only when every required partition and its exact coverage check succeed.

Built with claude 2.1.314 · opus · default effort