Skip to content

Refresh CodSpeed simulation counts and timings - #11231

Merged
connortsui20 merged 1 commit into
developfrom
ct/refresh-codspeed-timings
Nov 4, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/refresh-codspeed-timings

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

The snapshot used by #11180 predates the benchmark cleanup merges, so it overestimates simulation counts and target runtimes. Refresh both snapshots from the five successful simulation shards in develop run 41869576169 at 141abdc590.

Note that these are just heuristics, so this will never affect correctness / running of benchmarks.

Changes

Recorded simulation cases fall from 1,869 to 1,408, and total target execution time falls from 711s to 400s. The refreshed plan estimates 83-84s for each regular shard and 65s for the allocation-excluded shard. These are target execution estimates, not full CI job times.

Verified complete measurements for all 129 targets, exact target coverage, the 800-benchmark cap, and allocation-mode separation. The planner and git diff --check passed. No local tests or benchmarks were run. The pre-commit and pre-push hooks were bypassed as requested.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20

connortsui20 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

shard timings were:

Shard Build step Benchmark execution Total job
1 61s 84s 2m51s
2 61s 87s 2m56s
3 151s 82s 4m19s
4 144s 81s 4m12s
5 39s 64s 2m10s

We could add build time to this if we wanted...

anyways @joseph-isaacs since there are less benchmarks now (14 min total) we can reduce to 4 shards if you want. maybe 3, up to you

@codspeed

codspeed Bot commented Nov 4, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 21.72%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
✅ 1764 untouched benchmarks
⏩ 130 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation missing_stats[16] 92.7 µs 76.2 µs +21.72%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/refresh-codspeed-timings (2aabc08) with develop (bf3cfd0)

Open in CodSpeed

Footnotes

  1. 130 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 merged commit dd2eede into develop Nov 4, 2026
101 of 102 checks passed
@connortsui20
connortsui20 deleted the ct/refresh-codspeed-timings branch November 4, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants