Skip to content

Balance CodSpeed simulation shards by runtime - #11180

Merged
connortsui20 merged 1 commit into
developfrom
ct/codspeed-runtime-shards
Nov 4, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/codspeed-runtime-shards

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

The count-based planner fills only 3 of the 5 fixed shard slots and keeps most packages together, although benchmark counts do not predict execution time. Balance recorded target runtimes across all 5 slots, splitting large packages at benchmark-target boundaries.

The first CI run reduced the longest shard job from 10m03s to 5m07s. All 5 shards passed. Small packages stay together to limit repeated builds, and allocation-excluded targets remain separate.

Changes

  • Record target durations from develop run 41711616261, commit 24cd0eac8f, with a command to refresh the snapshot from simulation logs. Measure each complete binary execution, including startup and fixture construction, and use the median when several runs are supplied.
  • Allocate spare slots to the slower allocation-mode pool and assign the longest targets first. Split packages that exceed a shard's runtime budget or benchmark-count cap.
  • Preserve the 800-benchmark cap, allocation-mode separation, and complete benchmark binaries. A new target gets a five-second estimate and a warning until its runtime is recorded.
  • Add coverage for runtime balancing, package splitting, allocation-mode separation, count limits, deterministic output, and duration extraction.
Observed CI shard times
Shard Reference develop run This PR
1 10m03s 4m44s
2 6m13s 3m43s
3 2m43s 4m42s
4 6s, unused 5m07s
5 6s, unused 2m27s

Compared run 41711616261 with run 41716056593 on 863c58f026. These are elapsed CI job times including setup, builds, measurement, and uploads. Exact timings depend on cache and runner state.

CI passes the planner, all 5 simulation shards, Python formatting, linting, type checks, and all 58 script tests, including the 17 shard-planner tests. Other CI jobs are still running.

Executed the planner locally to inspect the proposed assignment. Tests and linting were left to CI. Local hooks were bypassed under the requested CI-only workflow.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@codspeed

codspeed Bot commented Nov 3, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 1876 untouched benchmarks
⏩ 481 skipped benchmarks1
🗄️ 47 archived benchmarks run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation compare[1] 54.6 µs 63.9 µs -14.63%
⚡ CPU Simulation compare[1] 59.9 µs 50 µs +19.97%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/codspeed-runtime-shards (863c58f) with develop (1012d0c)

Open in CodSpeed

Footnotes

  1. 481 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 47 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the annoying part of this is having to rebalance the timings and add new ones as you add benchmarks

@connortsui20

Copy link
Copy Markdown
Member Author

Maybe we can have a bot that does this automatically? Regardless this is a much faster process so I would like to merge this (or something similar). I'll hold off on this until I get into office

@robert3005

Copy link
Copy Markdown
Contributor

I think it's fine to merge it - that's why I approved it

@joseph-isaacs

Copy link
Copy Markdown
Contributor

I don't think we want to fill these

@joseph-isaacs

Copy link
Copy Markdown
Contributor

More jobs = more chance of spot kill

@connortsui20
connortsui20 merged commit 463d203 into develop Nov 4, 2026
102 of 104 checks passed
@connortsui20
connortsui20 deleted the ct/codspeed-runtime-shards branch November 4, 2026 14:53
@connortsui20

Copy link
Copy Markdown
Member Author

@joseph-isaacs this is the same number of jobs

connortsui20 added a commit that referenced this pull request Nov 4, 2026
## Summary

The snapshot used by #11180 predates the benchmark cleanup merges, so it
overestimates simulation counts and target runtimes. Refresh both
snapshots from the five successful simulation shards in [develop run
41869576169](https://github.com/vortex-data/vortex/actions/runs/41869576169)
at `141abdc590`.

Note that these are just heuristics, so this will never affect
correctness / running of benchmarks.

## Changes

Recorded simulation cases fall from 1,869 to 1,408, and total target
execution time falls from 711s to 400s. The refreshed plan estimates
83-84s for each regular shard and 65s for the allocation-excluded shard.
These are target execution estimates, not full CI job times.

Verified complete measurements for all 129 targets, exact target
coverage, the 800-benchmark cap, and allocation-mode separation. The
planner and `git diff --check` passed. No local tests or benchmarks were
run. The pre-commit and pre-push hooks were bypassed as requested.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants