Repository navigation
Balance CodSpeed simulation shards by runtime - #11180
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Merging this PR will regress 1 benchmark1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
robert3005
left a comment
There was a problem hiding this comment.
I think the annoying part of this is having to rebalance the timings and add new ones as you add benchmarks
|
Maybe we can have a bot that does this automatically? Regardless this is a much faster process so I would like to merge this (or something similar). I'll hold off on this until I get into office |
|
I think it's fine to merge it - that's why I approved it |
|
I don't think we want to fill these |
|
More jobs = more chance of spot kill |
|
@joseph-isaacs this is the same number of jobs |
## Summary The snapshot used by #11180 predates the benchmark cleanup merges, so it overestimates simulation counts and target runtimes. Refresh both snapshots from the five successful simulation shards in [develop run 41869576169](https://github.com/vortex-data/vortex/actions/runs/41869576169) at `141abdc590`. Note that these are just heuristics, so this will never affect correctness / running of benchmarks. ## Changes Recorded simulation cases fall from 1,869 to 1,408, and total target execution time falls from 711s to 400s. The refreshed plan estimates 83-84s for each regular shard and 65s for the allocation-excluded shard. These are target execution estimates, not full CI job times. Verified complete measurements for all 129 targets, exact target coverage, the 800-benchmark cap, and allocation-mode separation. The planner and `git diff --check` passed. No local tests or benchmarks were run. The pre-commit and pre-push hooks were bypassed as requested. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Summary
The count-based planner fills only 3 of the 5 fixed shard slots and keeps most packages together, although benchmark counts do not predict execution time. Balance recorded target runtimes across all 5 slots, splitting large packages at benchmark-target boundaries.
The first CI run reduced the longest shard job from 10m03s to 5m07s. All 5 shards passed. Small packages stay together to limit repeated builds, and allocation-excluded targets remain separate.
Changes
24cd0eac8f, with a command to refresh the snapshot from simulation logs. Measure each complete binary execution, including startup and fixture construction, and use the median when several runs are supplied.Observed CI shard times
Compared run 41711616261 with run 41716056593 on
863c58f026. These are elapsed CI job times including setup, builds, measurement, and uploads. Exact timings depend on cache and runner state.CI passes the planner, all 5 simulation shards, Python formatting, linting, type checks, and all 58 script tests, including the 17 shard-planner tests. Other CI jobs are still running.
Executed the planner locally to inspect the proposed assignment. Tests and linting were left to CI. Local hooks were bypassed under the requested CI-only workflow.