Repository navigation
Track benchmark results in CI and report regressions #258
Description
Split out of #151. The benchmark suite, the stock hxcpp comparison and the published tables are done (`bench/`, README.md and BENCHMARKS.md, written by `haxe --run Run --readme`). What is left is the CI half:
- run a small subset of the suite on each change and keep the raw results, so trends show over time;
- compare a change with its base commit on the same runner and report throughput, tail pauses and peak memory separately;
- block a merge only once normal run-to-run variation on the runners is measured, or on a controlled runner.
Shared runners are noisy, so the first step is recording how much the same commit varies between runs.
Activity
dimensionscape commentedon Oct 23, 2026
Moving this past 6.0.0: it is open-ended work, not a defect in the release.
dimensionscape commentedon Oct 27, 2026
Done on main in 01c56a4 (2e4e129): bench/ci.sh and the bench workflow compare each change with its base commit on the same runner and report throughput, tail pauses and peak memory separately, as warnings past their run-to-run noise. A daily run of main against itself measures that noise, runs on main keep their results on the bench-history branch, and bench/Compare.hx reports the differences.