Record which machine ran a test, and make performance comparisons machine-aware #301

The regression reports decide that something got slower by comparing a run of a branch against the previous run of the same branch:

# all-reports.py
timeMinPhase = 4   # Need to have completed code generation to report performance regressions
timeRel = 1.7      # Minimum 1.7x time is registered as a performance regression
timeAbs = 10       # Ignore performance regressions for times <10s...

It pairs consecutive dates of a branch from omcversion and flags a phase that took 1.7 times longer. That is sound as long as both runs happened on the same hardware, which until now they did, because each job was pinned to its machine.

That assumption is going away. Jobs are being shared between machines, and while ryzen-5950x-1 and ryzen-5950x-2 are the same model, a faster machine such as a ryzen-9950x is a different generation - and the difference will be largest exactly where the testing spends its time, compiling one model at a time, rather than spread evenly over the phases.

Two things then go wrong, and only one of them is noisy:

• a branch that runs on the fast machine one night and a slower one the next gets longer times with no change in OpenModelica, and every model that crosses 1.7x produces a regression and an email;
• a real regression that lands on the faster machine is hidden, which nobody notices at all.

What the database would need

Nothing records which machine produced a result. job_claim knows the host while a run is in progress, but that is not kept, and omcversion says only which compiler was used.

One row per run, keyed the way a run is already identified:

CREATE TABLE machine (
  branch text NOT NULL,
  date   bigint NOT NULL,
  host   text NOT NULL,      -- socket.gethostname(), as job_claim already uses
  cpu    text,               -- what cpu_name() reads out of /proc/cpuinfo
  cores  integer,
  memory_gb integer,
  system text,               -- lsb_release
  PRIMARY KEY (branch, date)
);

test.py already computes all of it for the HTML header and then throws it away:

sysInfo = "%s, %d GB RAM, %s%s" % (cpu_name(), memory_gb, docker, lsb_release)

so storing it is mostly plumbing.

Suggested order

1. Record the machine of every run. Cheap, and the prerequisite for the rest.
2. Make the comparison machine-aware. all-reports.py should either compare a run against the previous run on the same machine, or mark a comparison across machines as not a performance signal. This is the part that has to exist before a machine of a different speed joins, otherwise the first mixed night is all noise.
3. Then, if cross-machine numbers should be comparable, measure the ratio rather than guess it. A fixed model timed once per run - the testing already builds HelloWorld for its feature checks, so it is nearly free - gives an index per run. After a few runs it also shows how noisy that index is on one machine, which decides whether scaling by it is meaningful at all. A single number cannot really capture it: compile time is dominated by single-core speed and I/O, simulation by the CPU, and both depend on how many tests run in parallel.

I would not apply a correction factor silently in any case. Saying "this run came from a machine that measured 1.4 times slower" is honest; multiplying the numbers and presenting them as comparable is not.

Meanwhile

Jobs are being split so that the same libraries run on the same machine, which keeps the comparisons valid without any of the above.

Context: part of the network database work in #295, which is what made sharing jobs between machines possible in the first place.

Generated by Claude Code.