ubuntu-latest rolled 24.04 → 26.04 and the conformance gate changed its answer: main fails its own ratchet on re-run, same commit #838

main passes its own conformance gate in history and fails it on re-run, at the same commit, with nothing changed but the runner's operating system.

The control

Run 40355659600, commit e54df08, the ruby/spec ratchet lane, re-run 28 hours later:

passing examples: 23511 (baseline 23511, +0)

-- REGRESSED (1) --
   core/process/setrlimit_spec.rb: 28 passing, baseline 29 (-1)

-- improved (1) --
   core/kernel/require_spec.rb: 70 passing, baseline 69 (+1)

Same commit. Same go-version: 1.27.3. Same SPEC_SHA. runs-on: ubuntu-latest rolled from 24.04 to 26.04 between the two attempts, and that is the only variable.

It reproduces: three runs now (two on an unrelated branch, one on main) all name core/process/setrlimit_spec.rb: 28 passing, baseline 29. core/random/bytes_spec.rb moved in two of the three — that one is the known flake the BASELINE header documents, and it behaves exactly as documented.

Why this is worse than an ordinary red

Nobody re-runs a green lane. The gate's history says main is fine; the gate's answer today says otherwise. The drift entered silently, at a moment nobody chose, and the first PR to touch anything will be told it caused a regression it did not cause — which is how I found it. My branch adds a Go file under scripts/conformance/rubyspec/cmd/ and a README section; run.sh copies only spec_helper.rb into the corpus and sweeps only $SPECDIR/{language,core}, so it cannot reach the sweep at all.

That the file is setrlimit is not a coincidence: it measures resource limits, which is among the first things to differ between two Ubuntu majors. kernel/require is one of the five files the BASELINE header already records as "measured on darwin and recorded at CI's own LOWER ubuntu value" — that is, already known to be environment-sensitive.

What ubuntu-latest means for a measured gate

The BASELINE header is careful about reproducibility: a pinned SPEC_SHA, a real path rather than a symlink, TIMEOUT raised to 60s, an isolated snapshot of the corpus, three sweeps agreeing on all 2,206 records. Every one of those pins the measurement — and then the lane runs on a label that re-points without notice.

A gate whose environment is a moving label is not a gate; it is a reading. The whole point of recording per file was that "every move is attributable".

Two ways out, and they are different decisions

1. Pin the image — runs-on: ubuntu-24.04. The baseline keeps meaning what it meant; moving to a new Ubuntu becomes a deliberate change with its own measurement, like a SPEC_SHA bump. The cost is a pin to maintain.
2. Accept 26.04 and re-record. Then the one-line delta is reviewed on purpose and the header gains another entry, as it did for the darwin/ubuntu split. The cost is that it will happen again at the next roll.

⛔ Either way, re-recording blind is the thing to avoid: UPDATE_BASELINE=1 records whatever it sees, and here what it sees includes +1 on kernel/require — an improvement that is also environmental, and that would be absorbed as if it were progress.

Reproducing

gh run rerun 40355659600 --repo go-embedded-ruby/ruby
gh api repos/go-embedded-ruby/ruby/actions/runs/40355659600/attempts/1/jobs --jq '.jobs[0].id'
# then read "Runner Image / Image:" in each attempt's log

Measured 2026-10-27.

🤖 Generated with Claude Code