Skip to content

ratchet: a re-record that could not tell a win from a loss - #876

Merged
tannevaled merged 1 commit into
mainfrom
ratchet-rerecord-gate
Nov 3, 2026
Merged

tannevaled merged 1 commit into
mainfrom
ratchet-rerecord-gate

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Found while working out how to land the Ruby 4.0 version bump for #799, which
needs BASELINE re-recorded. It turns out nothing was watching that re-record.

-update never read the baseline it replaced

The entire body was:

f, err := os.Create(*baseline)
write(f, got, header)

So the one thing -update could never report is the one thing that matters:
whether the sweep replacing the record was better or worse than the record.
The comparison simply never happened.

It was written down twice, and gated in neither place

scripts/conformance/rubyspec/cmd/guards/main.go:18
    "UPDATE_BASELINE=1 records whatever it sees and would absorb a loss"

.github/workflows/rubyspec-ratchet.yml:41
    "the same run also showed kernel/require +1, equally environmental,
     and UPDATE_BASELINE=1 would bank it as progress."

Two notes about the same gap, written months apart, neither able to stop
anything. The second one is from the incident that pinned this lane to
ubuntu-24.04 — setrlimit read 28 on one Ubuntu major and 29 on the next,
same commit. The lane is pinned precisely because the environment changes the
answer
, which is exactly the condition under which a blind re-record is at its
most dangerous: it would bank the image roll as progress and erase the evidence
in the same write.

The refusal

You re-record to lock in a win. A file that went backwards is either a real
regression or an environment that moved, and both need a person.

$ UPDATE_BASELINE=1 scripts/conformance/rubyspec/run.sh
re-record: 2206 files measured, 2206 in the baseline being replaced

-- REGRESSED (1) --
   core/a_spec.rb: 9 passing, baseline 10 (-1)

::error::REFUSING to re-record: 1 file(s) went backwards -- named above
A re-record locks in a win. Fix the regression, or pass -absorb to record
it on purpose -- the baseline then carries a line naming every file absorbed.

errWouldAbsorb is kept distinct from errRegression in the source though
both exit 1: "this branch broke something" and "I declined to erase the
evidence that something is broken"
send a reader to different places.

The derogation stamps the artefact, not the log

A console warning scrolls away; a baseline outlives the run that wrote it by
months. -absorb writes the loss into the file:

# ⛔ ABSORBED 1 file(s) that went backwards, recorded with -absorb on 2026-11-03:
#     core/a_spec.rb: 9 passing, baseline 10 (-1)

Controls, beside each refusal

case why it is here
a refused re-record leaves BASELINE byte for byte as it was a refusal that truncated the file would destroy the record it declined to replace
a re-record that only improves is accepted, unstamped without it, the rest only proves the gate always fires
a first recording, with no baseline to be worse than an absent baseline is not a loss
a file that stops loading, and one that vanishes from the sweep the other two ways backwards, same refusal

The test fixture's loss is a_spec 10 → 9 while b_spec gains — a net of
zero, which is the shape no total can see. The existing round-trip test already
re-recorded an improvement, so it passes unchanged: the cheapest control of all.

Refs #799

🤖 Generated with Claude Code

`-update` wrote straight over BASELINE WITHOUT READING IT. The whole body was

    f, err := os.Create(*baseline)
    write(f, got, header)

so the one thing it could never report is the one thing that matters: whether
the sweep replacing the record was better or worse than the record. The
comparison never happened at all.

The hazard was already written down in this repository, TWICE, and gated in
neither:

  scripts/.../cmd/guards/main.go:18
      "UPDATE_BASELINE=1 records whatever it sees and would absorb a loss"

  .github/workflows/rubyspec-ratchet.yml:41
      "the same run also showed kernel/require +1, equally environmental, and
       UPDATE_BASELINE=1 would bank it as progress."

Two notes about the same gap, written months apart, neither of which could stop
anything. This is the third time that pattern has been paid for here, and the
answer is not a third note.

You re-record to lock in a WIN. A file that went backwards is either a real
regression or an environment that moved -- the lane is pinned to ubuntu-24.04
precisely because the environment changes the answer -- and both need a person,
not an overwrite that erases the evidence of itself.

-update now reads the old baseline, compares, prints every section the judging
path prints, and REFUSES:

    re-record: 2206 files measured, 2206 in the baseline being replaced
    -- REGRESSED (1) --
       core/a_spec.rb: 9 passing, baseline 10 (-1)
    ::error::REFUSING to re-record: 1 file(s) went backwards -- named above

errWouldAbsorb is kept DISTINCT from errRegression although both exit 1: "this
branch broke something" and "I declined to erase the evidence that something is
broken" send a reader to different places.

-absorb is the named derogation and it stamps the ARTEFACT, not just the log --
a console warning scrolls away and a baseline outlives the run that wrote it by
months:

    # ⛔ ABSORBED 1 file(s) that went backwards, recorded with -absorb on 2026-11-03:
    #     core/a_spec.rb: 9 passing, baseline 10 (-1)

Tested, with the controls beside the refusals:

  - the refused re-record leaves BASELINE byte for byte as it was; a refusal
    that truncated the file would destroy the record it declined to replace;
  - a file that stops loading, and a file that vanishes from the sweep, refuse
    the same way;
  - a re-record that only improves is ACCEPTED and is not stamped -- without
    that control the rest only proves the gate always fires;
  - a FIRST recording, with no baseline to be worse than, is not a refusal.

The existing round-trip test already re-recorded an improvement, so it passes
unchanged, which is the cheapest control of all.

Refs #799

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 3abe07d into main Nov 3, 2026
26 checks passed
@tannevaled
tannevaled deleted the ratchet-rerecord-gate branch November 3, 2026 12:41
tannevaled added a commit that referenced this pull request Nov 3, 2026
#876 made `-update` refuse to bank a loss. This is the other half: until now
there was no way to record a baseline on the machine that enforces it.

The gate is pinned to ubuntu-24.04 because the environment changes the answer
-- setrlimit read 28 on one Ubuntu major and 29 on the next, same commit
(#838). `UPDATE_BASELINE=1` was a local command and nothing else. So EVERY
baseline was recorded on a machine the gate will never run on, and the
difference is invisible because both machines print a number. Measured on one
developer's laptop: +9 examples against the CI-recorded baseline, for no change
in the code at all.

That left a genuine improvement with no honest way to be banked, and the Ruby
4.0 version move (#799) -- which re-records by definition -- with no correct way
to land at all.

  .github/workflows/rubyspec-sweep.yml     workflow_call, the sweep itself
  .github/workflows/rubyspec-ratchet.yml   push/PR          -> judge
  .github/workflows/rubyspec-baseline.yml  workflow_dispatch -> re-record

ONE definition, TWO callers, which is the point: the pinned image, the Go
version, the raised TIMEOUT and the memory watcher are written once, so what
records the baseline and what enforces it are the same machine by construction.
Two lanes agreeing about `runs-on` would be an agreement somebody has to
maintain, and the whole point of the pinning is that nobody notices when it
lapses.

The re-record uploads BASELINE as an artefact with its diffstat printed beside
it. Nothing is committed from the lane: a lane that pushed to a branch would be
a lane that can change the gate it just ran, and the person who asked for the
re-record is the person who should read the delta before it becomes the record.

Two details that are deliberate rather than incidental:

  - the diffstat step carries always(), so a REFUSED re-record still shows what
    it declined to do; the artifact step does NOT, because a refusal leaves
    BASELINE untouched and uploading it as "the re-recorded baseline" would
    hand the unchanged file to someone who had not read the log;
  - run.sh gains ABSORB=1 as a pass-through only. The refusal and the stamp
    stay in Go (cmd/ratchet), so there is no second place where the rule could
    differ.

actionlint is clean across every workflow in the repository, and `bash -n`
accepts run.sh.

Refs #799

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Nov 3, 2026
Recorded by the rubyspec-baseline lane (#877) on the pinned ubuntu-24.04 --
the same runner the ratchet gates on -- and downloaded as its artefact. Not
recorded here: this machine reads +9 against CI for no change in the code.

    23,511 -> 23,673 passing examples

## The refusal was exercised on the real case first

The lane was dispatched WITHOUT -absorb before it was dispatched with it,
because the gate added in #876 had only ever refused a synthetic loss:

    re-record: 2206 files measured, 2206 in the baseline being replaced
    -- REGRESSED (11) --
       core/objectspace/_id2ref_spec.rb: 0 passing, baseline 10 (-10)
       core/set/flatten_merge_spec.rb: 0 passing, baseline 3 (-3)
       ...
    ::error::REFUSING to re-record: 11 file(s) went backwards -- named above

Both of the lane's deliberate asymmetries held on that run: the diffstat step
ran (it carries always()) and the artefact step was SKIPPED (it does not), so
no unchanged BASELINE was offered as "the re-recorded baseline".

The second dispatch passed -absorb, and the baseline now carries the stamp:

    # ⛔ ABSORBED 11 file(s) that went backwards, recorded with -absorb on 2026-11-03:
    #     core/gc/config_spec.rb: 7 passing, baseline 8 (-1)
    #     ...

Six of the eleven are guards CLOSING -- examples 4.0 stops asking for, each
with a `ruby_version_is ""..."4.0"` site in the census -- and not defects.

## ⛔ What comparing the SETS found, which the counts hid

The local sweep predicted 13 files going backwards; CI named 11. Both missing
entries are informative:

  - language/numbered_parameters_spec.rb is absent because THIS BRANCH fixes
    it. Independent confirmation in CI of the +4 measured locally.

  - language/variables_spec.rb is absent because the baseline was STALE. CI
    reported "118 passing, baseline 115 (+3)" -- an improvement -- while the
    file has passed 119 since #871 and the move genuinely costs one example
    there. A real regression reported as a gain.

That is a structural blind spot in the gate: a per-file floor cannot see a
regression smaller than the file's unbanked improvement. Filed as #881. It is
the exact complement of #876, which stops a re-record banking a LOSS; this is
the cost of not banking a WIN -- and it existed because until #877 there was no
correct way to re-record at all.

This fresh measurement clears the accumulated slack as a side effect.

## README

TestReadmeConformanceCountMatchesTheBaseline named all three stale sites and
the new figure, which is how the README was updated rather than by hand. The
dated table is left as the 3.4.1 sweep it is and now says so.

Refs #799
Refs #881

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Nov 3, 2026
* Ruby 4.0.9: the version rbgo reports, and the numbered parameters

RUBY_VERSION moves from 3.4.1 to 4.0.9, and the one behaviour that could not
land without it lands with it.

## Why these two are one commit

compiler.go said so already: hiding `_1` from a Binding "belongs with the
RUBY_VERSION bump". Below 4.0 ruby/spec asserts the OPPOSITE --
language/numbered_parameters_spec.rb:93 is `ruby_version_is ""..."4.0"` and
pins `binding.local_variables == [:_1]` -- so doing it alone would have traded
four passing examples for four failing ones.

## The asymmetry is the whole change

`_1` leaves the LOCAL TABLE and stays in #parameters:

    proc { _1 }.parameters                        # [[:opt, :_1]]  -- UNGATED
    -> { _1; binding.local_variables }.call("a")  # [:_1] below 4.0, [] from 4.0

numbered_parameters_spec.rb:85 carries no version guard, so blanking b.params
as well would trade four examples for two. `it` is blanked from BOTH, because
MRI gives it no name in #parameters either. So: b.locals for both kinds,
b.params for `it` only.

## Refusing by NAME, before any lookup

MRI tests rb_numparam_id_p at the TOP of #local_variable_get, #local_variable_set
and #local_variable_defined? (proc.c v4.0.9:549, 602, 649):

    if (rb_numparam_id_p(lid)) {
        rb_name_err_raise("numbered parameter '%1$s' is not a local variable", ...)
    }

Two consequences that are easy to miss and are both tested:

  - it raises in a binding that has NO numbered parameter, and at the top
    level. `_1` is not a local being hidden; it is not a local variable NAME.
  - #local_variable_defined? RAISES rather than answering false. "Not defined"
    is the natural reading and the wrong one.

And it uses rb_numparam_id_p, NOT rb_implicit_param_p (proc.c:517-521), which
also covers `it`. That narrowing is deliberate in MRI and necessary here: `it`
is a LEGAL local variable name -- `it = 5` compiles -- so widening the refusal
would break every binding holding a variable called `it`. There is a control
for it, and for the `_0` / `_10` / `_` boundary.

## What the bump breaks, measured

`go test ./...` over the whole repository: FOUR assertions, every one of them a
deliberate pin of the pre-4.0 behaviour, and nothing else in a 210-second VM
suite.

    internal/compiler  TestImplicitBlockParamsAreHidden    "want the numbered names kept"
    internal/vm        TestImplicitItIsInvisible           "_1 is a binding local"
    internal/vm        TestImplicitParametersAreNotLocalVariables
                         subtest: "while _1 stays visible at the version rbgo reports"
    internal/vm        TestPBRbConfigKeys                  the version string

Each is updated to the 4.0 behaviour with its comment rewritten, not flipped
silently. The third gains a case for the point that `it` and `_1` are refused
by DIFFERENT messages.

The fourth raises a question this commit does not answer: MRI may report the
API version ("4.0.0") rather than the full one for
RbConfig::CONFIG["ruby_version"]. Nothing in ruby/spec asserts that key, so the
corpus cannot settle it. Filed as #878 rather than changed on a hunch.

Refs #799
Refs #878

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* BASELINE re-recorded at 4.0.9, on the machine that enforces it

Recorded by the rubyspec-baseline lane (#877) on the pinned ubuntu-24.04 --
the same runner the ratchet gates on -- and downloaded as its artefact. Not
recorded here: this machine reads +9 against CI for no change in the code.

    23,511 -> 23,673 passing examples

## The refusal was exercised on the real case first

The lane was dispatched WITHOUT -absorb before it was dispatched with it,
because the gate added in #876 had only ever refused a synthetic loss:

    re-record: 2206 files measured, 2206 in the baseline being replaced
    -- REGRESSED (11) --
       core/objectspace/_id2ref_spec.rb: 0 passing, baseline 10 (-10)
       core/set/flatten_merge_spec.rb: 0 passing, baseline 3 (-3)
       ...
    ::error::REFUSING to re-record: 11 file(s) went backwards -- named above

Both of the lane's deliberate asymmetries held on that run: the diffstat step
ran (it carries always()) and the artefact step was SKIPPED (it does not), so
no unchanged BASELINE was offered as "the re-recorded baseline".

The second dispatch passed -absorb, and the baseline now carries the stamp:

    # ⛔ ABSORBED 11 file(s) that went backwards, recorded with -absorb on 2026-11-03:
    #     core/gc/config_spec.rb: 7 passing, baseline 8 (-1)
    #     ...

Six of the eleven are guards CLOSING -- examples 4.0 stops asking for, each
with a `ruby_version_is ""..."4.0"` site in the census -- and not defects.

## ⛔ What comparing the SETS found, which the counts hid

The local sweep predicted 13 files going backwards; CI named 11. Both missing
entries are informative:

  - language/numbered_parameters_spec.rb is absent because THIS BRANCH fixes
    it. Independent confirmation in CI of the +4 measured locally.

  - language/variables_spec.rb is absent because the baseline was STALE. CI
    reported "118 passing, baseline 115 (+3)" -- an improvement -- while the
    file has passed 119 since #871 and the move genuinely costs one example
    there. A real regression reported as a gain.

That is a structural blind spot in the gate: a per-file floor cannot see a
regression smaller than the file's unbanked improvement. Filed as #881. It is
the exact complement of #876, which stops a re-record banking a LOSS; this is
the cost of not banking a WIN -- and it existed because until #877 there was no
correct way to re-record at all.

This fresh measurement clears the accumulated slack as a side effect.

## README

TestReadmeConformanceCountMatchesTheBaseline named all three stale sites and
the new figure, which is how the README was updated rather than by hand. The
dated table is left as the 3.4.1 sweep it is and now says so.

Refs #799
Refs #881

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant