Repository navigation
ratchet: a re-record that could not tell a win from a loss - #876
Merged
Merged
Conversation
`-update` wrote straight over BASELINE WITHOUT READING IT. The whole body was
f, err := os.Create(*baseline)
write(f, got, header)
so the one thing it could never report is the one thing that matters: whether
the sweep replacing the record was better or worse than the record. The
comparison never happened at all.
The hazard was already written down in this repository, TWICE, and gated in
neither:
scripts/.../cmd/guards/main.go:18
"UPDATE_BASELINE=1 records whatever it sees and would absorb a loss"
.github/workflows/rubyspec-ratchet.yml:41
"the same run also showed kernel/require +1, equally environmental, and
UPDATE_BASELINE=1 would bank it as progress."
Two notes about the same gap, written months apart, neither of which could stop
anything. This is the third time that pattern has been paid for here, and the
answer is not a third note.
You re-record to lock in a WIN. A file that went backwards is either a real
regression or an environment that moved -- the lane is pinned to ubuntu-24.04
precisely because the environment changes the answer -- and both need a person,
not an overwrite that erases the evidence of itself.
-update now reads the old baseline, compares, prints every section the judging
path prints, and REFUSES:
re-record: 2206 files measured, 2206 in the baseline being replaced
-- REGRESSED (1) --
core/a_spec.rb: 9 passing, baseline 10 (-1)
::error::REFUSING to re-record: 1 file(s) went backwards -- named above
errWouldAbsorb is kept DISTINCT from errRegression although both exit 1: "this
branch broke something" and "I declined to erase the evidence that something is
broken" send a reader to different places.
-absorb is the named derogation and it stamps the ARTEFACT, not just the log --
a console warning scrolls away and a baseline outlives the run that wrote it by
months:
# ⛔ ABSORBED 1 file(s) that went backwards, recorded with -absorb on 2026-11-03:
# core/a_spec.rb: 9 passing, baseline 10 (-1)
Tested, with the controls beside the refusals:
- the refused re-record leaves BASELINE byte for byte as it was; a refusal
that truncated the file would destroy the record it declined to replace;
- a file that stops loading, and a file that vanishes from the sweep, refuse
the same way;
- a re-record that only improves is ACCEPTED and is not stamped -- without
that control the rest only proves the gate always fires;
- a FIRST recording, with no baseline to be worse than, is not a refusal.
The existing round-trip test already re-recorded an improvement, so it passes
unchanged, which is the cheapest control of all.
Refs #799
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Nov 3, 2026
#876 made `-update` refuse to bank a loss. This is the other half: until now there was no way to record a baseline on the machine that enforces it. The gate is pinned to ubuntu-24.04 because the environment changes the answer -- setrlimit read 28 on one Ubuntu major and 29 on the next, same commit (#838). `UPDATE_BASELINE=1` was a local command and nothing else. So EVERY baseline was recorded on a machine the gate will never run on, and the difference is invisible because both machines print a number. Measured on one developer's laptop: +9 examples against the CI-recorded baseline, for no change in the code at all. That left a genuine improvement with no honest way to be banked, and the Ruby 4.0 version move (#799) -- which re-records by definition -- with no correct way to land at all. .github/workflows/rubyspec-sweep.yml workflow_call, the sweep itself .github/workflows/rubyspec-ratchet.yml push/PR -> judge .github/workflows/rubyspec-baseline.yml workflow_dispatch -> re-record ONE definition, TWO callers, which is the point: the pinned image, the Go version, the raised TIMEOUT and the memory watcher are written once, so what records the baseline and what enforces it are the same machine by construction. Two lanes agreeing about `runs-on` would be an agreement somebody has to maintain, and the whole point of the pinning is that nobody notices when it lapses. The re-record uploads BASELINE as an artefact with its diffstat printed beside it. Nothing is committed from the lane: a lane that pushed to a branch would be a lane that can change the gate it just ran, and the person who asked for the re-record is the person who should read the delta before it becomes the record. Two details that are deliberate rather than incidental: - the diffstat step carries always(), so a REFUSED re-record still shows what it declined to do; the artifact step does NOT, because a refusal leaves BASELINE untouched and uploading it as "the re-recorded baseline" would hand the unchanged file to someone who had not read the log; - run.sh gains ABSORB=1 as a pass-through only. The refusal and the stamp stay in Go (cmd/ratchet), so there is no second place where the rule could differ. actionlint is clean across every workflow in the repository, and `bash -n` accepts run.sh. Refs #799 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Nov 3, 2026
Recorded by the rubyspec-baseline lane (#877) on the pinned ubuntu-24.04 -- the same runner the ratchet gates on -- and downloaded as its artefact. Not recorded here: this machine reads +9 against CI for no change in the code. 23,511 -> 23,673 passing examples ## The refusal was exercised on the real case first The lane was dispatched WITHOUT -absorb before it was dispatched with it, because the gate added in #876 had only ever refused a synthetic loss: re-record: 2206 files measured, 2206 in the baseline being replaced -- REGRESSED (11) -- core/objectspace/_id2ref_spec.rb: 0 passing, baseline 10 (-10) core/set/flatten_merge_spec.rb: 0 passing, baseline 3 (-3) ... ::error::REFUSING to re-record: 11 file(s) went backwards -- named above Both of the lane's deliberate asymmetries held on that run: the diffstat step ran (it carries always()) and the artefact step was SKIPPED (it does not), so no unchanged BASELINE was offered as "the re-recorded baseline". The second dispatch passed -absorb, and the baseline now carries the stamp: # ⛔ ABSORBED 11 file(s) that went backwards, recorded with -absorb on 2026-11-03: # core/gc/config_spec.rb: 7 passing, baseline 8 (-1) # ... Six of the eleven are guards CLOSING -- examples 4.0 stops asking for, each with a `ruby_version_is ""..."4.0"` site in the census -- and not defects. ## ⛔ What comparing the SETS found, which the counts hid The local sweep predicted 13 files going backwards; CI named 11. Both missing entries are informative: - language/numbered_parameters_spec.rb is absent because THIS BRANCH fixes it. Independent confirmation in CI of the +4 measured locally. - language/variables_spec.rb is absent because the baseline was STALE. CI reported "118 passing, baseline 115 (+3)" -- an improvement -- while the file has passed 119 since #871 and the move genuinely costs one example there. A real regression reported as a gain. That is a structural blind spot in the gate: a per-file floor cannot see a regression smaller than the file's unbanked improvement. Filed as #881. It is the exact complement of #876, which stops a re-record banking a LOSS; this is the cost of not banking a WIN -- and it existed because until #877 there was no correct way to re-record at all. This fresh measurement clears the accumulated slack as a side effect. ## README TestReadmeConformanceCountMatchesTheBaseline named all three stale sites and the new figure, which is how the README was updated rather than by hand. The dated table is left as the 3.4.1 sweep it is and now says so. Refs #799 Refs #881 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Nov 3, 2026
* Ruby 4.0.9: the version rbgo reports, and the numbered parameters
RUBY_VERSION moves from 3.4.1 to 4.0.9, and the one behaviour that could not
land without it lands with it.
## Why these two are one commit
compiler.go said so already: hiding `_1` from a Binding "belongs with the
RUBY_VERSION bump". Below 4.0 ruby/spec asserts the OPPOSITE --
language/numbered_parameters_spec.rb:93 is `ruby_version_is ""..."4.0"` and
pins `binding.local_variables == [:_1]` -- so doing it alone would have traded
four passing examples for four failing ones.
## The asymmetry is the whole change
`_1` leaves the LOCAL TABLE and stays in #parameters:
proc { _1 }.parameters # [[:opt, :_1]] -- UNGATED
-> { _1; binding.local_variables }.call("a") # [:_1] below 4.0, [] from 4.0
numbered_parameters_spec.rb:85 carries no version guard, so blanking b.params
as well would trade four examples for two. `it` is blanked from BOTH, because
MRI gives it no name in #parameters either. So: b.locals for both kinds,
b.params for `it` only.
## Refusing by NAME, before any lookup
MRI tests rb_numparam_id_p at the TOP of #local_variable_get, #local_variable_set
and #local_variable_defined? (proc.c v4.0.9:549, 602, 649):
if (rb_numparam_id_p(lid)) {
rb_name_err_raise("numbered parameter '%1$s' is not a local variable", ...)
}
Two consequences that are easy to miss and are both tested:
- it raises in a binding that has NO numbered parameter, and at the top
level. `_1` is not a local being hidden; it is not a local variable NAME.
- #local_variable_defined? RAISES rather than answering false. "Not defined"
is the natural reading and the wrong one.
And it uses rb_numparam_id_p, NOT rb_implicit_param_p (proc.c:517-521), which
also covers `it`. That narrowing is deliberate in MRI and necessary here: `it`
is a LEGAL local variable name -- `it = 5` compiles -- so widening the refusal
would break every binding holding a variable called `it`. There is a control
for it, and for the `_0` / `_10` / `_` boundary.
## What the bump breaks, measured
`go test ./...` over the whole repository: FOUR assertions, every one of them a
deliberate pin of the pre-4.0 behaviour, and nothing else in a 210-second VM
suite.
internal/compiler TestImplicitBlockParamsAreHidden "want the numbered names kept"
internal/vm TestImplicitItIsInvisible "_1 is a binding local"
internal/vm TestImplicitParametersAreNotLocalVariables
subtest: "while _1 stays visible at the version rbgo reports"
internal/vm TestPBRbConfigKeys the version string
Each is updated to the 4.0 behaviour with its comment rewritten, not flipped
silently. The third gains a case for the point that `it` and `_1` are refused
by DIFFERENT messages.
The fourth raises a question this commit does not answer: MRI may report the
API version ("4.0.0") rather than the full one for
RbConfig::CONFIG["ruby_version"]. Nothing in ruby/spec asserts that key, so the
corpus cannot settle it. Filed as #878 rather than changed on a hunch.
Refs #799
Refs #878
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* BASELINE re-recorded at 4.0.9, on the machine that enforces it
Recorded by the rubyspec-baseline lane (#877) on the pinned ubuntu-24.04 --
the same runner the ratchet gates on -- and downloaded as its artefact. Not
recorded here: this machine reads +9 against CI for no change in the code.
23,511 -> 23,673 passing examples
## The refusal was exercised on the real case first
The lane was dispatched WITHOUT -absorb before it was dispatched with it,
because the gate added in #876 had only ever refused a synthetic loss:
re-record: 2206 files measured, 2206 in the baseline being replaced
-- REGRESSED (11) --
core/objectspace/_id2ref_spec.rb: 0 passing, baseline 10 (-10)
core/set/flatten_merge_spec.rb: 0 passing, baseline 3 (-3)
...
::error::REFUSING to re-record: 11 file(s) went backwards -- named above
Both of the lane's deliberate asymmetries held on that run: the diffstat step
ran (it carries always()) and the artefact step was SKIPPED (it does not), so
no unchanged BASELINE was offered as "the re-recorded baseline".
The second dispatch passed -absorb, and the baseline now carries the stamp:
# ⛔ ABSORBED 11 file(s) that went backwards, recorded with -absorb on 2026-11-03:
# core/gc/config_spec.rb: 7 passing, baseline 8 (-1)
# ...
Six of the eleven are guards CLOSING -- examples 4.0 stops asking for, each
with a `ruby_version_is ""..."4.0"` site in the census -- and not defects.
## ⛔ What comparing the SETS found, which the counts hid
The local sweep predicted 13 files going backwards; CI named 11. Both missing
entries are informative:
- language/numbered_parameters_spec.rb is absent because THIS BRANCH fixes
it. Independent confirmation in CI of the +4 measured locally.
- language/variables_spec.rb is absent because the baseline was STALE. CI
reported "118 passing, baseline 115 (+3)" -- an improvement -- while the
file has passed 119 since #871 and the move genuinely costs one example
there. A real regression reported as a gain.
That is a structural blind spot in the gate: a per-file floor cannot see a
regression smaller than the file's unbanked improvement. Filed as #881. It is
the exact complement of #876, which stops a re-record banking a LOSS; this is
the cost of not banking a WIN -- and it existed because until #877 there was no
correct way to re-record at all.
This fresh measurement clears the accumulated slack as a side effect.
## README
TestReadmeConformanceCountMatchesTheBaseline named all three stale sites and
the new figure, which is how the README was updated rather than by hand. The
dated table is left as the 3.4.1 sweep it is and now says so.
Refs #799
Refs #881
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while working out how to land the Ruby 4.0 version bump for #799, which
needs
BASELINEre-recorded. It turns out nothing was watching that re-record.-updatenever read the baseline it replacedThe entire body was:
So the one thing
-updatecould never report is the one thing that matters:whether the sweep replacing the record was better or worse than the record.
The comparison simply never happened.
It was written down twice, and gated in neither place
Two notes about the same gap, written months apart, neither able to stop
anything. The second one is from the incident that pinned this lane to
ubuntu-24.04—setrlimitread 28 on one Ubuntu major and 29 on the next,same commit. The lane is pinned precisely because the environment changes the
answer, which is exactly the condition under which a blind re-record is at its
most dangerous: it would bank the image roll as progress and erase the evidence
in the same write.
The refusal
You re-record to lock in a win. A file that went backwards is either a real
regression or an environment that moved, and both need a person.
errWouldAbsorbis kept distinct fromerrRegressionin the source thoughboth exit 1: "this branch broke something" and "I declined to erase the
evidence that something is broken" send a reader to different places.
The derogation stamps the artefact, not the log
A console warning scrolls away; a baseline outlives the run that wrote it by
months.
-absorbwrites the loss into the file:Controls, beside each refusal
BASELINEbyte for byte as it wasThe test fixture's loss is
a_spec 10 → 9whileb_specgains — a net ofzero, which is the shape no total can see. The existing round-trip test already
re-recorded an improvement, so it passes unchanged: the cheapest control of all.
Refs #799
🤖 Generated with Claude Code