Repository navigation
rubyspec: record the baseline where it is enforced - #877
Merged
Merged
Conversation
#876 made `-update` refuse to bank a loss. This is the other half: until now there was no way to record a baseline on the machine that enforces it. The gate is pinned to ubuntu-24.04 because the environment changes the answer -- setrlimit read 28 on one Ubuntu major and 29 on the next, same commit (#838). `UPDATE_BASELINE=1` was a local command and nothing else. So EVERY baseline was recorded on a machine the gate will never run on, and the difference is invisible because both machines print a number. Measured on one developer's laptop: +9 examples against the CI-recorded baseline, for no change in the code at all. That left a genuine improvement with no honest way to be banked, and the Ruby 4.0 version move (#799) -- which re-records by definition -- with no correct way to land at all. .github/workflows/rubyspec-sweep.yml workflow_call, the sweep itself .github/workflows/rubyspec-ratchet.yml push/PR -> judge .github/workflows/rubyspec-baseline.yml workflow_dispatch -> re-record ONE definition, TWO callers, which is the point: the pinned image, the Go version, the raised TIMEOUT and the memory watcher are written once, so what records the baseline and what enforces it are the same machine by construction. Two lanes agreeing about `runs-on` would be an agreement somebody has to maintain, and the whole point of the pinning is that nobody notices when it lapses. The re-record uploads BASELINE as an artefact with its diffstat printed beside it. Nothing is committed from the lane: a lane that pushed to a branch would be a lane that can change the gate it just ran, and the person who asked for the re-record is the person who should read the delta before it becomes the record. Two details that are deliberate rather than incidental: - the diffstat step carries always(), so a REFUSED re-record still shows what it declined to do; the artifact step does NOT, because a refusal leaves BASELINE untouched and uploading it as "the re-recorded baseline" would hand the unchanged file to someone who had not read the log; - run.sh gains ABSORB=1 as a pass-through only. The refusal and the stamp stay in Go (cmd/ratchet), so there is no second place where the rule could differ. actionlint is clean across every workflow in the repository, and `bash -n` accepts run.sh. Refs #799 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Nov 3, 2026
Recorded by the rubyspec-baseline lane (#877) on the pinned ubuntu-24.04 -- the same runner the ratchet gates on -- and downloaded as its artefact. Not recorded here: this machine reads +9 against CI for no change in the code. 23,511 -> 23,673 passing examples ## The refusal was exercised on the real case first The lane was dispatched WITHOUT -absorb before it was dispatched with it, because the gate added in #876 had only ever refused a synthetic loss: re-record: 2206 files measured, 2206 in the baseline being replaced -- REGRESSED (11) -- core/objectspace/_id2ref_spec.rb: 0 passing, baseline 10 (-10) core/set/flatten_merge_spec.rb: 0 passing, baseline 3 (-3) ... ::error::REFUSING to re-record: 11 file(s) went backwards -- named above Both of the lane's deliberate asymmetries held on that run: the diffstat step ran (it carries always()) and the artefact step was SKIPPED (it does not), so no unchanged BASELINE was offered as "the re-recorded baseline". The second dispatch passed -absorb, and the baseline now carries the stamp: # ⛔ ABSORBED 11 file(s) that went backwards, recorded with -absorb on 2026-11-03: # core/gc/config_spec.rb: 7 passing, baseline 8 (-1) # ... Six of the eleven are guards CLOSING -- examples 4.0 stops asking for, each with a `ruby_version_is ""..."4.0"` site in the census -- and not defects. ## ⛔ What comparing the SETS found, which the counts hid The local sweep predicted 13 files going backwards; CI named 11. Both missing entries are informative: - language/numbered_parameters_spec.rb is absent because THIS BRANCH fixes it. Independent confirmation in CI of the +4 measured locally. - language/variables_spec.rb is absent because the baseline was STALE. CI reported "118 passing, baseline 115 (+3)" -- an improvement -- while the file has passed 119 since #871 and the move genuinely costs one example there. A real regression reported as a gain. That is a structural blind spot in the gate: a per-file floor cannot see a regression smaller than the file's unbanked improvement. Filed as #881. It is the exact complement of #876, which stops a re-record banking a LOSS; this is the cost of not banking a WIN -- and it existed because until #877 there was no correct way to re-record at all. This fresh measurement clears the accumulated slack as a side effect. ## README TestReadmeConformanceCountMatchesTheBaseline named all three stale sites and the new figure, which is how the README was updated rather than by hand. The dated table is left as the 3.4.1 sweep it is and now says so. Refs #799 Refs #881 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Nov 3, 2026
* Ruby 4.0.9: the version rbgo reports, and the numbered parameters
RUBY_VERSION moves from 3.4.1 to 4.0.9, and the one behaviour that could not
land without it lands with it.
## Why these two are one commit
compiler.go said so already: hiding `_1` from a Binding "belongs with the
RUBY_VERSION bump". Below 4.0 ruby/spec asserts the OPPOSITE --
language/numbered_parameters_spec.rb:93 is `ruby_version_is ""..."4.0"` and
pins `binding.local_variables == [:_1]` -- so doing it alone would have traded
four passing examples for four failing ones.
## The asymmetry is the whole change
`_1` leaves the LOCAL TABLE and stays in #parameters:
proc { _1 }.parameters # [[:opt, :_1]] -- UNGATED
-> { _1; binding.local_variables }.call("a") # [:_1] below 4.0, [] from 4.0
numbered_parameters_spec.rb:85 carries no version guard, so blanking b.params
as well would trade four examples for two. `it` is blanked from BOTH, because
MRI gives it no name in #parameters either. So: b.locals for both kinds,
b.params for `it` only.
## Refusing by NAME, before any lookup
MRI tests rb_numparam_id_p at the TOP of #local_variable_get, #local_variable_set
and #local_variable_defined? (proc.c v4.0.9:549, 602, 649):
if (rb_numparam_id_p(lid)) {
rb_name_err_raise("numbered parameter '%1$s' is not a local variable", ...)
}
Two consequences that are easy to miss and are both tested:
- it raises in a binding that has NO numbered parameter, and at the top
level. `_1` is not a local being hidden; it is not a local variable NAME.
- #local_variable_defined? RAISES rather than answering false. "Not defined"
is the natural reading and the wrong one.
And it uses rb_numparam_id_p, NOT rb_implicit_param_p (proc.c:517-521), which
also covers `it`. That narrowing is deliberate in MRI and necessary here: `it`
is a LEGAL local variable name -- `it = 5` compiles -- so widening the refusal
would break every binding holding a variable called `it`. There is a control
for it, and for the `_0` / `_10` / `_` boundary.
## What the bump breaks, measured
`go test ./...` over the whole repository: FOUR assertions, every one of them a
deliberate pin of the pre-4.0 behaviour, and nothing else in a 210-second VM
suite.
internal/compiler TestImplicitBlockParamsAreHidden "want the numbered names kept"
internal/vm TestImplicitItIsInvisible "_1 is a binding local"
internal/vm TestImplicitParametersAreNotLocalVariables
subtest: "while _1 stays visible at the version rbgo reports"
internal/vm TestPBRbConfigKeys the version string
Each is updated to the 4.0 behaviour with its comment rewritten, not flipped
silently. The third gains a case for the point that `it` and `_1` are refused
by DIFFERENT messages.
The fourth raises a question this commit does not answer: MRI may report the
API version ("4.0.0") rather than the full one for
RbConfig::CONFIG["ruby_version"]. Nothing in ruby/spec asserts that key, so the
corpus cannot settle it. Filed as #878 rather than changed on a hunch.
Refs #799
Refs #878
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* BASELINE re-recorded at 4.0.9, on the machine that enforces it
Recorded by the rubyspec-baseline lane (#877) on the pinned ubuntu-24.04 --
the same runner the ratchet gates on -- and downloaded as its artefact. Not
recorded here: this machine reads +9 against CI for no change in the code.
23,511 -> 23,673 passing examples
## The refusal was exercised on the real case first
The lane was dispatched WITHOUT -absorb before it was dispatched with it,
because the gate added in #876 had only ever refused a synthetic loss:
re-record: 2206 files measured, 2206 in the baseline being replaced
-- REGRESSED (11) --
core/objectspace/_id2ref_spec.rb: 0 passing, baseline 10 (-10)
core/set/flatten_merge_spec.rb: 0 passing, baseline 3 (-3)
...
::error::REFUSING to re-record: 11 file(s) went backwards -- named above
Both of the lane's deliberate asymmetries held on that run: the diffstat step
ran (it carries always()) and the artefact step was SKIPPED (it does not), so
no unchanged BASELINE was offered as "the re-recorded baseline".
The second dispatch passed -absorb, and the baseline now carries the stamp:
# ⛔ ABSORBED 11 file(s) that went backwards, recorded with -absorb on 2026-11-03:
# core/gc/config_spec.rb: 7 passing, baseline 8 (-1)
# ...
Six of the eleven are guards CLOSING -- examples 4.0 stops asking for, each
with a `ruby_version_is ""..."4.0"` site in the census -- and not defects.
## ⛔ What comparing the SETS found, which the counts hid
The local sweep predicted 13 files going backwards; CI named 11. Both missing
entries are informative:
- language/numbered_parameters_spec.rb is absent because THIS BRANCH fixes
it. Independent confirmation in CI of the +4 measured locally.
- language/variables_spec.rb is absent because the baseline was STALE. CI
reported "118 passing, baseline 115 (+3)" -- an improvement -- while the
file has passed 119 since #871 and the move genuinely costs one example
there. A real regression reported as a gain.
That is a structural blind spot in the gate: a per-file floor cannot see a
regression smaller than the file's unbanked improvement. Filed as #881. It is
the exact complement of #876, which stops a re-record banking a LOSS; this is
the cost of not banking a WIN -- and it existed because until #877 there was no
correct way to re-record at all.
This fresh measurement clears the accumulated slack as a side effect.
## README
TestReadmeConformanceCountMatchesTheBaseline named all three stale sites and
the new figure, which is how the README was updated rather than by hand. The
dated table is left as the 3.4.1 sweep it is and now says so.
Refs #799
Refs #881
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#876 made
-updaterefuse to bank a loss. This is the other half: until nowthere was no way to record a baseline on the machine that enforces it.
Every baseline was recorded on the wrong machine
The gate is pinned to
ubuntu-24.04because the environment changes the answer—
setrlimitread 28 on one Ubuntu major and 29 on the next, same commit(#838). And
UPDATE_BASELINE=1was a local developer command and nothing else.Those two facts together mean every baseline this repository has ever had was
recorded somewhere the gate will never run, and the difference is invisible,
because both machines print a number. Measured on one developer's laptop:
+9 examples against the CI-recorded baseline, for no change in the code.
So a genuine improvement had no honest way to be banked, and the Ruby 4.0
version move (#799) — which re-records by definition — had no correct way to
land at all. That is what was actually blocking the bump.
One definition, two callers
The pinned image, the Go version, the raised
TIMEOUTand the memory watcherare written once, so what records the baseline and what enforces it are the
same machine by construction. Two lanes agreeing about
runs-onwould be anagreement somebody has to maintain, and the whole point of the pinning is that
nobody notices when it lapses.
Actions → rubyspec-baseline → Run workflow. It uploads
BASELINEas anartefact with its diffstat printed beside it; you commit that file.
Nothing is committed from the lane
A lane that pushed to a branch would be a lane that can change the gate it just
ran. The person who asked for the re-record is exactly the person who should
read the delta before it becomes the record.
Two deliberate asymmetries
always()?BASELINEuntouched, and uploading it under the name "the re-recorded baseline" would hand the unchanged file to someone who had not read the logrun.shgainsABSORB=1as a pass-through only — the refusal and the stampstay in Go (
cmd/ratchet), so there is no second place where the rule coulddiffer.
Checked before opening
actionlintis clean across every workflow in the repository, not justthe three touched here.
bash -n scripts/conformance/rubyspec/run.shaccepts it.mainhas no branch protection, so moving the ratchet into a reusableworkflow — which changes the reported check name to
ratchet / ruby/spec ratchet— breaks no required status check. Worth stating, because that isthe way this refactor fails silently and permanently.
through the new indirection, so a green run here is the test.
Refs #799
🤖 Generated with Claude Code