Repository navigation
rubyspec README: complain/output/be_computed_by are NOT stubbed, and the error ran the wrong way - #911
Merged
Conversation
…the error ran the wrong way
The Caveats section said:
The shim is not full MSpec: a few matchers (`complain`, `output`,
`be_computed_by`, `ruby_exe`, some `argf`/IO helpers) are stubbed, so those
examples count as skipped rather than passing.
Three of the four named matchers are implemented in the shim:
ComplainMatcher spec_helper.rb:204-249 (real $stderr capture; wired at :249)
OutputMatcher spec_helper.rb:254-279
be_computed_by spec_helper.rb:199
Genuinely stubbed, raising SpecSkip: ruby_exe (:882) and output_to_fd (:280).
## ⛔ Why it matters: the error ran in the direction that matters
A stubbed matcher makes an example SKIP, so the total is too LOW. An example
using `complain` RUNS, and when the behaviour is missing it FAILS and counts
against the ratchet. Those are opposite errors, and a reader trusting the old
sentence would have looked for such examples among the skips, where they are
not.
cmd/phantom's rationale quotes that sentence to argue "the stubbed matchers make
the total too LOW". That holds for ruby_exe and output_to_fd and not for the
three that run, so the note now says which.
## How it was caught
Not by re-reading the shim. core/enumerable/to_set_spec.rb asserts a 4.0
deprecation warning rbgo does not emit (#909) and reports
Enumerable#to_set | instantiates an object of provided as the first argument set class
expected a warning matching /Enumerable#to_set/, got ""
A SKIPPED example cannot report that. The output contradicted the README, which
is the only reason anyone looked.
Documentation only; no code or test changes.
Refs #909
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Nov 9, 2026
tannevaled
added a commit
that referenced
this pull request
Nov 9, 2026
Every number the README states about the ruby/spec measurement was derived
from a file in this repository, and every one of them except the example
count was still retyped by hand. All of them had drifted, in both
directions at once:
- the prose divided by 93.3 % while the table it interprets says 95.0 %,
and the headline warned against reading it as "93 % of Ruby";
- the prose counted 477 skips and 10 unreadable files where the table
says 491 and 4;
- the stubbed-helper list named `be_computed_by`, which is implemented
(#911 corrected that in the conformance README and nowhere else), and
"some argf/IO helpers", which do not exist, while omitting `ruby_cmd`,
one line below the `ruby_exe` it did name.
The last one is the error #911 fixed reached through only one of its two
callers, which is why this adds gates rather than more corrected prose.
readme_conformance_prose_test.go adds two:
TestReadmeStubbedHelpersMatchTheShim reads every SpecSkip raise out of
spec_helper.rb and requires both READMEs to name exactly that set, with
the right line citations. It refuses rather than under-reporting: a
SpecSkip raise that is not a one-line `def` fails the test asking for the
parser to be widened. Its population is stated -- `skip` raises
SkippedSpecError because the spec asked to skip, which is mspec behaviour
and not a hole in the shim.
TestReadmeConformanceProseMatchesItsOwnTable checks the table adds up
(pass+fail+error = ran, files with a result + files without = total),
recomputes the printed share to the tenth it is printed to, and requires
the paragraph under it to quote the table's own skip count, file count and
ratio. It does not re-measure conformance: the table is one dated sweep
and is allowed to be older than BASELINE. A dated measurement going stale
is not an error; a derived number contradicting the measurement it is
derived from is.
Both gates were run against the pre-fix READMEs before being run against
the corrected ones: 8 failures, one per drifted fact, including the
phantom `argf` row whose empty "where" cell was the tell.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Caveats section of
scripts/conformance/rubyspec/README.mdsaid:Three of the four named matchers are implemented.
complainComplainMatcher,spec_helper.rb:204-249, real$stderrcapture, wired at:249outputOutputMatcher,spec_helper.rb:254-279be_computed_byspec_helper.rb:199ruby_exeSpecSkip,spec_helper.rb:882output_to_fdSpecSkip,spec_helper.rb:280⛔ The error ran in the direction that matters
A stubbed matcher makes an example skip, so the total is too LOW. An example using
complainruns, and when the behaviour is missing it fails and counts against the ratchet. Those are opposite errors.cmd/phantom's rationale quotes this very sentence to argue "the stubbed matchers make the total too LOW, and a pass bought by an absence would make it too high". That reasoning is sound forruby_exeandoutput_to_fdand does not transfer to the three matchers that run, so the corrected text says which is which.The practical cost: a reader looking for examples blocked by a missing warning would search the skips, where they are not — they are in the failures, already counted against the baseline.
How it was caught
Not by re-reading the shim.
core/enumerable/to_set_spec.rbasserts a 4.0 deprecation warning rbgo does not emit, and reports:A skipped example cannot report that. The output contradicted the README, which is the only reason anyone looked — I had quoted the stale sentence myself earlier the same day, as a reason a
to_setrow "passed for the wrong reason".Scope
Documentation only — no code, no tests, no behaviour. The missing warning itself is Refs #909, which stays open: this PR corrects what the README claims about the shim, not the shim.
🤖 Generated with Claude Code