Repository navigation
README: derive the conformance prose from the table and the shim - #913
Merged
Merged
Conversation
Every number the README states about the ruby/spec measurement was derived
from a file in this repository, and every one of them except the example
count was still retyped by hand. All of them had drifted, in both
directions at once:
- the prose divided by 93.3 % while the table it interprets says 95.0 %,
and the headline warned against reading it as "93 % of Ruby";
- the prose counted 477 skips and 10 unreadable files where the table
says 491 and 4;
- the stubbed-helper list named `be_computed_by`, which is implemented
(#911 corrected that in the conformance README and nowhere else), and
"some argf/IO helpers", which do not exist, while omitting `ruby_cmd`,
one line below the `ruby_exe` it did name.
The last one is the error #911 fixed reached through only one of its two
callers, which is why this adds gates rather than more corrected prose.
readme_conformance_prose_test.go adds two:
TestReadmeStubbedHelpersMatchTheShim reads every SpecSkip raise out of
spec_helper.rb and requires both READMEs to name exactly that set, with
the right line citations. It refuses rather than under-reporting: a
SpecSkip raise that is not a one-line `def` fails the test asking for the
parser to be widened. Its population is stated -- `skip` raises
SkippedSpecError because the spec asked to skip, which is mspec behaviour
and not a hole in the shim.
TestReadmeConformanceProseMatchesItsOwnTable checks the table adds up
(pass+fail+error = ran, files with a result + files without = total),
recomputes the printed share to the tenth it is printed to, and requires
the paragraph under it to quote the table's own skip count, file count and
ratio. It does not re-measure conformance: the table is one dated sweep
and is allowed to be older than BASELINE. A dated measurement going stale
is not an error; a derived number contradicting the measurement it is
derived from is.
Both gates were run against the pre-fix READMEs before being run against
the corrected ones: 8 failures, one per drifted fact, including the
phantom `argf` row whose empty "where" cell was the tell.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestReadmeConformanceCountMatchesTheBaselinealready ties the ruby/spec example count to the gate that measures it, on the stated principle that "a number a human retypes is a number that rots". Everything else the README says about that measurement was still retyped. All of it had rotted, and in both directions at once.What was wrong
`be_computed_by`spec_helper.rb:199)argf/IO helpers"SpecSkipfor themruby_cmdis stubbed (spec_helper.rb:883) and was listed nowhereThe last three are the error #911 fixed, reached through only one of its two callers: #911 corrected
scripts/conformance/rubyspec/README.mdand the top-level README kept the wrong list. #911's own table also missedruby_cmd, one line below theruby_exeit named.That is why this PR adds gates rather than a third round of corrected prose.
The gates
TestReadmeStubbedHelpersMatchTheShimreads everySpecSkipraise out ofspec_helper.rband requires both READMEs to name exactly that set, with the right line citations — in both directions, because a stub the README omits hides a skip, and a helper the README calls stubbed when it is implemented sends a reader hunting among the skips for an example that actually failed. That is how #909 was nearly missed.Two things it does on purpose:
SpecSkipraise that is not a one-linedefis a stub its parser cannot name, and it fails asking for the parser to be widened instead of quietly dropping it.skipraisesSkippedSpecErrorat:318because the spec asked to skip — mspec behaviour rbgo implements, not a hole in the shim — so it is deliberately outside the census, and the comment says so.TestReadmeConformanceProseMatchesItsOwnTablechecks the table adds up (pass+fail+error = ran,files with a result + files without = total), recomputes the printed share to the tenth it is printed to, and requires the paragraph beneath it to quote the table's own skip count, file count and ratio.It deliberately does not re-measure conformance. The table is one dated sweep and is allowed to be older than
BASELINE, exactly as the note beside it says — a dated measurement going stale is not an error; a derived number contradicting the measurement it is derived from is.Control
Both gates were run against the pre-fix READMEs before being run against the corrected ones. They produced 8 failures, one per drifted fact:
The phantom row was findable without running anything: its
wherecell was—. The gate now rejects a row that cites no line, which is the shape a made-up entry takes.Refs #911.
Refs #909 — it stays open; this only stops the README sending its reader to look
for that failure among the skips.
🤖 Generated with Claude Code