Skip to content

README: derive the conformance prose from the table and the shim - #913

Merged
tannevaled merged 1 commit into
mainfrom
readme-derived-numbers-gate
Nov 9, 2026
Merged

tannevaled merged 1 commit into
mainfrom
readme-derived-numbers-gate

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

TestReadmeConformanceCountMatchesTheBaseline already ties the ruby/spec example count to the gate that measures it, on the stated principle that "a number a human retypes is a number that rots". Everything else the README says about that measurement was still retyped. All of it had rotted, and in both directions at once.

What was wrong

claim in the README what the file it is derived from says
"The 93.3 % divides passes by the examples that ran" 23,471 / 24,718 = 95.0 %, as the table three lines above it already printed
not "rbgo implements 93 % of Ruby" 95 %
"the 477 skips" the table says 491
"the 10 unreadable files" the table says 4
stubbed: `be_computed_by` it is implemented (spec_helper.rb:199)
stubbed: "some argf/IO helpers" nothing in the shim raises SpecSkip for them
— ruby_cmd is stubbed (spec_helper.rb:883) and was listed nowhere

The last three are the error #911 fixed, reached through only one of its two callers: #911 corrected scripts/conformance/rubyspec/README.md and the top-level README kept the wrong list. #911's own table also missed ruby_cmd, one line below the ruby_exe it named.

That is why this PR adds gates rather than a third round of corrected prose.

The gates

TestReadmeStubbedHelpersMatchTheShim reads every SpecSkip raise out of spec_helper.rb and requires both READMEs to name exactly that set, with the right line citations — in both directions, because a stub the README omits hides a skip, and a helper the README calls stubbed when it is implemented sends a reader hunting among the skips for an example that actually failed. That is how #909 was nearly missed.

Two things it does on purpose:

  • It refuses rather than under-reporting. A SpecSkip raise that is not a one-line def is a stub its parser cannot name, and it fails asking for the parser to be widened instead of quietly dropping it.
  • It states its population. skip raises SkippedSpecError at :318 because the spec asked to skip — mspec behaviour rbgo implements, not a hole in the shim — so it is deliberately outside the census, and the comment says so.

TestReadmeConformanceProseMatchesItsOwnTable checks the table adds up (pass+fail+error = ran, files with a result + files without = total), recomputes the printed share to the tenth it is printed to, and requires the paragraph beneath it to quote the table's own skip count, file count and ratio.

It deliberately does not re-measure conformance. The table is one dated sweep and is allowed to be older than BASELINE, exactly as the note beside it says — a dated measurement going stale is not an error; a derived number contradicting the measurement it is derived from is.

Control

Both gates were run against the pre-fix READMEs before being run against the corrected ones. They produced 8 failures, one per drifted fact:

README.md's "The shim is not MSpec" bullet does not name `output_to_fd` …
README.md's "The shim is not MSpec" bullet does not name `ruby_cmd` …
README.md's "The shim is not MSpec" bullet calls `be_computed_by` stubbed …
README.md's "The shim is not MSpec" bullet calls `argf` stubbed …
scripts/conformance/rubyspec/README.md:523 names `argf` but cites no spec_helper.rb line …
spec_helper.rb stubs `ruby_cmd` at :883, and the table does not list it
README.md does not contain "The 95.0 % divides passes by the examples that" …
README.md does not contain "the 491 skips" …

The phantom row was findable without running anything: its where cell was —. The gate now rejects a row that cites no line, which is the shape a made-up entry takes.

Refs #911.
Refs #909 — it stays open; this only stops the README sending its reader to look
for that failure among the skips.

🤖 Generated with Claude Code

Every number the README states about the ruby/spec measurement was derived
from a file in this repository, and every one of them except the example
count was still retyped by hand. All of them had drifted, in both
directions at once:

  - the prose divided by 93.3 % while the table it interprets says 95.0 %,
    and the headline warned against reading it as "93 % of Ruby";
  - the prose counted 477 skips and 10 unreadable files where the table
    says 491 and 4;
  - the stubbed-helper list named `be_computed_by`, which is implemented
    (#911 corrected that in the conformance README and nowhere else), and
    "some argf/IO helpers", which do not exist, while omitting `ruby_cmd`,
    one line below the `ruby_exe` it did name.

The last one is the error #911 fixed reached through only one of its two
callers, which is why this adds gates rather than more corrected prose.

readme_conformance_prose_test.go adds two:

TestReadmeStubbedHelpersMatchTheShim reads every SpecSkip raise out of
spec_helper.rb and requires both READMEs to name exactly that set, with
the right line citations. It refuses rather than under-reporting: a
SpecSkip raise that is not a one-line `def` fails the test asking for the
parser to be widened. Its population is stated -- `skip` raises
SkippedSpecError because the spec asked to skip, which is mspec behaviour
and not a hole in the shim.

TestReadmeConformanceProseMatchesItsOwnTable checks the table adds up
(pass+fail+error = ran, files with a result + files without = total),
recomputes the printed share to the tenth it is printed to, and requires
the paragraph under it to quote the table's own skip count, file count and
ratio. It does not re-measure conformance: the table is one dated sweep
and is allowed to be older than BASELINE. A dated measurement going stale
is not an error; a derived number contradicting the measurement it is
derived from is.

Both gates were run against the pre-fix READMEs before being run against
the corrected ones: 8 failures, one per drifted fact, including the
phantom `argf` row whose empty "where" cell was the tell.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 5aec013 into main Nov 9, 2026
26 checks passed
@tannevaled
tannevaled deleted the readme-derived-numbers-gate branch November 9, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant