Skip to content

conformance: derive the stubbed-helper list from the shim - #47

Merged
tannevaled merged 1 commit into
mainfrom
conformance-derived-numbers
Nov 9, 2026
Merged

tannevaled merged 1 commit into
mainfrom
conformance-derived-numbers

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

conformance.md told readers the shim stubs "a few MSpec matchers (complain, output, ruby_exe, …)". Read against spec_helper.rb:

helpers
stubbed (raise SpecSkip) ruby_exe (:882), ruby_cmd (:883), output_to_fd (:280)
implemented complain (:249), output (:279), be_computed_by (:199)

Two of the three named are implemented, and ruby_cmd — stubbed one line below the ruby_exe that was named — appeared nowhere on the site.

The error runs in the direction that hides work. An example using a genuinely stubbed helper skips; one using complain runs, and when the behaviour is missing it fails. A reader sent to look for it among the skips does not find it there — ruby#909 was nearly missed exactly so. ruby#911 corrected the same sentence in the ruby repository and this page kept the wrong one, which is why this PR adds a check rather than a third corrected paragraph.

The check

tools/conformancefigure already fetches spec_helper.rb for CUR_VERSION. It now also reads every def … raise SpecSkip out of it and judges the pages against that set in both directions — a helper called stubbed that is not, and a stubbed helper the site names nowhere.

Three decisions worth the review:

  • It is a vocabulary with its own self-test, like the total's and the version's. "Stub" is an ordinary word on this site: webassembly.md has a graceful stub row listing sqlite3, sequel, activerecord, and the paragraph recording this very correction says "an example using a stubbed helper skips; one using complain runs". Matched on the word alone, both read as claims — measured: the first run of this check failed conformance.md on its own explanation. It is now three phrasings, with those two lines as negative cases.
  • The unreadable-shim refusal is held until the findings are printed. The file's own ⛔ note says a structural complaint must never arrive before a concrete disagreement. Written the obvious way (refuse early, like the baseline and the version), a missing stub list swallowed every stale-total finding in the suite — eight tests went from naming a stale figure to naming an unreadable shim.
  • skip is deliberately outside the set, and the comment says why: it raises SkippedSpecError because the spec asked to be skipped. That is mspec behaviour rbgo implements, not a hole in the shim. A census that does not state its population gets read as a total.

One hole the walk cannot close by itself: the completeness check only applies where the list is made at all, so deleting the enumeration outright would leave nothing to be incomplete about. TestTheSiteNamesEveryStubbedHelper walks the real content/en and requires every stubbed helper to be named, independently of any figure on the page, so a figure refresh never makes it noisy.

Also

The unswept ruby/spec trees are library/, optional/, command_line/ and security/. command_line/ — 32 spec files — was missing from that list here, on the landing page, and in the ruby README. Corrected on this page; the landing follows once this tool is tagged.

Verification

$ go test ./tools/conformancefigure/
ok

$ go run ./tools/conformancefigure --root content/en --baseline … --runtime … --shim …
5 total claim(s), 2 version claim(s) and 6 stubbed-helper mention(s) checked
against the gate's 23672 at Ruby 4.0.9 (derived from RUBY_VERSION);
the shim stubs output_to_fd, ruby_cmd, ruby_exe

run against the real BASELINE, kernel_introspect.go and spec_helper.rb from a current rbgo checkout.

Refs ruby#911.
Refs go-embedded-ruby/ruby#909 — it stays open; this page only stops sending its
reader to look for that failure among the skips.

🤖 Generated with Claude Code

conformance.md said the shim stubs "a few MSpec matchers (`complain`,
`output`, `ruby_exe`, …)". Read against spec_helper.rb, two of the three
named are implemented and the list was missing one:

  stubbed   ruby_exe (:882), ruby_cmd (:883), output_to_fd (:280)
  NOT       complain (:249), output (:279), be_computed_by (:199)

The error runs in the direction that hides work. An example using a
genuinely stubbed helper SKIPS; one using `complain` RUNS, and when the
behaviour is missing it FAILS. A reader sent to look for it among the
skips does not find it there -- ruby#909 was nearly missed exactly so.
ruby#911 corrected the same sentence in the ruby repository and this
page kept the wrong one, which is why this adds a check rather than
another corrected paragraph.

tools/conformancefigure now reads every `def … raise SpecSkip` out of
spec_helper.rb -- the file it already fetches for CUR_VERSION -- and
judges the pages against that set in both directions: a helper called
stubbed that is not, and a stubbed helper the site names nowhere.

Three things it does on purpose:

  - It is a VOCABULARY with its own self-test, like the total's and the
    version's. "Stub" is an ordinary word here: webassembly.md has a
    "graceful stub" row listing `sqlite3`, `sequel`, `activerecord`, and
    the paragraph recording this very correction says "an example using
    a stubbed helper skips; one using `complain` runs". Matched on the
    word alone, both were read as claims and this check failed
    conformance.md on its own explanation. Measured, then narrowed to
    three phrasings, with those two lines as negative cases.
  - The unreadable-shim refusal is HELD until the findings are printed.
    The file's own note says a structural complaint must never arrive
    before a concrete disagreement; written the obvious way, a missing
    stub list swallowed every stale-total finding in the suite.
  - `skip` is outside the set and the comment says so: it raises
    SkippedSpecError because the spec asked to be skipped, which is
    mspec behaviour rbgo implements, not a hole in the shim.

Also: the unswept trees are `library/`, `optional/`, `command_line/` and
`security/`. `command_line/` -- 32 spec files -- was missing from that
list on this page, on the landing page and in the ruby README.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit d988658 into main Nov 9, 2026
2 checks passed
@tannevaled
tannevaled deleted the conformance-derived-numbers branch November 9, 2026 12:32
tannevaled added a commit to go-embedded-ruby/go-embedded-ruby.github.io that referenced this pull request Nov 9, 2026
Two corrections, both of a list this page states and nothing checked.

The skip line said 477 examples are skipped "because our shim stubs the
matcher they need" and named none of them. Read out of spec_helper.rb,
they are `ruby_exe` (:882), `ruby_cmd` (:883) and `output_to_fd` (:280).
The docs site named two helpers that are IMPLEMENTED (`complain`,
`output`) and missed `ruby_cmd` entirely; that is fixed in
go-embedded-ruby/docs#47, and the same read is now a gate.

The unswept trees are `library/`, `optional/`, `command_line/` and
`security/`. `command_line/` -- 32 spec files -- was missing from that
sentence here, in index.html, on the docs site and in the ruby README.

The conformancefigure pin moves to v0.8.0, which reads every
`def … raise SpecSkip` out of spec_helper.rb and judges this page in both
directions: a helper called stubbed that is not, and a stubbed helper the
page names nowhere.

Verified with the exact published command this lane runs, fetched through
the proxy:

    $ go run github.com/go-embedded-ruby/docs/tools/conformancefigure@v0.8.0 --root .
    6 total claim(s), 1 version claim(s) and 3 stubbed-helper mention(s) checked
    against the gate's 23672 at Ruby 4.0.9 (derived from RUBY_VERSION);
    the shim stubs output_to_fd, ruby_cmd, ruby_exe

and it bites: with the line rewritten to "stubs `ruby_exe` and
`complain`", the same command exits 1 naming list.html:75 and `complain`.

⛔ What it still cannot do: the completeness half applies only once a
helper is named, so deleting the enumeration outright would leave nothing
to be incomplete about and the lane would stay green. The docs repository
closes that with a test over its own content tree; this page has no
equivalent yet.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant