From 8e27bd6de9a09aefd4465e51b3a10f152a3b7500 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Wed, 4 Nov 2026 15:21:34 +0100 Subject: [PATCH 1/2] Gate the README's conformance total, which nothing checked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs site and the landing page have had conformancefigure for weeks. The README — the page most people actually read — did not, and it showed. Pointing the published v0.7.0 tool at this repository reported: 0 total claim(s) and 0 version claim(s) checked conformancefigure: found no statement of the total; either the vocabulary no longer matches how the docs phrase it, or the figure was dropped Zero, on a README that states the total twice. It refused rather than passing, which is the only reason the gap was visible at all. Three sentences were phrased so the gate could not reach them, and one of the three was wrong: * line 16 said **"It runs 23,672"**. 23,672 is what *passes*; rbgo runs considerably more — the dated sweep ran 24,116 examples. The figure was right and the verb was not, and a hand-maintained README is exactly where that survives. Now "It passes", which is also what the docs site says. * line 1267 said "the total, 23,672, is a derived summary" — an apposition the pattern cannot cross. Now "their sum is 23,672,". * line 782 said "| `RUBY_VERSION` | rbgo reports `"4.0.9"`" — the table pipe and "rbgo" sit between the constant and the verb. Now the cell repeats the constant. Nothing was added to the tool's vocabulary for this. The alternative was to teach it "runs", and that would have taught it to bless a sentence that was wrong. After: **2 total claim(s) and 1 version claim(s) checked**, all agreeing. The new `readme-figure` job passes every input locally (--baseline/--runtime/--shim). The tool's default is to fetch those three files from this repository's main branch, which inside this repository is a self-reference: a pull request that re-records BASELINE and updates the README in one commit would be refused for disagreeing with the main it is about to replace. Verified to bite before committing — README.md:16 set to 23,999: README.md:16: claims 23999, the gate records 23672 conformancefigure: 1 claim(s) disagree with the ruby repository exit status 1 Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 38 ++++++++++++++++++++++++++++++++++++++ README.md | 6 +++--- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1f751d0a..677c5b52 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -411,3 +411,41 @@ jobs: env: RBGO_BUILD_IT: '1' run: go test -count=1 -timeout=30m -v ./cmd/rbgo + + # ⛔ The README states the conformance total and the Ruby version the gate + # counts at, and nothing checked either. The docs site and the landing page + # have had this guard for weeks; the README -- the page most people actually + # read -- was maintained by hand. On 2026-11-04 it said "It RUNS 23,672 of + # ruby/spec's examples": the figure was right and the verb was not, because + # 23,672 is what passes and rbgo runs considerably more. + # + # Every input is LOCAL, which is the point. The published tool fetches + # BASELINE, kernel_introspect.go and spec_helper.rb from this repository's + # main branch by default; inside this repository that is a self-reference, + # and a pull request that re-records the baseline and updates the README in + # the same commit would be refused for disagreeing with a main it is about to + # replace. --baseline/--runtime/--shim point it at the tree being tested. + # + # The version is PINNED and must be moved deliberately: a gate pinned to a + # version whose vocabulary cannot reach a sentence passes without reading it + # (go-embedded-ruby/docs#44), and that is indistinguishable from a page with + # nothing wrong. + # + # Verified to BITE before it was committed: with README.md:16 changed to + # 23,999 the step exits 1 with + # README.md:16: claims 23999, the gate records 23672 + readme-figure: + name: the README's conformance total matches BASELINE + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-go@v7 + with: + go-version: 1.27.3 + - name: The stated total and version match this tree's own gate + run: >- + go run github.com/go-embedded-ruby/docs/tools/conformancefigure@v0.7.0 + --root . + --baseline scripts/conformance/rubyspec/BASELINE + --runtime internal/vm/kernel_introspect.go + --shim scripts/conformance/rubyspec/spec_helper.rb diff --git a/README.md b/README.md index b7dd2c76..156429bc 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ in a Go program with `import "github.com/go-embedded-ruby/ruby"`, or ship it as single static binary that cross-compiles wherever Go does, with no C toolchain and no libruby. -**How complete is it?** It runs **23,672** of ruby/spec's `language/` + `core/` +**How complete is it?** It passes **23,672** of ruby/spec's `language/` + `core/` examples ([what that counts](#runtime-conformance--rubyspec)) — a large and growing subset of the language, and **not** a drop-in replacement for CRuby. Read [What does not work yet](#what-does-not-work-yet) before you depend on it; the @@ -779,7 +779,7 @@ Otherwise: | | how it differs from MRI 4.0.5 | | --- | --- | -| `RUBY_VERSION` | rbgo reports `"4.0.9"`, the latest MRI release (2026-10-23). The Ruby 4.0 **`Ruby` namespace** *is* implemented — all nine of `Ruby::{VERSION,RELEASE_DATE,PLATFORM,PATCHLEVEL,REVISION,COPYRIGHT,ENGINE,ENGINE_VERSION,DESCRIPTION}` are the **same objects** as their `RUBY_`-prefixed globals, as `version.c` defines them, so `Ruby::VERSION.equal?(RUBY_VERSION)` is `true`. The move from `3.4.1` was measured before it was made, with both engines built from one commit minus the version string: **+144 passing** over `core` and `language`, and **27 examples that went red**, which no pass total shows — see [#799](https://github.com/go-embedded-ruby/ruby/issues/799) for the list of what each of the 27 needs | +| `RUBY_VERSION` | `RUBY_VERSION` reports `"4.0.9"`, the latest MRI release (2026-10-23). The Ruby 4.0 **`Ruby` namespace** *is* implemented — all nine of `Ruby::{VERSION,RELEASE_DATE,PLATFORM,PATCHLEVEL,REVISION,COPYRIGHT,ENGINE,ENGINE_VERSION,DESCRIPTION}` are the **same objects** as their `RUBY_`-prefixed globals, as `version.c` defines them, so `Ruby::VERSION.equal?(RUBY_VERSION)` is `true`. The move from `3.4.1` was measured before it was made, with both engines built from one commit minus the version string: **+144 passing** over `core` and `language`, and **27 examples that went red**, which no pass total shows — see [#799](https://github.com/go-embedded-ruby/ruby/issues/799) for the list of what each of the 27 needs | | `Numeric#to_int` | not defined — `Complex(2.9, 0).to_int` raises `NoMethodError`; MRI returns `2` | | `File::Stat#==` | `a == b` is `false` for two stats of the same file (MRI: `true`); `a.==(b)` answers `true` | | `File#stat` on a file unlinked while open | raises `Errno::ENOENT`; MRI answers from the open descriptor. rbgo's streams are buffered by path and hold no descriptor | @@ -1264,7 +1264,7 @@ specification of the language — through `rbgo` under a minimal MSpec-compatibl shim, and gates CI on a **per-file** baseline ([`scripts/conformance/rubyspec/`](scripts/conformance/rubyspec/), in `BASELINE`). No file may pass fewer examples than its record or stop loading, so -measured language conformance moves in one direction; the total, **23,672**, is +measured language conformance moves in one direction; their sum is **23,672**, a derived summary rather than the thing gated. Run it with `scripts/conformance/rubyspec/run.sh`, and see *Runtime conformance* under *Status* for the full breakdown. From 8f1e330806e50dde7c63af6f4e85cc1c7d013399 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Wed, 4 Nov 2026 16:40:06 +0100 Subject: [PATCH 2/2] Gate the Ruby version the README counts at, and drop a limitation that is gone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This branch started on a false premise, which is worth stating because the correction is the interesting part. I set out to gate the README's conformance total, believing nothing checked it, and wired a CI job around the published `conformancefigure`. Then I read `TestReadmeConformanceCountMatchesTheBaseline`, which has been checking exactly that for weeks — against BASELINE in the same tree, with no network and no pinned external version. My job would have duplicated a better check and added a dependency. It is not in this commit. What that test does *not* cover is the version the total is counted at, and that version selects the population counted: the move from 3.4.1 to 4.0.9 changed which `ruby_version_is` guards admit examples (+144 passing, 27 gone red). The same number means a different thing at a different version. TestReadmeRubyVersionMatchesTheRuntime reads `rubyVersion` from internal/vm/kernel_introspect.go and checks it two ways: * forward — every present-tense claim ("rbgo reports `"4.0.9"`", "already reflect 4.0.9") must equal it, and fewer matches than shapes is a failure, so a rewording that escapes the patterns fails instead of silently checking less; * inverse — a line mentioning RUBY_VERSION and naming a different version must declare itself historical ("sweep", "the move from"), and the exemptions are logged rather than applied silently. Both halves are needed, which a deliberate break showed. With the table row set to 4.0.8 the forward check named it while the inverse check EXEMPTED the same 4.0.8, because that line also says "the move from": a line-granular marker meant for the version left behind covered a wrong present-tense claim sitting beside it. Two README corrections, both measured rather than read: * line 16 said "It **runs** 23,672". 23,672 is what passes; rbgo runs considerably more. The figure was right and the verb was not, which is why the existing numeric gate was green the whole time — no check on a number can see a wrong verb. * *"A backtick command holds the interpreter lock"* no longer reproduces. The probe the README carried printed `frozen` in October and prints `ran` now, five runs out of five, and rubydiff reports the program AGREE with MRI 4.0.7. Removed, with no cause attributed: naming one without a control would be a guess dressed as a fact. `Timeout.timeout` was re-checked on the same pass and still fails open — rbgo prints `returned` then `nil`, MRI 4.0.7 prints `raised` then `0.3` — so it stays, now with that measurement beside it. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 38 ---------- README.md | 30 ++++---- readme_ruby_version_test.go | 137 ++++++++++++++++++++++++++++++++++++ 3 files changed, 154 insertions(+), 51 deletions(-) create mode 100644 readme_ruby_version_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 677c5b52..1f751d0a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -411,41 +411,3 @@ jobs: env: RBGO_BUILD_IT: '1' run: go test -count=1 -timeout=30m -v ./cmd/rbgo - - # ⛔ The README states the conformance total and the Ruby version the gate - # counts at, and nothing checked either. The docs site and the landing page - # have had this guard for weeks; the README -- the page most people actually - # read -- was maintained by hand. On 2026-11-04 it said "It RUNS 23,672 of - # ruby/spec's examples": the figure was right and the verb was not, because - # 23,672 is what passes and rbgo runs considerably more. - # - # Every input is LOCAL, which is the point. The published tool fetches - # BASELINE, kernel_introspect.go and spec_helper.rb from this repository's - # main branch by default; inside this repository that is a self-reference, - # and a pull request that re-records the baseline and updates the README in - # the same commit would be refused for disagreeing with a main it is about to - # replace. --baseline/--runtime/--shim point it at the tree being tested. - # - # The version is PINNED and must be moved deliberately: a gate pinned to a - # version whose vocabulary cannot reach a sentence passes without reading it - # (go-embedded-ruby/docs#44), and that is indistinguishable from a page with - # nothing wrong. - # - # Verified to BITE before it was committed: with README.md:16 changed to - # 23,999 the step exits 1 with - # README.md:16: claims 23999, the gate records 23672 - readme-figure: - name: the README's conformance total matches BASELINE - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - - uses: actions/setup-go@v7 - with: - go-version: 1.27.3 - - name: The stated total and version match this tree's own gate - run: >- - go run github.com/go-embedded-ruby/docs/tools/conformancefigure@v0.7.0 - --root . - --baseline scripts/conformance/rubyspec/BASELINE - --runtime internal/vm/kernel_introspect.go - --shim scripts/conformance/rubyspec/spec_helper.rb diff --git a/README.md b/README.md index 156429bc..7b8df778 100644 --- a/README.md +++ b/README.md @@ -749,6 +749,19 @@ been removed — two were, on that pass: sends one byte to each; `warn` goes to stderr. Covered by `internal/vm/diagnostic_stream_test.go`. +The two prose entries were re-checked again on `6eb7e4ad` (darwin/arm64, +2026-11-04), same method and same MRI; the table further down was *not* re-swept +on that pass, so its dates still stand. One of the two had gone: + +- *"A backtick command holds the interpreter lock."* While `` `cmd` `` ran, + every other Ruby `Thread` was said to stop. The probe this file carried + printed `frozen` in October and prints `ran` now, five runs out of five, and + `rubydiff` reports the whole program AGREE with MRI 4.0.7 — the two engines + produce the same output, not merely different-but-no-longer-compared output. + It is removed rather than reworded, and what closed it is deliberately *not* + attributed here: an entry that stops reproducing is a measurement, and naming + a cause without a control would be a guess dressed as a fact. + **`Timeout.timeout` never times out.** The module and `Timeout::Error` exist and `rescue Timeout::Error` resolves, but the block runs to completion and its value is returned: enforcing a deadline needs pre-emptive interruption of a running @@ -763,23 +776,14 @@ This one **fails open**, which is why it is stated here rather than left in the table below. `Timeout.timeout` is the ordinary Ruby way to bound work that might hang, so code relying on it for that is unprotected under rbgo — and unprotected *silently*, because the call returns successfully. The block is also yielded -`nil` where MRI yields the limit. - -**A backtick command holds the interpreter lock.** While `` `cmd` `` runs, every -other Ruby `Thread` is stopped; under MRI they keep running. `Kernel#sleep` does -yield, so this is specific to the subprocess path: - -```console -$ rbgo -e 't = Thread.new { loop { $n = ($n||0)+1; sleep 0.01 } }; sleep 0.05 - b = $n; `sleep 1`; puts($n > b ? "ran" : "frozen")' -frozen # MRI 4.0.7: ran -``` +`nil` where MRI yields the limit — re-measured on 2026-11-04, where rbgo prints +`returned` then `nil` and MRI 4.0.7 prints `raised` then `0.3`. Otherwise: | | how it differs from MRI 4.0.5 | | --- | --- | -| `RUBY_VERSION` | `RUBY_VERSION` reports `"4.0.9"`, the latest MRI release (2026-10-23). The Ruby 4.0 **`Ruby` namespace** *is* implemented — all nine of `Ruby::{VERSION,RELEASE_DATE,PLATFORM,PATCHLEVEL,REVISION,COPYRIGHT,ENGINE,ENGINE_VERSION,DESCRIPTION}` are the **same objects** as their `RUBY_`-prefixed globals, as `version.c` defines them, so `Ruby::VERSION.equal?(RUBY_VERSION)` is `true`. The move from `3.4.1` was measured before it was made, with both engines built from one commit minus the version string: **+144 passing** over `core` and `language`, and **27 examples that went red**, which no pass total shows — see [#799](https://github.com/go-embedded-ruby/ruby/issues/799) for the list of what each of the 27 needs | +| `RUBY_VERSION` | rbgo reports `"4.0.9"`, the latest MRI release (2026-10-23). The Ruby 4.0 **`Ruby` namespace** *is* implemented — all nine of `Ruby::{VERSION,RELEASE_DATE,PLATFORM,PATCHLEVEL,REVISION,COPYRIGHT,ENGINE,ENGINE_VERSION,DESCRIPTION}` are the **same objects** as their `RUBY_`-prefixed globals, as `version.c` defines them, so `Ruby::VERSION.equal?(RUBY_VERSION)` is `true`. The move from `3.4.1` was measured before it was made, with both engines built from one commit minus the version string: **+144 passing** over `core` and `language`, and **27 examples that went red**, which no pass total shows — see [#799](https://github.com/go-embedded-ruby/ruby/issues/799) for the list of what each of the 27 needs | | `Numeric#to_int` | not defined — `Complex(2.9, 0).to_int` raises `NoMethodError`; MRI returns `2` | | `File::Stat#==` | `a == b` is `false` for two stats of the same file (MRI: `true`); `a.==(b)` answers `true` | | `File#stat` on a file unlinked while open | raises `Errno::ENOENT`; MRI answers from the open descriptor. rbgo's streams are buffered by path and hold no descriptor | @@ -1264,7 +1268,7 @@ specification of the language — through `rbgo` under a minimal MSpec-compatibl shim, and gates CI on a **per-file** baseline ([`scripts/conformance/rubyspec/`](scripts/conformance/rubyspec/), in `BASELINE`). No file may pass fewer examples than its record or stop loading, so -measured language conformance moves in one direction; their sum is **23,672**, +measured language conformance moves in one direction; the total, **23,672**, is a derived summary rather than the thing gated. Run it with `scripts/conformance/rubyspec/run.sh`, and see *Runtime conformance* under *Status* for the full breakdown. diff --git a/readme_ruby_version_test.go b/readme_ruby_version_test.go new file mode 100644 index 00000000..36109317 --- /dev/null +++ b/readme_ruby_version_test.go @@ -0,0 +1,137 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +package ruby_test + +import ( + "os" + "regexp" + "strings" + "testing" +) + +// runtimeRubyVersion reads the single constant the interpreter answers +// RUBY_VERSION with. It is the same file the documentation site's figure gate +// reads, and it is the only authority here: a README that names a version the +// runtime does not report is wrong whichever of the two moved. +func runtimeRubyVersion(t *testing.T) string { + t.Helper() + const path = "internal/vm/kernel_introspect.go" + src, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + m := regexp.MustCompile(`(?m)^\trubyVersion\s*=\s*"(\d+\.\d+\.\d+)"`).FindSubmatch(src) + if m == nil { + t.Fatalf("%s has no `rubyVersion = \"x.y.z\"`; this test cannot judge the README without it", path) + } + return string(m[1]) +} + +// readmeRubyVersionClaims are the shapes in which the README states the version +// the interpreter reports NOW. Each is a sentence that exists in the file +// today; nothing speculative is listed, because a pattern written for a +// sentence nobody writes is a false positive waiting for the sentence that +// does. +// +// ⛔ The count is asserted below. A vocabulary list is a filter, and a filter +// that matches nothing looks exactly like a file with nothing wrong -- which is +// how go-embedded-ruby/docs#44 shipped two stale figures past a gate that +// reported itself clean. If a future rewording drops out of this list the +// denominator falls and the test fails, instead of quietly checking less. +var readmeRubyVersionClaims = []*regexp.Regexp{ + // | `RUBY_VERSION` | rbgo reports `"4.0.9"`, the latest MRI release … + regexp.MustCompile("rbgo reports `\"(\\d+\\.\\d+\\.\\d+)\"`"), + // … and the summary above count the current `BASELINE` and so already + // reflect 4.0.9. + regexp.MustCompile(`already reflect (\d+\.\d+\.\d+)`), +} + +// historicalMarkers are the phrases that let a line name a version OTHER than +// the one the runtime reports. Two lines qualify today, and they are different +// kinds of past tense: +// +// "a sweep taken at `RUBY_VERSION` 3.4.1" -- a dated measurement +// "The move from `3.4.1` was measured …" -- the version left behind +// +// Naming the exemptions rather than forbidding every deviation is deliberate. A +// check loosened to "any mismatch is fine" has stopped checking; this one still +// fails on an undeclared mismatch, and logs the lines it let through so the +// exemption stays visible instead of becoming invisible slack. +var historicalMarkers = []string{"sweep", "the move from"} + +// historicalMarkerOf reports which marker, if any, lets this line name an older +// version. +func historicalMarkerOf(line string) (string, bool) { + low := strings.ToLower(line) + for _, m := range historicalMarkers { + if strings.Contains(low, m) { + return m, true + } + } + return "", false +} + +// TestReadmeRubyVersionMatchesTheRuntime: the README's conformance TOTAL is +// gated by TestReadmeConformanceCountMatchesTheBaseline, which reads BASELINE. +// The version that total is counted AT was gated by nothing, and it selects the +// population counted: the move from 3.4.1 to 4.0.9 changed which +// `ruby_version_is` guards admit examples (+144 passing, 27 gone red), so the +// same number means a different thing at a different version. +func TestReadmeRubyVersionMatchesTheRuntime(t *testing.T) { + want := runtimeRubyVersion(t) + + readme, err := os.ReadFile("README.md") + if err != nil { + t.Fatalf("read README.md: %v", err) + } + lines := strings.Split(string(readme), "\n") + + checked := 0 + for i, line := range lines { + for _, re := range readmeRubyVersionClaims { + for _, m := range re.FindAllStringSubmatch(line, -1) { + checked++ + if m[1] != want { + t.Errorf("README.md:%d claims the runtime reports %s, kernel_introspect.go says %s\n %s", + i+1, m[1], want, strings.TrimSpace(line)) + } + } + } + } + if checked < len(readmeRubyVersionClaims) { + t.Errorf("only %d present-tense version claim(s) matched in README.md, one per shape was expected at least; "+ + "a rewording has taken a claim out of this test's reach rather than out of the README", checked) + } + + // The inverse: a line that talks about RUBY_VERSION and names a DIFFERENT + // version must say it is historical. + // + // ⛔ The two halves cover each other's blind spot, which is why both are + // here. Measured by setting the table row to 4.0.8: the forward check named + // it, while this one EXEMPTED the same 4.0.8 — the line also says "the move + // from", so a marker meant for the version left behind covered a wrong + // present-tense claim sitting beside it. A line-granular exemption cannot + // tell the two apart; the forward check, which reads the claim itself, can. + version := regexp.MustCompile(`\b(\d+\.\d+\.\d+)\b`) + exempt := 0 + for i, line := range lines { + if !strings.Contains(line, "RUBY_VERSION") { + continue + } + for _, m := range version.FindAllStringSubmatch(line, -1) { + if m[1] == want { + continue + } + if marker, ok := historicalMarkerOf(line); ok { + exempt++ + t.Logf("README.md:%d names %s and is exempt: the line says %q", i+1, m[1], marker) + continue + } + t.Errorf("README.md:%d names RUBY_VERSION %s, the runtime reports %s, and the line says none of %v\n %s", + i+1, m[1], want, historicalMarkers, strings.TrimSpace(line)) + } + } + t.Logf("%d present-tense version claim(s) checked against %s, %d historical line(s) exempt", checked, want, exempt) +}