diff --git a/README.md b/README.md index e1f342bc..95723dbd 100644 --- a/README.md +++ b/README.md @@ -6,14 +6,14 @@ [![License](https://img.shields.io/badge/license-BSD--3--Clause-blue)](LICENSE) [![Go](https://img.shields.io/github/go-mod/go-version/go-embedded-ruby/ruby?label=go&color=00ADD8)](https://go.dev/dl/) [![Release](https://img.shields.io/github/v/release/go-embedded-ruby/ruby?label=release&color=blue)](https://github.com/go-embedded-ruby/ruby/releases/latest) -[![ruby/spec](https://img.shields.io/badge/ruby%2Fspec-23%2C673%20examples%20passing-1a7f37)](#runtime-conformance--rubyspec) +[![ruby/spec](https://img.shields.io/badge/ruby%2Fspec-23%2C672%20examples%20passing-1a7f37)](#runtime-conformance--rubyspec) **A Ruby interpreter written in pure Go, with cgo disabled** — so you can embed it in a Go program with `import "github.com/go-embedded-ruby/ruby"`, or ship it as a single static binary that cross-compiles wherever Go does, with no C toolchain and no libruby. -**How complete is it?** It runs **23,673** of ruby/spec's `language/` + `core/` +**How complete is it?** It runs **23,672** of ruby/spec's `language/` + `core/` examples ([what that counts](#runtime-conformance--rubyspec)) — a large and growing subset of the language, and **not** a drop-in replacement for CRuby. Read [What does not work yet](#what-does-not-work-yet) before you depend on it; the @@ -1256,7 +1256,7 @@ specification of the language — through `rbgo` under a minimal MSpec-compatibl shim, and gates CI on a **per-file** baseline ([`scripts/conformance/rubyspec/`](scripts/conformance/rubyspec/), in `BASELINE`). No file may pass fewer examples than its record or stop loading, so -measured language conformance moves in one direction; the total, **23,673**, is +measured language conformance moves in one direction; the total, **23,672**, is a derived summary rather than the thing gated. Run it with `scripts/conformance/rubyspec/run.sh`, and see *Runtime conformance* under *Status* for the full breakdown. diff --git a/scripts/conformance/rubyspec/BASELINE b/scripts/conformance/rubyspec/BASELINE index 56ea62d7..e85ccd94 100644 --- a/scripts/conformance/rubyspec/BASELINE +++ b/scripts/conformance/rubyspec/BASELINE @@ -1577,7 +1577,7 @@ OK core/queue/pop_spec.rb 20 OK core/queue/push_spec.rb 1 OK core/queue/shift_spec.rb 1 OK core/queue/size_spec.rb 1 -OK core/random/bytes_spec.rb 9 +OK core/random/bytes_spec.rb 8 OK core/random/default_spec.rb 1 OK core/random/equal_value_spec.rb 2 OK core/random/new_seed_spec.rb 3 diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index de702197..448cb228 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -106,6 +106,34 @@ lane that can change the gate it just ran, and the person who asked for the re-record is exactly the person who should read the delta before it becomes the record. +### ⛔ A flaky file must be recorded at its FLOOR, not at the draw you got + +`core/random/bytes_spec.rb` answers **8 or 9 on the same binary** — documented +in four places in this repository before it bit, including `cmd/specdelta`'s own +comment, which exists *because* that file once reported `-1` between two engines +that differed only in a version string. + +The 4.0.9 re-record drew **9** and banked it. The gate then failed on the very +next pull request, naming a regression that was a coin flip: + +```text +passing examples: 23672 (baseline 23673, -1) +-- REGRESSED (1) -- + core/random/bytes_spec.rb: 8 passing, baseline 9 (-1) +``` + +That is not a conformance failure, it is a **~50 % flaky gate**, self-inflicted +by recording a maximum as if it were a guarantee. A baseline row means *"no run +may come in below this"*, so for an unstable file the only honest row is its +**lowest** observed value: at 8 the gate is deterministic, a 9 shows up as an +improvement, and a genuine drop to 7 is still caught. + +Its row is therefore pinned to `8` by hand, and a re-record that raises it again +should be corrected rather than accepted — the number to trust is the floor, not +the luckiest sweep. `cmd/specdelta` already refuses to build a verdict on a +single run (`-repeat`, and the `UNSTABLE` marking); the ratchet has no such +notion, which is the real gap. + ### ⛔ A re-record reads the old baseline first, and refuses to bank a loss `-update` used to write straight over `BASELINE` **without reading it**, so the