From c33e588854eed042b2a309d01d5cab8077b59682 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Tue, 3 Nov 2026 21:25:27 +0100 Subject: [PATCH] Pin the flaky spec file's baseline to its FLOOR, not to the draw we got `core/random/bytes_spec.rb` answers 8 or 9 on the same binary. That is documented in FOUR places in this repository, including cmd/specdelta's own comment -- which exists because that file once reported -1 between two engines differing only in a version string. The 4.0.9 re-record (#882) drew 9 and banked it. The gate then failed on the very next pull request, naming a regression that was a coin flip: passing examples: 23672 (baseline 23673, -1) -- REGRESSED (1) -- core/random/bytes_spec.rb: 8 passing, baseline 9 (-1) That is not a conformance failure. It is a ~50% flaky gate on main and on every branch, self-inflicted by recording a maximum as though it were a guarantee. A baseline row means "no run may come in below this". For an unstable file the only honest row is its LOWEST observed value: at 8 the gate is deterministic, a 9 is reported as an improvement, and a real drop to 7 is still caught. Nothing about the shrink-only property is weakened -- the floor is what the gate was always supposed to hold. BASELINE core/random/bytes_spec.rb 9 -> 8 README 23,673 -> 23,672 in all three places, named by TestReadmeConformanceCountMatchesTheBaseline rather than by hand The conformance README now carries why, so the next re-record does not put it back. The deeper gap is that `cmd/ratchet` has no notion of an unstable file at all, while `cmd/specdelta` already refuses to build a verdict on a single run (-repeat, and the UNSTABLE marking). Filed separately. Refs #799 Co-Authored-By: Claude Opus 5 --- README.md | 6 +++--- scripts/conformance/rubyspec/BASELINE | 2 +- scripts/conformance/rubyspec/README.md | 28 ++++++++++++++++++++++++++ 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e1f342bc..95723dbd 100644 --- a/README.md +++ b/README.md @@ -6,14 +6,14 @@ [![License](https://img.shields.io/badge/license-BSD--3--Clause-blue)](LICENSE) [![Go](https://img.shields.io/github/go-mod/go-version/go-embedded-ruby/ruby?label=go&color=00ADD8)](https://go.dev/dl/) [![Release](https://img.shields.io/github/v/release/go-embedded-ruby/ruby?label=release&color=blue)](https://github.com/go-embedded-ruby/ruby/releases/latest) -[![ruby/spec](https://img.shields.io/badge/ruby%2Fspec-23%2C673%20examples%20passing-1a7f37)](#runtime-conformance--rubyspec) +[![ruby/spec](https://img.shields.io/badge/ruby%2Fspec-23%2C672%20examples%20passing-1a7f37)](#runtime-conformance--rubyspec) **A Ruby interpreter written in pure Go, with cgo disabled** — so you can embed it in a Go program with `import "github.com/go-embedded-ruby/ruby"`, or ship it as a single static binary that cross-compiles wherever Go does, with no C toolchain and no libruby. -**How complete is it?** It runs **23,673** of ruby/spec's `language/` + `core/` +**How complete is it?** It runs **23,672** of ruby/spec's `language/` + `core/` examples ([what that counts](#runtime-conformance--rubyspec)) — a large and growing subset of the language, and **not** a drop-in replacement for CRuby. Read [What does not work yet](#what-does-not-work-yet) before you depend on it; the @@ -1256,7 +1256,7 @@ specification of the language — through `rbgo` under a minimal MSpec-compatibl shim, and gates CI on a **per-file** baseline ([`scripts/conformance/rubyspec/`](scripts/conformance/rubyspec/), in `BASELINE`). No file may pass fewer examples than its record or stop loading, so -measured language conformance moves in one direction; the total, **23,673**, is +measured language conformance moves in one direction; the total, **23,672**, is a derived summary rather than the thing gated. Run it with `scripts/conformance/rubyspec/run.sh`, and see *Runtime conformance* under *Status* for the full breakdown. diff --git a/scripts/conformance/rubyspec/BASELINE b/scripts/conformance/rubyspec/BASELINE index 56ea62d7..e85ccd94 100644 --- a/scripts/conformance/rubyspec/BASELINE +++ b/scripts/conformance/rubyspec/BASELINE @@ -1577,7 +1577,7 @@ OK core/queue/pop_spec.rb 20 OK core/queue/push_spec.rb 1 OK core/queue/shift_spec.rb 1 OK core/queue/size_spec.rb 1 -OK core/random/bytes_spec.rb 9 +OK core/random/bytes_spec.rb 8 OK core/random/default_spec.rb 1 OK core/random/equal_value_spec.rb 2 OK core/random/new_seed_spec.rb 3 diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index de702197..448cb228 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -106,6 +106,34 @@ lane that can change the gate it just ran, and the person who asked for the re-record is exactly the person who should read the delta before it becomes the record. +### ⛔ A flaky file must be recorded at its FLOOR, not at the draw you got + +`core/random/bytes_spec.rb` answers **8 or 9 on the same binary** — documented +in four places in this repository before it bit, including `cmd/specdelta`'s own +comment, which exists *because* that file once reported `-1` between two engines +that differed only in a version string. + +The 4.0.9 re-record drew **9** and banked it. The gate then failed on the very +next pull request, naming a regression that was a coin flip: + +```text +passing examples: 23672 (baseline 23673, -1) +-- REGRESSED (1) -- + core/random/bytes_spec.rb: 8 passing, baseline 9 (-1) +``` + +That is not a conformance failure, it is a **~50 % flaky gate**, self-inflicted +by recording a maximum as if it were a guarantee. A baseline row means *"no run +may come in below this"*, so for an unstable file the only honest row is its +**lowest** observed value: at 8 the gate is deterministic, a 9 shows up as an +improvement, and a genuine drop to 7 is still caught. + +Its row is therefore pinned to `8` by hand, and a re-record that raises it again +should be corrected rather than accepted — the number to trust is the floor, not +the luckiest sweep. `cmd/specdelta` already refuses to build a verdict on a +single run (`-repeat`, and the `UNSTABLE` marking); the ratchet has no such +notion, which is the real gap. + ### ⛔ A re-record reads the old baseline first, and refuses to bank a loss `-update` used to write straight over `BASELINE` **without reading it**, so the