Skip to content

Pin the flaky spec file's baseline to its floor, not to the draw we got #885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tannevaled merged 1 commit into from
Nov 3, 2026
Merged

Pin the flaky spec file's baseline to its floor, not to the draw we got #885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@
[![License](https://img.shields.io/badge/license-BSD--3--Clause-blue)](LICENSE)
[![Go](https://img.shields.io/github/go-mod/go-version/go-embedded-ruby/ruby?label=go&color=00ADD8)](https://go.dev/dl/)
[![Release](https://img.shields.io/github/v/release/go-embedded-ruby/ruby?label=release&color=blue)](https://github.com/go-embedded-ruby/ruby/releases/latest)
[![ruby/spec](https://img.shields.io/badge/ruby%2Fspec-23%2C673%20examples%20passing-1a7f37)](#runtime-conformance--rubyspec)
[![ruby/spec](https://img.shields.io/badge/ruby%2Fspec-23%2C672%20examples%20passing-1a7f37)](#runtime-conformance--rubyspec)

**A Ruby interpreter written in pure Go, with cgo disabled** — so you can embed it
in a Go program with `import "github.com/go-embedded-ruby/ruby"`, or ship it as a
single static binary that cross-compiles wherever Go does, with no C toolchain and
no libruby.

**How complete is it?** It runs **23,673** of ruby/spec's `language/` + `core/`
**How complete is it?** It runs **23,672** of ruby/spec's `language/` + `core/`
examples ([what that counts](#runtime-conformance--rubyspec)) — a large and growing
subset of the language, and **not** a drop-in replacement for CRuby. Read
[What does not work yet](#what-does-not-work-yet) before you depend on it; the
Expand Down Expand Up @@ -1256,7 +1256,7 @@ specification of the language — through `rbgo` under a minimal MSpec-compatibl
shim, and gates CI on a **per-file** baseline
([`scripts/conformance/rubyspec/`](scripts/conformance/rubyspec/), in
`BASELINE`). No file may pass fewer examples than its record or stop loading, so
measured language conformance moves in one direction; the total, **23,673**, is
measured language conformance moves in one direction; the total, **23,672**, is
a derived summary rather than the thing gated. Run it with
`scripts/conformance/rubyspec/run.sh`, and see *Runtime conformance* under
*Status* for the full breakdown.
Expand Down
2 changes: 1 addition & 1 deletion scripts/conformance/rubyspec/BASELINE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1577,7 +1577,7 @@ OK core/queue/pop_spec.rb 20
OK core/queue/push_spec.rb 1
OK core/queue/shift_spec.rb 1
OK core/queue/size_spec.rb 1
OK core/random/bytes_spec.rb 9
OK core/random/bytes_spec.rb 8
OK core/random/default_spec.rb 1
OK core/random/equal_value_spec.rb 2
OK core/random/new_seed_spec.rb 3
Expand Down
28 changes: 28 additions & 0 deletions scripts/conformance/rubyspec/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,34 @@ lane that can change the gate it just ran, and the person who asked for the
re-record is exactly the person who should read the delta before it becomes the
record.

### ⛔ A flaky file must be recorded at its FLOOR, not at the draw you got

`core/random/bytes_spec.rb` answers **8 or 9 on the same binary** — documented
in four places in this repository before it bit, including `cmd/specdelta`'s own
comment, which exists *because* that file once reported `-1` between two engines
that differed only in a version string.

The 4.0.9 re-record drew **9** and banked it. The gate then failed on the very
next pull request, naming a regression that was a coin flip:

```text
passing examples: 23672 (baseline 23673, -1)
-- REGRESSED (1) --
core/random/bytes_spec.rb: 8 passing, baseline 9 (-1)
```

That is not a conformance failure, it is a **~50 % flaky gate**, self-inflicted
by recording a maximum as if it were a guarantee. A baseline row means *"no run
may come in below this"*, so for an unstable file the only honest row is its
**lowest** observed value: at 8 the gate is deterministic, a 9 shows up as an
improvement, and a genuine drop to 7 is still caught.

Its row is therefore pinned to `8` by hand, and a re-record that raises it again
should be corrected rather than accepted — the number to trust is the floor, not
the luckiest sweep. `cmd/specdelta` already refuses to build a verdict on a
single run (`-repeat`, and the `UNSTABLE` marking); the ratchet has no such
notion, which is the real gap.

### ⛔ A re-record reads the old baseline first, and refuses to bank a loss

`-update` used to write straight over `BASELINE` **without reading it**, so the
Expand Down
Loading