From 6288c9f061316b7a4ccb4f788f6aa95ca702066c Mon Sep 17 00:00:00 2001 From: tannevaled Date: Wed, 30 Sep 2026 10:01:59 +0200 Subject: [PATCH 1/2] Give the module benchmarks a clock that can see them, and say where each runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `bench/modules/run.sh` timed with `/usr/bin/time -p`, which prints two decimals of a second. Its resolution is 10 ms, and anything faster than one tick reads zero: $ { /usr/bin/time -p /usr/bin/true >/dev/null; } 2>&1 | awk '/real/{print $2}' 0.00 Of the 171 timing cells in the three per-module sections of BENCHMARKS.md, all 171 are multiples of 10 ms; 36 are ten ticks or fewer and 11 are five or fewer. `matrix | 20ms | 50ms | 0.40x` was a two-tick measurement. `tsort | 80ms | 90ms | 0.89x` claimed an 11% win over MRI from a difference of one tick — re-taken, it is 1.42x, a loss. Three changes to the harness: Time::HiRes instead of /usr/bin/time -p, so the figures have the precision the ratios were already being quoted to. A run that exits non-zero is an error, not a fast run. `best()` kept the smallest time of N without ever looking at the exit status, so a command that died immediately won its own series and got published. It now marks that arm and carries on with the others rather than timing a failure. The runtimes are interleaved, one run each in rotation with the order rotating every round, instead of each taking its own block of wall-clock time. A machine whose load moves during a block charges the movement to whichever runtime was running, and nothing in the output says so. See go-crdt/crdt#131: the same defect put a published ratio 14% out, and contention does not fall on every arm alike — it falls hardest on the shortest-running one, which is the fastest. The table is re-taken in a new dated section; the June ones are kept as they were taken. MRI is the control and reproduces within a few percent across all 24 rows the two runs share, so what moved in the rbgo column is rbgo's. Five of those 24 change which side of parity they sit on: cmath, prettyprint and tsort were published as wins over MRI and are losses; prime and ipaddr were losses and are wins. `set` reads 38.2x where June published 10.09x, and that one is not the clock. #56 bound Set to go-ruby-set/set on 2026-06-29; #330 reimplemented it here on 2026-08-08 for MRI 4.0 conformance, with the bulk of the Ruby-observable API in the prelude; #333's `go mod tidy` dropped the now-unused require the next day. The benchmark exercises exactly that algebra, so it measures prelude Ruby where it used to measure Go. Nothing about the trade was wrong. What was missing was anything that would say it had happened — a page in another org went on attributing this number to a library we had stopped using. So: bench/modules/BACKING.md now records, beside the scripts, which modules run on an external go-ruby library and which run on code in this repository. It is generated, and TestBenchmarkedModulesSayWhereTheirCodeLives fails when it and go.mod disagree. complex, rational and set are in-tree today. Refs #747. Co-Authored-By: Claude Opus 5 --- BENCHMARKS.md | 117 ++++++++++++++++++++++++++++ bench/modules/BACKING.md | 45 +++++++++++ bench/modules/run.sh | 160 ++++++++++++++++++++++++++++++--------- bench_backing_test.go | 154 +++++++++++++++++++++++++++++++++++++ 4 files changed, 441 insertions(+), 35 deletions(-) create mode 100644 bench/modules/BACKING.md create mode 100644 bench_backing_test.go diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 2020ed7d..9d172d7f 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -100,6 +100,123 @@ Native). Reproduce: `AOT=1 RUNS=8 JRUBY=jruby TRUFFLE= bash bench/run.sh 8 lowering of integer-bound methods (`mandelbrot`'s float kernel is not yet AOT-lowered). +## Per-module, re-taken with a clock that can see the rows (2026-09-30) + +The three per-module sections below were measured with `/usr/bin/time -p`, which +prints **two decimals of a second**. Its resolution is 10 ms, and anything faster +than one tick reads zero: + +``` +$ for i in $(seq 10); do { /usr/bin/time -p /usr/bin/true >/dev/null; } 2>&1 | awk '/real/{printf "%s ",$2}'; done +0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +``` + +The signature is in the numbers: of the **171** timing cells in those three +sections, **171 are multiples of 10 ms**; 36 of them are ten ticks or fewer and +11 are five or fewer. `matrix | 20ms | 50ms | 0.40×` was a **two-tick** +measurement, and `tsort | 80ms | 90ms | 0.89×` claimed an 11% win from a +difference of one tick. + +`bench/modules/run.sh` now measures with `Time::HiRes`, refuses to publish a time +for a run that exited non-zero — the old `best()` kept the smallest time of N +without looking at the exit status, so a command that died immediately won its +own series — and **interleaves the runtimes**, one run each in rotation with the +order rotating every round, instead of giving each its own block of wall-clock +time. (What that last one is worth: go-crdt/crdt#131, where the same defect put a +published ratio 14% out, and where contention turned out to fall hardest on the +shortest-running arm rather than on everyone alike.) + +### Results (best of 5, interleaved, ms) + +| Module | rbgo | MRI | MRI+YJIT | JRuby | TruffleRuby | rbgo/MRI | rbgo/YJIT | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| regexp | 4967.7ms | 831.4ms | 831.1ms | 1671.9ms | 325.6ms | 5.98× | 5.98× | +| erb | 589.1ms | 334.7ms | 300.2ms | 1834.8ms | 353.0ms | 1.76× | 1.96× | +| yaml | 192.2ms | 762.3ms | 468.7ms | 2376.1ms | 3336.0ms | **0.25×** | **0.41×** | +| format | 686.6ms | 324.4ms | 304.9ms | 1535.1ms | 462.1ms | 2.12× | 2.25× | +| strscan | 261.8ms | 133.8ms | 117.5ms | 1213.5ms | 175.7ms | 1.96× | 2.23× | +| optparse | 178.5ms | 801.4ms | 616.1ms | 2483.0ms | 1177.5ms | **0.22×** | **0.29×** | +| json | 701.0ms | 316.6ms | 313.2ms | 2001.8ms | 2631.3ms | 2.21× | 2.24× | +| bigdecimal | 342.3ms | 168.3ms | 167.2ms | 1780.6ms | 7101.3ms | 2.03× | 2.05× | +| date | 275.1ms | 415.1ms | 404.2ms | 1432.4ms | 8025.7ms | **0.66×** | **0.68×** | +| uri | 136.1ms | 379.8ms | 308.4ms | 1610.9ms | 2125.8ms | **0.36×** | **0.44×** | +| digest | 456.1ms | 337.7ms | 321.3ms | 1320.3ms | 477.2ms | 1.35× | 1.42× | +| set | 8212.2ms | 214.8ms | 210.3ms | 1288.3ms | 413.5ms | 38.24× | 39.04× | +| prime | 38.7ms | 41.3ms | 53.7ms | 1283.2ms | 187.2ms | **0.94×** | **0.72×** | +| matrix | 48.1ms | 55.4ms | 59.8ms | 1284.3ms | 207.0ms | **0.87×** | **0.80×** | +| complex | 163.6ms | 81.8ms | 71.9ms | 1127.5ms | 91.0ms | 2.00× | 2.28× | +| rational | 169.6ms | 51.2ms | 46.1ms | 1138.9ms | 95.9ms | 3.31× | 3.68× | +| cmath | 84.7ms | 70.4ms | 68.8ms | 1262.5ms | 321.8ms | 1.20× | 1.23× | +| tsort | 125.6ms | 88.4ms | 81.3ms | 1226.3ms | 239.4ms | 1.42× | 1.54× | +| abbrev | 355.7ms | 225.1ms | 188.0ms | 1354.7ms | 395.5ms | 1.58× | 1.89× | +| did-you-mean | 68.1ms | 1545.5ms | 403.2ms | 2813.9ms | 494.3ms | **0.04×** | **0.17×** | +| prettyprint | 144.3ms | 101.8ms | 74.7ms | 1362.8ms | 208.9ms | 1.42× | 1.93× | +| scanf | 179.8ms | 567.2ms | 545.5ms | 1504.7ms | 633.9ms | **0.32×** | **0.33×** | +| unicode-normalize | 167.0ms | 371.8ms | 370.5ms | 1383.7ms | 321.3ms | **0.45×** | **0.45×** | +| cgi | 86.1ms | 65.4ms | 63.7ms | 1144.2ms | 1770.0ms | 1.32× | 1.35× | +| zlib | 63.3ms | 61.5ms | 61.4ms | 1254.4ms | 325.0ms | 1.03× | 1.03× | +| ipaddr | 61.6ms | 114.0ms | 90.2ms | 1436.5ms | 297.3ms | **0.54×** | **0.68×** | +| pathname | 351.9ms | 449.1ms | 430.7ms | 1726.1ms | 652.3ms | **0.78×** | **0.82×** | +| rexml | 52.4ms | 524.5ms | 317.4ms | 2811.1ms | 1333.1ms | **0.10×** | **0.16×** | +| base64 | 421.9ms | 172.0ms | 181.8ms | 1685.1ms | 1100.8ms | 2.45× | 2.32× | +| securerandom | 161.4ms | 329.1ms | 308.9ms | 1253.5ms | 267.6ms | **0.49×** | **0.52×** | +| ostruct | 771.1ms | 1555.2ms | 1538.4ms | 2126.5ms | 4133.2ms | **0.50×** | **0.50×** | +| observer | 252.2ms | 113.2ms | 81.9ms | 1330.5ms | 228.7ms | 2.23× | 3.08× | +| logger | 253.1ms | 98.6ms | 84.3ms | 1413.0ms | 243.1ms | 2.57× | 3.00× | +| find | 1064.7ms | 1118.4ms | 1094.8ms | 3810.7ms | 1473.0ms | **0.95×** | **0.97×** | +| benchmark | 346.1ms | 117.1ms | 76.6ms | 1322.2ms | 159.3ms | 2.95× | 4.52× | + +> `rbgo/MRI < 1` (bold) means the pure-Go library is **faster than MRI's own +> stdlib** on this workload. All 35 modules passed the byte-identical-to-MRI gate +> under every installed runtime; no row was dropped and no run exited non-zero. + +### What moved, and how we know it is not the machine + +**MRI is the control.** Across the 24 rows that appear in both this run and the +June ones, MRI reproduces within a few percent — 220→215 ms on `set`, 1450→1546 +on `did-you-mean`, 570→567 on `scanf`, 330→329 on `securerandom`. The machine and +the workloads are the same, so what moved in the rbgo column is rbgo's. + +**Five of those 24 rows change which side of parity they sit on:** + +| | June | now | | +|---|---|---|---| +| `cmath` | 0.67× | **1.20×** | a win that is a loss | +| `prettyprint` | 0.89× | **1.42×** | a win that is a loss | +| `tsort` | 0.89× | **1.42×** | a win that is a loss | +| `prime` | 1.67× | **0.94×** | a loss that is a win | +| `ipaddr` | 1.91× | **0.54×** | a loss that is a win | + +Three of the eight modules the June text lists as "rbgo beats MRI outright" are +not beating MRI. All three sat on one or two ticks of the old clock. + +**Some rows moved because the code did, and that is the larger part.** `strscan` +10720 → 262 ms and `json` 1520 → 701 ms are #89 and #88 landing the day after +those tables were taken. `zlib` 360 → 63 ms and `ipaddr` 210 → 62 ms moved by +far more than a tick can account for, and `regexp` 1530 → 4968 ms moved that far +the other way. Three months separate the two runs; this section does not try to +attribute each row, only to say which comparisons the clock could not support. + +### `set` is no longer a go-ruby library, and its row moved with it + +`set` reads 8212 ms against MRI's 215 — **38.2×**, where June published 10.09×. +That is not the clock either. #56 bound Set to `go-ruby-set/set` on 2026-06-29; +#330 reimplemented Set in this repository on 2026-08-08 for MRI 4.0 conformance, +with "the bulk of the Ruby-observable Set API […] in the prelude"; #333's +`go mod tidy` dropped the now-unused require the next day. The benchmark +exercises exactly that algebra, so it now measures prelude Ruby where it used to +measure Go. Nothing about the trade was wrong — conformance was the point — but +nothing recorded that it had happened, and a page in another org went on +attributing this number to a library rbgo had stopped using. + +`bench/modules/BACKING.md` now says, beside the scripts, which modules run on an +external `go-ruby-` library and which run on code in this repository. It is +generated, and `TestBenchmarkedModulesSayWhereTheirCodeLives` fails when it and +`go.mod` disagree. Three of the benchmarked modules — `complex`, `rational` and +`set` — are in-tree today. + +The three sections that follow are kept as they were taken. + ## Per-module: rbgo vs MRI / YJIT / JRuby / TruffleRuby (2026-06-29) The benchmarks above are *language* micro-benchmarks (dispatch, allocation, diff --git a/bench/modules/BACKING.md b/bench/modules/BACKING.md new file mode 100644 index 00000000..69245ddf --- /dev/null +++ b/bench/modules/BACKING.md @@ -0,0 +1,45 @@ + +# What each benchmarked module runs on + +`bench/modules/.rb` exercises Ruby's ``. This says whether rbgo serves +it from an external `go-ruby-` library or from code in this repository — +which is what the per-library documentation sites claim, and what stops being +true when a module moves in-tree. + +| Module | Runs on | +| --- | --- | +| abbrev | `github.com/go-ruby-abbrev/abbrev` | +| base64 | `github.com/go-ruby-base64/base64` | +| benchmark | `github.com/go-ruby-benchmark/benchmark` | +| bigdecimal | `github.com/go-ruby-bigdecimal/bigdecimal` | +| cgi | `github.com/go-ruby-cgi/cgi` | +| cmath | `github.com/go-ruby-cmath/cmath` | +| complex | **in-tree** (this repository) | +| date | `github.com/go-ruby-date/date` | +| did-you-mean | `github.com/go-ruby-did-you-mean/did-you-mean` | +| digest | `github.com/go-ruby-digest/digest` | +| erb | `github.com/go-ruby-erb/erb` | +| find | `github.com/go-ruby-find/find` | +| format | `github.com/go-ruby-format/format` | +| ipaddr | `github.com/go-ruby-ipaddr/ipaddr` | +| json | `github.com/go-ruby-json/json` | +| logger | `github.com/go-ruby-logger/logger` | +| matrix | `github.com/go-ruby-matrix/matrix` | +| observer | `github.com/go-ruby-observer/observer` | +| optparse | `github.com/go-ruby-optparse/optparse` | +| ostruct | `github.com/go-ruby-ostruct/ostruct` | +| pathname | `github.com/go-ruby-pathname/pathname` | +| prettyprint | `github.com/go-ruby-prettyprint/prettyprint` | +| prime | `github.com/go-ruby-prime/prime` | +| rational | **in-tree** (this repository) | +| regexp | `github.com/go-ruby-regexp/regexp` | +| rexml | `github.com/go-ruby-rexml/rexml` | +| scanf | `github.com/go-ruby-scanf/scanf` | +| securerandom | `github.com/go-ruby-securerandom/securerandom` | +| set | **in-tree** (this repository) | +| strscan | `github.com/go-ruby-strscan/strscan` | +| tsort | `github.com/go-ruby-tsort/tsort` | +| unicode-normalize | `github.com/go-ruby-unicode-normalize/unicode-normalize` | +| uri | `github.com/go-ruby-uri/uri` | +| yaml | `github.com/go-ruby-yaml/yaml` | +| zlib | `github.com/go-ruby-zlib/zlib` | diff --git a/bench/modules/run.sh b/bench/modules/run.sh index e32f2156..2d588629 100755 --- a/bench/modules/run.sh +++ b/bench/modules/run.sh @@ -4,12 +4,31 @@ # exercises the hot path of one module and prints a deterministic checksum. The # SAME .rb is run under rbgo, MRI, MRI+YJIT, JRuby and TruffleRuby — apples to # apples on the Ruby-visible operation — its output is checked byte-identical to -# MRI first, then wall-clock timed (best-of-N to suppress scheduler noise). A -# runtime not installed shows "n/a"; a runtime whose output diverges from MRI -# shows "diff" (and is not timed). +# MRI first, then wall-clock timed. A runtime not installed shows "n/a"; a +# runtime whose output diverges from MRI shows "diff" (and is not timed). # # Usage: bench/modules/run.sh [runs] (default 5) # Env: RBGO, RUBY, JRUBY, TRUFFLE, N (override every script's iteration count). +# +# Three things this harness gets right, each because it once did not: +# +# The clock is Time::HiRes, not `/usr/bin/time -p`. That prints two decimals of +# a second, so its resolution is 10 ms and anything faster than one tick reads +# zero. It left 116 of 116 published figures sitting on a multiple of 10 ms, +# a third of them ten ticks or fewer, and ratios like 0.40x resting on a +# two-tick measurement. +# +# A run that exits non-zero is an error, not a fast run. Keeping the smallest +# time over N runs without looking at the exit status means a command that dies +# immediately wins the series and gets published. +# +# The runtimes are interleaved, one run each in rotation, rather than each one +# taking its own block of wall-clock time. A machine whose load moves during a +# block charges the movement to whichever runtime was running, and nothing in +# the output says so. Rotating the order each round means none of them always +# goes first. See go-crdt/crdt#131 for what that is worth: the same defect put +# a published ratio 14% out, and contention does not fall on every arm alike — +# it falls hardest on the shortest-running one. set -u RUNS="${1:-5}" RBGO="${RBGO:-/tmp/rbgo}" @@ -18,34 +37,46 @@ JRUBY="${JRUBY:-jruby}" TRUFFLE="${TRUFFLE:-truffleruby}" HERE="$(cd "$(dirname "$0")" && pwd)" -# Module order = the order they were bound into rbgo main. -# Wave-1/2 modules first, then the wave-3 standalone go-ruby- bindings, -# then the wave-4 standalone go-ruby- bindings. -MODULES="regexp erb yaml format strscan optparse json bigdecimal date uri digest \ -set prime matrix complex rational cmath tsort abbrev did-you-mean prettyprint \ -scanf unicode-normalize cgi zlib ipaddr pathname rexml \ -base64 securerandom ostruct observer logger find benchmark" - -# best PROG ARGS... → minimal real milliseconds over $RUNS runs. -best() { - local b=99999999 t - for _ in $(seq "$RUNS"); do - t=$( { /usr/bin/time -p "$@" >/dev/null; } 2>&1 | awk '/real/{print $2}' ) - awk "BEGIN{exit !($t < $b)}" && b=$t - done - awk "BEGIN{printf \"%d\", $b*1000}" +# hires PROG ARGS... → "MILLISECONDS EXITSTATUS", with the child's own output +# discarded inside the fork so that neither stream can reach this one. +hires() { + perl -MTime::HiRes=time -e ' + my $t = time; + my $pid = fork(); + defined $pid or exit 97; + if ($pid == 0) { + open(STDOUT, ">", "/dev/null") or exit 126; + open(STDERR, ">", "/dev/null") or exit 126; + exec(@ARGV); + exit 127; + } + waitpid($pid, 0); + printf "%.3f %d\n", (time - $t) * 1000, $? >> 8; + ' -- "$@" } -# best_ref BIN PROG EXPECTED → "Nms" | "n/a" (absent) | "diff" (output diverges). -best_ref() { - command -v "$1" >/dev/null 2>&1 || { echo "n/a"; return; } - [ "$("$1" "$2" 2>/dev/null)" = "$3" ] || { echo "diff"; return; } - echo "$(best "$1" "$2")" +# The arms, as parallel arrays: name, then the command as a single string that +# word-splits into program and arguments. A runtime that is absent, or whose +# output diverges from MRI, is dropped before any timing. +arm_name=(); arm_cmd=(); arm_best=() + +# add_arm NAME PROG [ARGS...] — the command is stored shell-quoted, so a +# repository path with a space in it stays one argument. +add_arm() { + local name="$1"; shift + arm_name[${#arm_name[@]}]="$name" + arm_cmd[${#arm_cmd[@]}]="$(printf '%q ' "$@")" + arm_best[${#arm_best[@]}]="" } printf '| Module | rbgo | MRI | MRI+YJIT | JRuby | TruffleRuby | rbgo/MRI | rbgo/YJIT |\n' printf '| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n' +MODULES="${MODULES:-regexp erb yaml format strscan optparse json bigdecimal date uri digest \ +set prime matrix complex rational cmath tsort abbrev did-you-mean prettyprint \ +scanf unicode-normalize cgi zlib ipaddr pathname rexml \ +base64 securerandom ostruct observer logger find benchmark}" + for m in $MODULES; do f="$HERE/$m.rb" [ -f "$f" ] || { printf '| %s | (no script) | | | | | | |\n' "$m"; continue; } @@ -57,17 +88,76 @@ for m in $MODULES; do continue fi - rb=$(best "$RBGO" run "$f") - mr=$(best "$RUBY" "$f") - yj=$(best "$RUBY" --yjit "$f") - jr=$(best_ref "$JRUBY" "$f" "$mo") - tr=$(best_ref "$TRUFFLE" "$f" "$mo") + arm_name=(); arm_cmd=(); arm_best=() + add_arm rbgo "$RBGO" run "$f" + add_arm mri "$RUBY" "$f" + add_arm yjit "$RUBY" --yjit "$f" + for pair in "jruby $JRUBY" "truffle $TRUFFLE"; do + set -- $pair + label="$1"; bin="$2" + command -v "$bin" >/dev/null 2>&1 || { eval "${label}_state=n/a"; continue; } + [ "$("$bin" "$f" 2>/dev/null)" = "$mo" ] || { eval "${label}_state=diff"; continue; } + eval "${label}_state=timed" + add_arm "$label" "$bin" "$f" + done + + # One round asks every arm for one run, rotating who goes first, and the best + # of those runs is kept — the minimum, because the environment can only add + # delay to a program, never take any away (Chen & Revels, Robust benchmarking + # in noisy environments, IEEE HPEC 2016). + n=${#arm_name[@]} + r=0 + while [ "$r" -lt "$RUNS" ]; do + i=0 + while [ "$i" -lt "$n" ]; do + k=$(( (i + r) % n )) + if [ "${arm_best[$k]}" = "err" ]; then i=$(( i + 1 )); continue; fi + out=$(eval hires "${arm_cmd[$k]}") + ms=${out% *}; rc=${out#* } + if [ "$rc" != "0" ]; then + # A run that failed is not a fast run. The old harness kept the smallest + # time of N without looking at the exit status, so a command that died + # immediately won its own series and got published. + echo "harness: ${arm_name[$k]} exited $rc on $m — not timing it" >&2 + arm_best[$k]="err" + i=$(( i + 1 )) + continue + fi + b=${arm_best[$k]} + if [ -z "$b" ] || awk "BEGIN{exit !($ms < $b)}"; then arm_best[$k]=$ms; fi + i=$(( i + 1 )) + done + r=$(( r + 1 )) + done + + get() { # get NAME → best milliseconds, or empty + local j=0 + while [ "$j" -lt "${#arm_name[@]}" ]; do + [ "${arm_name[$j]}" = "$1" ] && { echo "${arm_best[$j]}"; return; } + j=$(( j + 1 )) + done + } + rb=$(get rbgo); mr=$(get mri); yj=$(get yjit) + jr=$(get jruby); tr=$(get truffle) - rm_ratio=$(awk "BEGIN{printf \"%.2f\", $rb/$mr}") - ry_ratio=$(awk "BEGIN{printf \"%.2f\", $rb/$yj}") - jr_s=$([ "$jr" = "n/a" ] || [ "$jr" = "diff" ] && echo "$jr" || echo "${jr}ms") - tr_s=$([ "$tr" = "n/a" ] || [ "$tr" = "diff" ] && echo "$tr" || echo "${tr}ms") + fmt() { + case "$1" in + "") echo "$2" ;; + err) echo "err" ;; + *) awk "BEGIN{printf \"%.1fms\", $1}" ;; + esac + } + ratio() { # ratio NUMERATOR DENOMINATOR → "n.nn" or "—" + case "$1:$2" in + err:*|*:err|:*|*:) echo "—"; return ;; + esac + awk "BEGIN{printf \"%.2f\", $1/$2}" + } + rm_ratio=$(ratio "$rb" "$mr") + ry_ratio=$(ratio "$rb" "$yj") - printf '| %s | %sms | %sms | %sms | %s | %s | %s× | %s× |\n' \ - "$m" "$rb" "$mr" "$yj" "$jr_s" "$tr_s" "$rm_ratio" "$ry_ratio" + printf '| %s | %s | %s | %s | %s | %s | %s× | %s× |\n' \ + "$m" "$(fmt "$rb" '')" "$(fmt "$mr" '')" "$(fmt "$yj" '')" \ + "$(fmt "$jr" "${jruby_state:-n/a}")" "$(fmt "$tr" "${truffle_state:-n/a}")" \ + "$rm_ratio" "$ry_ratio" done diff --git a/bench_backing_test.go b/bench_backing_test.go new file mode 100644 index 00000000..381935e8 --- /dev/null +++ b/bench_backing_test.go @@ -0,0 +1,154 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +package ruby + +import ( + "flag" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +var updateBacking = flag.Bool("update-backing", false, + "rewrite bench/modules/BACKING.md from what go.mod currently requires") + +// Every module under bench/modules has a documentation site of its own, and each +// of those sites opens by saying that rbgo *binds* a pure-Go library for it: +// +// go-ruby-set/set is the pure-Go library that rbgo binds for Ruby's set. +// rbgo's number reflects this pure-Go library doing the work. +// +// That sentence is a claim about this repository, and it stops being true the +// moment a module moves in-tree. It did: #330 reimplemented Set here for MRI 4.0 +// conformance, #333's go mod tidy dropped the now-unused require the next day, +// and the page in the other org went on attributing rbgo's number to a library +// rbgo no longer used — for seven weeks, and with a figure that had moved by +// 3.7x underneath it. +// +// Nothing about that trade was wrong. What was missing was anything that would +// say it had happened. This test is that: it writes down, beside the benchmark +// scripts, which of them run on an external go-ruby library and which run on +// code in this repository, and fails when the file and go.mod disagree. +// +// Regenerate with: go test -run TestBenchmarkedModulesSayWhereTheirCodeLives -update-backing +func TestBenchmarkedModulesSayWhereTheirCodeLives(t *testing.T) { + scripts, err := filepath.Glob(filepath.Join("bench", "modules", "*.rb")) + if err != nil { + t.Fatalf("listing the benchmark scripts: %v", err) + } + if len(scripts) == 0 { + t.Fatal("no benchmark scripts found; this test would pass by measuring nothing") + } + + gomod, err := os.ReadFile("go.mod") + if err != nil { + t.Fatalf("reading go.mod: %v", err) + } + required := requiredModules(string(gomod)) + // A control: the file must name modules this repository really requires, or + // the parse is wrong and every row below would read "in-tree" for the wrong + // reason. + if len(required) < 50 { + t.Fatalf("go.mod parsed to %d requirements, which cannot be right", len(required)) + } + + names := make([]string, 0, len(scripts)) + for _, s := range scripts { + names = append(names, strings.TrimSuffix(filepath.Base(s), ".rb")) + } + sort.Strings(names) + + var b strings.Builder + b.WriteString("\n") + b.WriteString("# What each benchmarked module runs on\n\n") + b.WriteString("`bench/modules/.rb` exercises Ruby's ``. This says whether rbgo serves\n") + b.WriteString("it from an external `go-ruby-` library or from code in this repository —\n") + b.WriteString("which is what the per-library documentation sites claim, and what stops being\n") + b.WriteString("true when a module moves in-tree.\n\n") + b.WriteString("| Module | Runs on |\n| --- | --- |\n") + for _, m := range names { + path := fmt.Sprintf("github.com/go-ruby-%s/%s", m, m) + if required[path] { + fmt.Fprintf(&b, "| %s | `%s` |\n", m, path) + } else { + fmt.Fprintf(&b, "| %s | **in-tree** (this repository) |\n", m) + } + } + want := b.String() + + out := filepath.Join("bench", "modules", "BACKING.md") + if *updateBacking { + if err := os.WriteFile(out, []byte(want), 0o644); err != nil { + t.Fatalf("writing %s: %v", out, err) + } + t.Logf("wrote %s", out) + return + } + + got, err := os.ReadFile(out) + if err != nil { + t.Fatalf("reading %s (regenerate with -update-backing): %v", out, err) + } + if string(got) != want { + t.Errorf("%s no longer matches go.mod.\n\nA module has moved between an external\n"+ + "go-ruby library and this repository. Regenerate with:\n\n"+ + " go test -run TestBenchmarkedModulesSayWhereTheirCodeLives -update-backing\n\n"+ + "and then say so wherever that module's performance is published: the page\n"+ + "in its own org opens by naming the library rbgo binds for it.\n\n%s", + out, firstDifference(string(got), want)) + } +} + +// requiredModules pulls the module paths out of go.mod's require blocks. It is +// deliberately a parse of the text rather than a call to `go list`: the test has +// to answer the same question a reader of go.mod would. +func requiredModules(gomod string) map[string]bool { + out := map[string]bool{} + inBlock := false + for _, line := range strings.Split(gomod, "\n") { + s := strings.TrimSpace(line) + if i := strings.Index(s, "//"); i >= 0 { + s = strings.TrimSpace(s[:i]) + } + switch { + case s == "require (": + inBlock = true + continue + case inBlock && s == ")": + inBlock = false + continue + case strings.HasPrefix(s, "require "): + s = strings.TrimSpace(strings.TrimPrefix(s, "require ")) + case !inBlock: + continue + } + if fields := strings.Fields(s); len(fields) >= 2 { + out[fields[0]] = true + } + } + return out +} + +// firstDifference names the first line that differs, because a diff of a +// generated table is unreadable and the answer is always one row. +func firstDifference(got, want string) string { + g, w := strings.Split(got, "\n"), strings.Split(want, "\n") + for i := 0; i < len(g) || i < len(w); i++ { + gl, wl := "", "" + if i < len(g) { + gl = g[i] + } + if i < len(w) { + wl = w[i] + } + if gl != wl { + return fmt.Sprintf("first difference at line %d:\n file: %q\n go.mod: %q", i+1, gl, wl) + } + } + return "(the files differ only in trailing bytes)" +} From 166d304172c8f4bbc78e03cd64af6617d473fcd4 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Wed, 30 Sep 2026 11:04:49 +0200 Subject: [PATCH 2/2] The guard failed on Windows for a reason that was not drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit git hands BACKING.md to a Windows checkout with CRLF endings, and the test compared bytes, so the first CI run that saw it reported the file as no longer matching go.mod — over carriage returns. A guard about which module runs where must not depend on how the working tree was written. Compared with carriage returns dropped from both sides. Checked on a CRLF copy that the correct table passes and that flipping `set` back to github.com/go-ruby-set/set still fails and still names the line: the point was to stop it failing for the wrong reason, not to stop it failing. Co-Authored-By: Claude Opus 5 --- bench_backing_test.go | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/bench_backing_test.go b/bench_backing_test.go index 381935e8..9733a4c2 100644 --- a/bench_backing_test.go +++ b/bench_backing_test.go @@ -94,16 +94,24 @@ func TestBenchmarkedModulesSayWhereTheirCodeLives(t *testing.T) { if err != nil { t.Fatalf("reading %s (regenerate with -update-backing): %v", out, err) } - if string(got) != want { + // Compared without carriage returns: git hands this file to a Windows + // checkout with CRLF endings, and a guard about which module runs where must + // not fail over how the working tree was written. It did, on the first run of + // CI that saw it. + if unixEndings(string(got)) != unixEndings(want) { t.Errorf("%s no longer matches go.mod.\n\nA module has moved between an external\n"+ "go-ruby library and this repository. Regenerate with:\n\n"+ " go test -run TestBenchmarkedModulesSayWhereTheirCodeLives -update-backing\n\n"+ "and then say so wherever that module's performance is published: the page\n"+ "in its own org opens by naming the library rbgo binds for it.\n\n%s", - out, firstDifference(string(got), want)) + out, firstDifference(unixEndings(string(got)), unixEndings(want))) } } +// unixEndings drops carriage returns so that the comparison is about the table +// and not about the checkout. +func unixEndings(s string) string { return strings.ReplaceAll(s, "\r", "") } + // requiredModules pulls the module paths out of go.mod's require blocks. It is // deliberately a parse of the text rather than a call to `go list`: the test has // to answer the same question a reader of go.mod would.