diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 2020ed7d..9d172d7f 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -100,6 +100,123 @@ Native). Reproduce: `AOT=1 RUNS=8 JRUBY=jruby TRUFFLE= bash bench/run.sh 8 lowering of integer-bound methods (`mandelbrot`'s float kernel is not yet AOT-lowered). +## Per-module, re-taken with a clock that can see the rows (2026-09-30) + +The three per-module sections below were measured with `/usr/bin/time -p`, which +prints **two decimals of a second**. Its resolution is 10 ms, and anything faster +than one tick reads zero: + +``` +$ for i in $(seq 10); do { /usr/bin/time -p /usr/bin/true >/dev/null; } 2>&1 | awk '/real/{printf "%s ",$2}'; done +0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +``` + +The signature is in the numbers: of the **171** timing cells in those three +sections, **171 are multiples of 10 ms**; 36 of them are ten ticks or fewer and +11 are five or fewer. `matrix | 20ms | 50ms | 0.40×` was a **two-tick** +measurement, and `tsort | 80ms | 90ms | 0.89×` claimed an 11% win from a +difference of one tick. + +`bench/modules/run.sh` now measures with `Time::HiRes`, refuses to publish a time +for a run that exited non-zero — the old `best()` kept the smallest time of N +without looking at the exit status, so a command that died immediately won its +own series — and **interleaves the runtimes**, one run each in rotation with the +order rotating every round, instead of giving each its own block of wall-clock +time. (What that last one is worth: go-crdt/crdt#131, where the same defect put a +published ratio 14% out, and where contention turned out to fall hardest on the +shortest-running arm rather than on everyone alike.) + +### Results (best of 5, interleaved, ms) + +| Module | rbgo | MRI | MRI+YJIT | JRuby | TruffleRuby | rbgo/MRI | rbgo/YJIT | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| regexp | 4967.7ms | 831.4ms | 831.1ms | 1671.9ms | 325.6ms | 5.98× | 5.98× | +| erb | 589.1ms | 334.7ms | 300.2ms | 1834.8ms | 353.0ms | 1.76× | 1.96× | +| yaml | 192.2ms | 762.3ms | 468.7ms | 2376.1ms | 3336.0ms | **0.25×** | **0.41×** | +| format | 686.6ms | 324.4ms | 304.9ms | 1535.1ms | 462.1ms | 2.12× | 2.25× | +| strscan | 261.8ms | 133.8ms | 117.5ms | 1213.5ms | 175.7ms | 1.96× | 2.23× | +| optparse | 178.5ms | 801.4ms | 616.1ms | 2483.0ms | 1177.5ms | **0.22×** | **0.29×** | +| json | 701.0ms | 316.6ms | 313.2ms | 2001.8ms | 2631.3ms | 2.21× | 2.24× | +| bigdecimal | 342.3ms | 168.3ms | 167.2ms | 1780.6ms | 7101.3ms | 2.03× | 2.05× | +| date | 275.1ms | 415.1ms | 404.2ms | 1432.4ms | 8025.7ms | **0.66×** | **0.68×** | +| uri | 136.1ms | 379.8ms | 308.4ms | 1610.9ms | 2125.8ms | **0.36×** | **0.44×** | +| digest | 456.1ms | 337.7ms | 321.3ms | 1320.3ms | 477.2ms | 1.35× | 1.42× | +| set | 8212.2ms | 214.8ms | 210.3ms | 1288.3ms | 413.5ms | 38.24× | 39.04× | +| prime | 38.7ms | 41.3ms | 53.7ms | 1283.2ms | 187.2ms | **0.94×** | **0.72×** | +| matrix | 48.1ms | 55.4ms | 59.8ms | 1284.3ms | 207.0ms | **0.87×** | **0.80×** | +| complex | 163.6ms | 81.8ms | 71.9ms | 1127.5ms | 91.0ms | 2.00× | 2.28× | +| rational | 169.6ms | 51.2ms | 46.1ms | 1138.9ms | 95.9ms | 3.31× | 3.68× | +| cmath | 84.7ms | 70.4ms | 68.8ms | 1262.5ms | 321.8ms | 1.20× | 1.23× | +| tsort | 125.6ms | 88.4ms | 81.3ms | 1226.3ms | 239.4ms | 1.42× | 1.54× | +| abbrev | 355.7ms | 225.1ms | 188.0ms | 1354.7ms | 395.5ms | 1.58× | 1.89× | +| did-you-mean | 68.1ms | 1545.5ms | 403.2ms | 2813.9ms | 494.3ms | **0.04×** | **0.17×** | +| prettyprint | 144.3ms | 101.8ms | 74.7ms | 1362.8ms | 208.9ms | 1.42× | 1.93× | +| scanf | 179.8ms | 567.2ms | 545.5ms | 1504.7ms | 633.9ms | **0.32×** | **0.33×** | +| unicode-normalize | 167.0ms | 371.8ms | 370.5ms | 1383.7ms | 321.3ms | **0.45×** | **0.45×** | +| cgi | 86.1ms | 65.4ms | 63.7ms | 1144.2ms | 1770.0ms | 1.32× | 1.35× | +| zlib | 63.3ms | 61.5ms | 61.4ms | 1254.4ms | 325.0ms | 1.03× | 1.03× | +| ipaddr | 61.6ms | 114.0ms | 90.2ms | 1436.5ms | 297.3ms | **0.54×** | **0.68×** | +| pathname | 351.9ms | 449.1ms | 430.7ms | 1726.1ms | 652.3ms | **0.78×** | **0.82×** | +| rexml | 52.4ms | 524.5ms | 317.4ms | 2811.1ms | 1333.1ms | **0.10×** | **0.16×** | +| base64 | 421.9ms | 172.0ms | 181.8ms | 1685.1ms | 1100.8ms | 2.45× | 2.32× | +| securerandom | 161.4ms | 329.1ms | 308.9ms | 1253.5ms | 267.6ms | **0.49×** | **0.52×** | +| ostruct | 771.1ms | 1555.2ms | 1538.4ms | 2126.5ms | 4133.2ms | **0.50×** | **0.50×** | +| observer | 252.2ms | 113.2ms | 81.9ms | 1330.5ms | 228.7ms | 2.23× | 3.08× | +| logger | 253.1ms | 98.6ms | 84.3ms | 1413.0ms | 243.1ms | 2.57× | 3.00× | +| find | 1064.7ms | 1118.4ms | 1094.8ms | 3810.7ms | 1473.0ms | **0.95×** | **0.97×** | +| benchmark | 346.1ms | 117.1ms | 76.6ms | 1322.2ms | 159.3ms | 2.95× | 4.52× | + +> `rbgo/MRI < 1` (bold) means the pure-Go library is **faster than MRI's own +> stdlib** on this workload. All 35 modules passed the byte-identical-to-MRI gate +> under every installed runtime; no row was dropped and no run exited non-zero. + +### What moved, and how we know it is not the machine + +**MRI is the control.** Across the 24 rows that appear in both this run and the +June ones, MRI reproduces within a few percent — 220→215 ms on `set`, 1450→1546 +on `did-you-mean`, 570→567 on `scanf`, 330→329 on `securerandom`. The machine and +the workloads are the same, so what moved in the rbgo column is rbgo's. + +**Five of those 24 rows change which side of parity they sit on:** + +| | June | now | | +|---|---|---|---| +| `cmath` | 0.67× | **1.20×** | a win that is a loss | +| `prettyprint` | 0.89× | **1.42×** | a win that is a loss | +| `tsort` | 0.89× | **1.42×** | a win that is a loss | +| `prime` | 1.67× | **0.94×** | a loss that is a win | +| `ipaddr` | 1.91× | **0.54×** | a loss that is a win | + +Three of the eight modules the June text lists as "rbgo beats MRI outright" are +not beating MRI. All three sat on one or two ticks of the old clock. + +**Some rows moved because the code did, and that is the larger part.** `strscan` +10720 → 262 ms and `json` 1520 → 701 ms are #89 and #88 landing the day after +those tables were taken. `zlib` 360 → 63 ms and `ipaddr` 210 → 62 ms moved by +far more than a tick can account for, and `regexp` 1530 → 4968 ms moved that far +the other way. Three months separate the two runs; this section does not try to +attribute each row, only to say which comparisons the clock could not support. + +### `set` is no longer a go-ruby library, and its row moved with it + +`set` reads 8212 ms against MRI's 215 — **38.2×**, where June published 10.09×. +That is not the clock either. #56 bound Set to `go-ruby-set/set` on 2026-06-29; +#330 reimplemented Set in this repository on 2026-08-08 for MRI 4.0 conformance, +with "the bulk of the Ruby-observable Set API […] in the prelude"; #333's +`go mod tidy` dropped the now-unused require the next day. The benchmark +exercises exactly that algebra, so it now measures prelude Ruby where it used to +measure Go. Nothing about the trade was wrong — conformance was the point — but +nothing recorded that it had happened, and a page in another org went on +attributing this number to a library rbgo had stopped using. + +`bench/modules/BACKING.md` now says, beside the scripts, which modules run on an +external `go-ruby-` library and which run on code in this repository. It is +generated, and `TestBenchmarkedModulesSayWhereTheirCodeLives` fails when it and +`go.mod` disagree. Three of the benchmarked modules — `complex`, `rational` and +`set` — are in-tree today. + +The three sections that follow are kept as they were taken. + ## Per-module: rbgo vs MRI / YJIT / JRuby / TruffleRuby (2026-06-29) The benchmarks above are *language* micro-benchmarks (dispatch, allocation, diff --git a/bench/modules/BACKING.md b/bench/modules/BACKING.md new file mode 100644 index 00000000..69245ddf --- /dev/null +++ b/bench/modules/BACKING.md @@ -0,0 +1,45 @@ + +# What each benchmarked module runs on + +`bench/modules/.rb` exercises Ruby's ``. This says whether rbgo serves +it from an external `go-ruby-` library or from code in this repository — +which is what the per-library documentation sites claim, and what stops being +true when a module moves in-tree. + +| Module | Runs on | +| --- | --- | +| abbrev | `github.com/go-ruby-abbrev/abbrev` | +| base64 | `github.com/go-ruby-base64/base64` | +| benchmark | `github.com/go-ruby-benchmark/benchmark` | +| bigdecimal | `github.com/go-ruby-bigdecimal/bigdecimal` | +| cgi | `github.com/go-ruby-cgi/cgi` | +| cmath | `github.com/go-ruby-cmath/cmath` | +| complex | **in-tree** (this repository) | +| date | `github.com/go-ruby-date/date` | +| did-you-mean | `github.com/go-ruby-did-you-mean/did-you-mean` | +| digest | `github.com/go-ruby-digest/digest` | +| erb | `github.com/go-ruby-erb/erb` | +| find | `github.com/go-ruby-find/find` | +| format | `github.com/go-ruby-format/format` | +| ipaddr | `github.com/go-ruby-ipaddr/ipaddr` | +| json | `github.com/go-ruby-json/json` | +| logger | `github.com/go-ruby-logger/logger` | +| matrix | `github.com/go-ruby-matrix/matrix` | +| observer | `github.com/go-ruby-observer/observer` | +| optparse | `github.com/go-ruby-optparse/optparse` | +| ostruct | `github.com/go-ruby-ostruct/ostruct` | +| pathname | `github.com/go-ruby-pathname/pathname` | +| prettyprint | `github.com/go-ruby-prettyprint/prettyprint` | +| prime | `github.com/go-ruby-prime/prime` | +| rational | **in-tree** (this repository) | +| regexp | `github.com/go-ruby-regexp/regexp` | +| rexml | `github.com/go-ruby-rexml/rexml` | +| scanf | `github.com/go-ruby-scanf/scanf` | +| securerandom | `github.com/go-ruby-securerandom/securerandom` | +| set | **in-tree** (this repository) | +| strscan | `github.com/go-ruby-strscan/strscan` | +| tsort | `github.com/go-ruby-tsort/tsort` | +| unicode-normalize | `github.com/go-ruby-unicode-normalize/unicode-normalize` | +| uri | `github.com/go-ruby-uri/uri` | +| yaml | `github.com/go-ruby-yaml/yaml` | +| zlib | `github.com/go-ruby-zlib/zlib` | diff --git a/bench/modules/run.sh b/bench/modules/run.sh index e32f2156..2d588629 100755 --- a/bench/modules/run.sh +++ b/bench/modules/run.sh @@ -4,12 +4,31 @@ # exercises the hot path of one module and prints a deterministic checksum. The # SAME .rb is run under rbgo, MRI, MRI+YJIT, JRuby and TruffleRuby — apples to # apples on the Ruby-visible operation — its output is checked byte-identical to -# MRI first, then wall-clock timed (best-of-N to suppress scheduler noise). A -# runtime not installed shows "n/a"; a runtime whose output diverges from MRI -# shows "diff" (and is not timed). +# MRI first, then wall-clock timed. A runtime not installed shows "n/a"; a +# runtime whose output diverges from MRI shows "diff" (and is not timed). # # Usage: bench/modules/run.sh [runs] (default 5) # Env: RBGO, RUBY, JRUBY, TRUFFLE, N (override every script's iteration count). +# +# Three things this harness gets right, each because it once did not: +# +# The clock is Time::HiRes, not `/usr/bin/time -p`. That prints two decimals of +# a second, so its resolution is 10 ms and anything faster than one tick reads +# zero. It left 116 of 116 published figures sitting on a multiple of 10 ms, +# a third of them ten ticks or fewer, and ratios like 0.40x resting on a +# two-tick measurement. +# +# A run that exits non-zero is an error, not a fast run. Keeping the smallest +# time over N runs without looking at the exit status means a command that dies +# immediately wins the series and gets published. +# +# The runtimes are interleaved, one run each in rotation, rather than each one +# taking its own block of wall-clock time. A machine whose load moves during a +# block charges the movement to whichever runtime was running, and nothing in +# the output says so. Rotating the order each round means none of them always +# goes first. See go-crdt/crdt#131 for what that is worth: the same defect put +# a published ratio 14% out, and contention does not fall on every arm alike — +# it falls hardest on the shortest-running one. set -u RUNS="${1:-5}" RBGO="${RBGO:-/tmp/rbgo}" @@ -18,34 +37,46 @@ JRUBY="${JRUBY:-jruby}" TRUFFLE="${TRUFFLE:-truffleruby}" HERE="$(cd "$(dirname "$0")" && pwd)" -# Module order = the order they were bound into rbgo main. -# Wave-1/2 modules first, then the wave-3 standalone go-ruby- bindings, -# then the wave-4 standalone go-ruby- bindings. -MODULES="regexp erb yaml format strscan optparse json bigdecimal date uri digest \ -set prime matrix complex rational cmath tsort abbrev did-you-mean prettyprint \ -scanf unicode-normalize cgi zlib ipaddr pathname rexml \ -base64 securerandom ostruct observer logger find benchmark" - -# best PROG ARGS... → minimal real milliseconds over $RUNS runs. -best() { - local b=99999999 t - for _ in $(seq "$RUNS"); do - t=$( { /usr/bin/time -p "$@" >/dev/null; } 2>&1 | awk '/real/{print $2}' ) - awk "BEGIN{exit !($t < $b)}" && b=$t - done - awk "BEGIN{printf \"%d\", $b*1000}" +# hires PROG ARGS... → "MILLISECONDS EXITSTATUS", with the child's own output +# discarded inside the fork so that neither stream can reach this one. +hires() { + perl -MTime::HiRes=time -e ' + my $t = time; + my $pid = fork(); + defined $pid or exit 97; + if ($pid == 0) { + open(STDOUT, ">", "/dev/null") or exit 126; + open(STDERR, ">", "/dev/null") or exit 126; + exec(@ARGV); + exit 127; + } + waitpid($pid, 0); + printf "%.3f %d\n", (time - $t) * 1000, $? >> 8; + ' -- "$@" } -# best_ref BIN PROG EXPECTED → "Nms" | "n/a" (absent) | "diff" (output diverges). -best_ref() { - command -v "$1" >/dev/null 2>&1 || { echo "n/a"; return; } - [ "$("$1" "$2" 2>/dev/null)" = "$3" ] || { echo "diff"; return; } - echo "$(best "$1" "$2")" +# The arms, as parallel arrays: name, then the command as a single string that +# word-splits into program and arguments. A runtime that is absent, or whose +# output diverges from MRI, is dropped before any timing. +arm_name=(); arm_cmd=(); arm_best=() + +# add_arm NAME PROG [ARGS...] — the command is stored shell-quoted, so a +# repository path with a space in it stays one argument. +add_arm() { + local name="$1"; shift + arm_name[${#arm_name[@]}]="$name" + arm_cmd[${#arm_cmd[@]}]="$(printf '%q ' "$@")" + arm_best[${#arm_best[@]}]="" } printf '| Module | rbgo | MRI | MRI+YJIT | JRuby | TruffleRuby | rbgo/MRI | rbgo/YJIT |\n' printf '| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n' +MODULES="${MODULES:-regexp erb yaml format strscan optparse json bigdecimal date uri digest \ +set prime matrix complex rational cmath tsort abbrev did-you-mean prettyprint \ +scanf unicode-normalize cgi zlib ipaddr pathname rexml \ +base64 securerandom ostruct observer logger find benchmark}" + for m in $MODULES; do f="$HERE/$m.rb" [ -f "$f" ] || { printf '| %s | (no script) | | | | | | |\n' "$m"; continue; } @@ -57,17 +88,76 @@ for m in $MODULES; do continue fi - rb=$(best "$RBGO" run "$f") - mr=$(best "$RUBY" "$f") - yj=$(best "$RUBY" --yjit "$f") - jr=$(best_ref "$JRUBY" "$f" "$mo") - tr=$(best_ref "$TRUFFLE" "$f" "$mo") + arm_name=(); arm_cmd=(); arm_best=() + add_arm rbgo "$RBGO" run "$f" + add_arm mri "$RUBY" "$f" + add_arm yjit "$RUBY" --yjit "$f" + for pair in "jruby $JRUBY" "truffle $TRUFFLE"; do + set -- $pair + label="$1"; bin="$2" + command -v "$bin" >/dev/null 2>&1 || { eval "${label}_state=n/a"; continue; } + [ "$("$bin" "$f" 2>/dev/null)" = "$mo" ] || { eval "${label}_state=diff"; continue; } + eval "${label}_state=timed" + add_arm "$label" "$bin" "$f" + done + + # One round asks every arm for one run, rotating who goes first, and the best + # of those runs is kept — the minimum, because the environment can only add + # delay to a program, never take any away (Chen & Revels, Robust benchmarking + # in noisy environments, IEEE HPEC 2016). + n=${#arm_name[@]} + r=0 + while [ "$r" -lt "$RUNS" ]; do + i=0 + while [ "$i" -lt "$n" ]; do + k=$(( (i + r) % n )) + if [ "${arm_best[$k]}" = "err" ]; then i=$(( i + 1 )); continue; fi + out=$(eval hires "${arm_cmd[$k]}") + ms=${out% *}; rc=${out#* } + if [ "$rc" != "0" ]; then + # A run that failed is not a fast run. The old harness kept the smallest + # time of N without looking at the exit status, so a command that died + # immediately won its own series and got published. + echo "harness: ${arm_name[$k]} exited $rc on $m — not timing it" >&2 + arm_best[$k]="err" + i=$(( i + 1 )) + continue + fi + b=${arm_best[$k]} + if [ -z "$b" ] || awk "BEGIN{exit !($ms < $b)}"; then arm_best[$k]=$ms; fi + i=$(( i + 1 )) + done + r=$(( r + 1 )) + done + + get() { # get NAME → best milliseconds, or empty + local j=0 + while [ "$j" -lt "${#arm_name[@]}" ]; do + [ "${arm_name[$j]}" = "$1" ] && { echo "${arm_best[$j]}"; return; } + j=$(( j + 1 )) + done + } + rb=$(get rbgo); mr=$(get mri); yj=$(get yjit) + jr=$(get jruby); tr=$(get truffle) - rm_ratio=$(awk "BEGIN{printf \"%.2f\", $rb/$mr}") - ry_ratio=$(awk "BEGIN{printf \"%.2f\", $rb/$yj}") - jr_s=$([ "$jr" = "n/a" ] || [ "$jr" = "diff" ] && echo "$jr" || echo "${jr}ms") - tr_s=$([ "$tr" = "n/a" ] || [ "$tr" = "diff" ] && echo "$tr" || echo "${tr}ms") + fmt() { + case "$1" in + "") echo "$2" ;; + err) echo "err" ;; + *) awk "BEGIN{printf \"%.1fms\", $1}" ;; + esac + } + ratio() { # ratio NUMERATOR DENOMINATOR → "n.nn" or "—" + case "$1:$2" in + err:*|*:err|:*|*:) echo "—"; return ;; + esac + awk "BEGIN{printf \"%.2f\", $1/$2}" + } + rm_ratio=$(ratio "$rb" "$mr") + ry_ratio=$(ratio "$rb" "$yj") - printf '| %s | %sms | %sms | %sms | %s | %s | %s× | %s× |\n' \ - "$m" "$rb" "$mr" "$yj" "$jr_s" "$tr_s" "$rm_ratio" "$ry_ratio" + printf '| %s | %s | %s | %s | %s | %s | %s× | %s× |\n' \ + "$m" "$(fmt "$rb" '')" "$(fmt "$mr" '')" "$(fmt "$yj" '')" \ + "$(fmt "$jr" "${jruby_state:-n/a}")" "$(fmt "$tr" "${truffle_state:-n/a}")" \ + "$rm_ratio" "$ry_ratio" done diff --git a/bench_backing_test.go b/bench_backing_test.go new file mode 100644 index 00000000..9733a4c2 --- /dev/null +++ b/bench_backing_test.go @@ -0,0 +1,162 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +package ruby + +import ( + "flag" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +var updateBacking = flag.Bool("update-backing", false, + "rewrite bench/modules/BACKING.md from what go.mod currently requires") + +// Every module under bench/modules has a documentation site of its own, and each +// of those sites opens by saying that rbgo *binds* a pure-Go library for it: +// +// go-ruby-set/set is the pure-Go library that rbgo binds for Ruby's set. +// rbgo's number reflects this pure-Go library doing the work. +// +// That sentence is a claim about this repository, and it stops being true the +// moment a module moves in-tree. It did: #330 reimplemented Set here for MRI 4.0 +// conformance, #333's go mod tidy dropped the now-unused require the next day, +// and the page in the other org went on attributing rbgo's number to a library +// rbgo no longer used — for seven weeks, and with a figure that had moved by +// 3.7x underneath it. +// +// Nothing about that trade was wrong. What was missing was anything that would +// say it had happened. This test is that: it writes down, beside the benchmark +// scripts, which of them run on an external go-ruby library and which run on +// code in this repository, and fails when the file and go.mod disagree. +// +// Regenerate with: go test -run TestBenchmarkedModulesSayWhereTheirCodeLives -update-backing +func TestBenchmarkedModulesSayWhereTheirCodeLives(t *testing.T) { + scripts, err := filepath.Glob(filepath.Join("bench", "modules", "*.rb")) + if err != nil { + t.Fatalf("listing the benchmark scripts: %v", err) + } + if len(scripts) == 0 { + t.Fatal("no benchmark scripts found; this test would pass by measuring nothing") + } + + gomod, err := os.ReadFile("go.mod") + if err != nil { + t.Fatalf("reading go.mod: %v", err) + } + required := requiredModules(string(gomod)) + // A control: the file must name modules this repository really requires, or + // the parse is wrong and every row below would read "in-tree" for the wrong + // reason. + if len(required) < 50 { + t.Fatalf("go.mod parsed to %d requirements, which cannot be right", len(required)) + } + + names := make([]string, 0, len(scripts)) + for _, s := range scripts { + names = append(names, strings.TrimSuffix(filepath.Base(s), ".rb")) + } + sort.Strings(names) + + var b strings.Builder + b.WriteString("\n") + b.WriteString("# What each benchmarked module runs on\n\n") + b.WriteString("`bench/modules/.rb` exercises Ruby's ``. This says whether rbgo serves\n") + b.WriteString("it from an external `go-ruby-` library or from code in this repository —\n") + b.WriteString("which is what the per-library documentation sites claim, and what stops being\n") + b.WriteString("true when a module moves in-tree.\n\n") + b.WriteString("| Module | Runs on |\n| --- | --- |\n") + for _, m := range names { + path := fmt.Sprintf("github.com/go-ruby-%s/%s", m, m) + if required[path] { + fmt.Fprintf(&b, "| %s | `%s` |\n", m, path) + } else { + fmt.Fprintf(&b, "| %s | **in-tree** (this repository) |\n", m) + } + } + want := b.String() + + out := filepath.Join("bench", "modules", "BACKING.md") + if *updateBacking { + if err := os.WriteFile(out, []byte(want), 0o644); err != nil { + t.Fatalf("writing %s: %v", out, err) + } + t.Logf("wrote %s", out) + return + } + + got, err := os.ReadFile(out) + if err != nil { + t.Fatalf("reading %s (regenerate with -update-backing): %v", out, err) + } + // Compared without carriage returns: git hands this file to a Windows + // checkout with CRLF endings, and a guard about which module runs where must + // not fail over how the working tree was written. It did, on the first run of + // CI that saw it. + if unixEndings(string(got)) != unixEndings(want) { + t.Errorf("%s no longer matches go.mod.\n\nA module has moved between an external\n"+ + "go-ruby library and this repository. Regenerate with:\n\n"+ + " go test -run TestBenchmarkedModulesSayWhereTheirCodeLives -update-backing\n\n"+ + "and then say so wherever that module's performance is published: the page\n"+ + "in its own org opens by naming the library rbgo binds for it.\n\n%s", + out, firstDifference(unixEndings(string(got)), unixEndings(want))) + } +} + +// unixEndings drops carriage returns so that the comparison is about the table +// and not about the checkout. +func unixEndings(s string) string { return strings.ReplaceAll(s, "\r", "") } + +// requiredModules pulls the module paths out of go.mod's require blocks. It is +// deliberately a parse of the text rather than a call to `go list`: the test has +// to answer the same question a reader of go.mod would. +func requiredModules(gomod string) map[string]bool { + out := map[string]bool{} + inBlock := false + for _, line := range strings.Split(gomod, "\n") { + s := strings.TrimSpace(line) + if i := strings.Index(s, "//"); i >= 0 { + s = strings.TrimSpace(s[:i]) + } + switch { + case s == "require (": + inBlock = true + continue + case inBlock && s == ")": + inBlock = false + continue + case strings.HasPrefix(s, "require "): + s = strings.TrimSpace(strings.TrimPrefix(s, "require ")) + case !inBlock: + continue + } + if fields := strings.Fields(s); len(fields) >= 2 { + out[fields[0]] = true + } + } + return out +} + +// firstDifference names the first line that differs, because a diff of a +// generated table is unreadable and the answer is always one row. +func firstDifference(got, want string) string { + g, w := strings.Split(got, "\n"), strings.Split(want, "\n") + for i := 0; i < len(g) || i < len(w); i++ { + gl, wl := "", "" + if i < len(g) { + gl = g[i] + } + if i < len(w) { + wl = w[i] + } + if gl != wl { + return fmt.Sprintf("first difference at line %d:\n file: %q\n go.mod: %q", i+1, gl, wl) + } + } + return "(the files differ only in trailing bytes)" +}