The conformance harness reads its tally out of the spec's own stdout: a spec file can dictate its recorded pass count #843

A spec file can dictate its own recorded pass count

The shim reports its tally on stdout, into the same stream the spec file
writes to:

# scripts/conformance/rubyspec/spec_helper.rb:1137
$stdout.puts "RBGO_RESULT pass=#{$RB_PASS} fail=#{$RB_FAIL} error=#{$RB_ERROR} skip=#{$RB_SKIP}"

and run.sh fishes it back out of that stream:

# run.sh:92-99
out="$(cd "$SPECDIR" && timeout "$TIMEOUT" "$RBGO" "$f" 2>&1 | tr -d '\000')"
res="$(printf '%s\n' "$out" | grep '^RBGO_RESULT' | head -1)"
...
p="$(printf '%s' "$res" | sed -E 's/.*pass=([0-9]+).*/\1/')"

Data and protocol share one channel, and head -1 resolves the ambiguity in
favour of whichever came first. The shim's line is printed at exit, so
anything the spec prints during the run is first.

Measured

A probe placed in the corpus, run through the real binary (built from
3a3fb81):

require_relative '../spec_helper'
puts "RBGO_RESULT pass=9999 fail=0 error=0 skip=0"
describe "a probe" do
  it "fails on purpose" do
    1.should == 2
  end
end

$ rbgo core/fakechannel_probe_spec.rb | grep -n '^RBGO_RESULT'
1:RBGO_RESULT pass=9999 fail=0 error=0 skip=0     <- the spec's own output
2:RBGO_RESULT pass=0 fail=1 error=0 skip=0        <- the shim, at exit

head -1 takes line 1. A file that fails every example would be recorded as
OK <path> 9999, and with UPDATE_BASELINE=1 that number is banked. The probe
was deleted immediately; it is quoted here rather than committed.

How much does this matter today

Not as an attack: SPEC_SHA pins the corpus to a ruby/spec commit, and nothing
in ruby/spec prints that string. It matters as a confused channel, in three
ways that do not need an adversary:

1. a SPEC_SHA bump pulls in whatever upstream has written since;
2. any spec that legitimately prints a line starting with RBGO_RESULT — a spec
   for puts, for $stdout, for String#% with crafted input — does it by
   accident, and the harness cannot tell;
3. 2>&1 means stderr can do it too, so a warning text is enough.

The gate's own BASELINE header argues at length that a measurement must pin
every variable it can. The reporting channel is one of them.

Suggested fix, which also removes the sed

Have the shim write its tally to a path the harness names, e.g.

if (p = ENV["RBGO_RESULT_FILE"])
  File.write(p, "pass=#{$RB_PASS}\tfail=#{$RB_FAIL}\terror=#{$RB_ERROR}\tskip=#{$RB_SKIP}\n")
else
  $stdout.puts "RBGO_RESULT ..."   # unchanged for a hand-run
end

A file the spec does not know the name of cannot be impersonated by printing.
The harness then reads that file instead of pattern-matching a stream, which
deletes the grep | head -1 and the sed -E 's/.*pass=([0-9]+).*/\1/' — the
last field extraction left in run_one, and the one place where a
not-quite-matching line yields a non-numeric count (caught today only because
cmd/ratchet refuses it with exit 2). An absent or empty file is a FILEFAIL,
which is the behaviour already wanted for a crash or a timeout.

run.sh's own comment says "everything above this line runs processes; everything
below it decides, and that half is Go". This moves the last piece of deciding
below that line.

Secondary, same file

RBGO defaults to /tmp/rbgo-rubyspec and CACHE to /tmp/rubyspec-corpus
(run.sh:46-47). Correct on an ephemeral runner; locally those are directories a
reboot empties, so every reboot silently re-clones the corpus and rebuilds.
A default under the user's cache directory would keep both.

Measured 2026-10-31 on darwin/arm64, corpus at 87b1631.

🤖 Generated with Claude Code