arch-qemu-nightly: internal/vm outgrew the 60m wall on riscv64, and s390x is at 96% of it #842

Activity

1. tannevaled commented on Nov 7, 2026

   

   tannevaled

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Today's figures: s390x is at 97.7% of the wall, and riscv64 has now failed 10 nights running

   Measured from run 42363542913 (2026-11-07, scheduled, 0b8ea8ff), reading the Test under qemu step durations:

   arch	step	verdict
   arm	42 min	success
   loong64	43 min	success
   ppc64le	54 min	success
   s390x	63 min	success
   riscv64	65 min	failure

   And the number this issue is actually about — internal/vm alone on s390x, from the job log:

   ok  github.com/go-embedded-ruby/ruby/internal/vm   3517.118s
   

   3517.1 s = 58.62 min against a 60 min wall: 97.7%, so about 84 seconds of headroom. This issue records 96%; it has crept up since.

   ⛔ Two things that make these numbers easy to misread

   go test -timeout=60m is per test BINARY, not per run (.github/workflows/arch-qemu-nightly.yml:122 runs ./...). That is why s390x's step can take 63 minutes and still pass — no single package exceeded the wall. A reader comparing the 63 against the 60 would conclude the timeout is not enforced; it is, per package.

   A job duration is not a test duration. My first reading of this run used job durations, which carry checkout, setup-go and the qemu install. The step figures above are the comparable ones, and the 3517 s is the only figure directly against the wall.

   The lane has been red every night for at least 10 nights

   2026-11-07  0b8ea8ff  failure      2026-11-02  c83590c3  failure
   2026-11-06  0b8ea8ff  failure      2026-11-01  452d895f  failure
   2026-11-05  6e86fac3  failure      2026-10-31  e54df083  failure
   2026-11-04  2a35f2ca  failure      2026-10-30  e54df083  failure
   2026-11-03  5f51213b  failure      2026-10-29  e54df083  failure
   

   Ten different commits, same outcome — so this is the known internal/vm growth and not anything a particular change introduced. Worth saying explicitly, because arch-qemu-nightly is not one of the 26 checks a pull request runs: nothing merged in those ten days was ever judged by this lane, and a lane that has been red for a known reason is exactly where a second, newer cause would sit unnoticed. The four passing arches above are the evidence that there isn't one today.

   I came to this while checking main after merging #904 and #906, nearly filed a duplicate, and found this issue already naming it.

   🤖 Generated with Claude Code
2. tannevaled commented on Nov 7, 2026

   

   tannevaled

   on Nov 7, 2026

   ContributorAuthor

   More actions

   internal/vm on REAL s390x: 5m 12s, against 58m 37s under QEMU

   Measured on the LinuxONE host (ssh linuxone — IBM z17, 8 vCPU, load 0.00, 39 days up), 2026-11-07 21:25–21:30:

   $ cd ~/rbgo-s390 && time ./vm-s390x.test -test.timeout=90m
   
   real    5m11.926s
   user    8m3.020s
   sys     0m5.436s
   

   Against the same package in the nightly QEMU lane, from run 42363542913:

   ok  github.com/go-embedded-ruby/ruby/internal/vm   3517.118s
   

   3517.1 s ÷ 311.9 s ≈ 11.3× slower under emulation.

   ⛔ This reframes the issue

   The title reads "internal/vm outgrew the 60m wall". On the metal the package takes 5 minutes. It has not outgrown anything — QEMU is 11× slower than the hardware, and the 58.6 min that sits at 97.7% of the wall is almost entirely emulation overhead.

   That matters because the two diagnoses call for opposite remedies. If the package were too big, the answers are splitting it, sharding the matrix, or raising the timeout. If the emulator is the cost, those are all beside the point, and the real options are:

   • run the s390x lane on this host instead of QEMU (it is idle, and ssh linuxone works non-interactively),
   • or treat the QEMU lane as advisory and stop gating on a wall it will keep crossing as the package grows at all.

   user 8m3s against real 5m12s is ~1.55 cores busy, so the 8 vCPUs are nowhere near saturated — the run is not fighting the host.

   What the figure includes, and what it does not

   9 tests failed, every one of them in 0.00 s, and every one for want of repository files rather than anything architectural:

   message_quoting_test.go:95: only 0 source files scanned — the gate is looking in the wrong place
   nethttp_bind_test.go:399: go.mod not found above test dir
   rubyspec_shim_test.go:106: open ../../scripts/conformance/rubyspec/spec_helper.rb: no such file or directory
   

   A compiled test binary does not carry its tree. At 0.00 s each they cannot move the 5m12s, so the comparison stands — but the figure is a lower bound by a few seconds, and those 9 tests were not exercised on big-endian. I am re-running with go.mod, scripts/ and internal/ copied alongside so the number needs no caveat; I will post it.

   ⛔ No -race: it needs CGO on s390x and the host has no Go toolchain. This run says nothing about data races.

   ⭐ Everything that ran, ran correctly — on big-endian, for the first time

   CI cross-compiles for linux/s390x on every job and runs the tests on amd64; the nightly QEMU lane has been red for ten nights. So this is the first execution of internal/vm on big-endian hardware, and apart from the 9 path failures it is clean. The one real s390x defect this fleet has had was binary.NativeEndian on auxv in go-ndarray — the kind of thing only a big-endian run can find.

   ⭐ Incidentally, TestErrorMessagesUseStraightQuotesNotTheOldBacktickForm (added in #900) refused rather than reporting a clean sweep over an empty population: "only 0 source files scanned". A gate that counts its denominator caught a condition on hardware nobody had it in mind for.

   Method, for whoever repeats it

   GOOS=linux GOARCH=s390x CGO_ENABLED=0 go test -c -o vm-s390x.test ./internal/vm
   scp vm-s390x.test linuxone:~/rbgo-s390/
   scp internal/vm/testdata/native_arity_oracle.tsv linuxone:~/rbgo-s390/testdata/   # fixtures do NOT travel in the binary
   ssh linuxone 'cd ~/rbgo-s390 && time ./vm-s390x.test -test.timeout=90m > run1.log 2>&1'

   Write results to a file on the host: they outlive the local scratchpad.

   🤖 Generated with Claude Code
3. tannevaled commented on Nov 7, 2026

   

   tannevaled

   on Nov 7, 2026

   ContributorAuthor

   More actions

   The clean rerun: 5m 14s, and the caveat was worth two seconds

   The figure above carried a caveat — 9 tests failed for want of repository files, each in 0.00 s, so the 5m11.9s was a lower bound "by a few seconds". Copied go.mod, go.sum, scripts/ and internal/ to the host (18 MB) and ran the same binary from repo/internal/vm so those paths resolve:

   real    5m14.018s
   user    8m3.813s
   sys     0m5.722s
   

   5m 14.0s against 5m 11.9s — the reasoning held, and now it is measured rather than argued. Against QEMU's 3517.1 s that is ≈11.2×.

   Failures: 9 → 2, and the two left are the same shape as the seven that went away:

   nethttp_bind_test.go:402: cannot build cmd/rbgo (exec: "go": executable file not found in $PATH)
   socket_test.go:435:      cannot build cmd/rbgo (exec: "go": executable file not found in $PATH)
   

   They shell out to go build, and the host has no Go toolchain — which is why the binary is cross-compiled in the first place. Both refuse rather than skipping, so they cannot pass vacuously there.

   So, on big-endian hardware

   Everything in internal/vm runs clean except two tests that need a compiler on the host. That includes the seven that could not resolve their paths on the first run — the repository-wide quoting gate and every Shim* test — so they have now been exercised on big-endian too.

   ⛔ Still not covered, and worth repeating so it is not read as a clean bill: no -race (needs CGO on s390x), and these are the package's own tests, not the ruby/spec corpus.

   🤖 Generated with Claude Code