Skip to content

fix(vm): stop leaking VMs, isolate runs and keep VM logs - #26

Merged
mattmattox merged 7 commits into
mainfrom
claude/vm-harness-reliability-7296cb
Oct 24, 2026
Merged

mattmattox merged 7 commits into
mainfrom
claude/vm-harness-reliability-7296cb

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

VM runs on the shared test host leaked QEMU and run-vm-tests.sh processes when they failed or were killed. In one incident, an orphan QEMU held port 2222 after its run's files were overwritten mid-run. Bring-up also failed now and then with "SSH verification failed after 60s".

  • When the harness or runner dies, traps plus setpriv --pdeathsig stop that run's VMs and child processes.
  • Each run gets its own SSH key, a copy of goclusterfs-mount taken at start, and SSH ports from VM_SSH_BASE_PORT (default 2222) bound to 127.0.0.1. sshpass and the root password are gone.
  • The SSH failures came from busybox ash wrapping typed lines at 80 columns and mangling them. VM setup now runs from a small read-only disk, and a failed bring-up is retried once.
  • The mount log streams to tests/vm/results/vmN.mount.log over virtio-serial, with test, boot, stop and kill markers. The in-VM /var/log/goclusterfs.log stays for the fix(journal): give each node registry slot its own journal region #15 tests.
  • A lock allows one run per checkout. A new run stops VMs that an earlier run left behind, and --cleanup keeps an active run's directory.
  • The Alpine ISO is checked against a pinned SHA-256.
  • Removes the unused Packer template, build-vm-image.sh and vm-entrypoint.sh.

I stream the logs instead of copying them over SSH before each stop or kill. An SSH copy would delay the kill in the power-failure tests, and streaming keeps everything up to the kill.

Results on .68, under the lock

Run Passed Time
main 4dde499 (ports 2222+) 45/45 2238s
branch at the same time (ports 2322+) 45/45 823s
branch alone 45/45 816s
branch c65734c, with the #15 and #16 VM tests (ports 2322+) 55/55 1555s
main afe6f90 at the same time 54/55: Concurrent File Creation failed, the known concurrent-create flake 3638s

Both pass the same tests. The branch is faster because a VM boots in about 12s, against about 70s of fixed sleeps on main.

Kill tests repeated on c65734c gave the same results. After a KILL to the harness the run directory is now removed too, and a second run in the same checkout is refused with the lock error. --start-vm, --stop-vm and --cleanup leave nothing behind.

Kills during a sleep 25 in VM Kill Recovery:

  • TERM to the harness: nothing left after 2s.
  • KILL to the harness: the runner gets SIGTERM and cleans up when its current command ends (23s).
  • KILL to the harness and runner: QEMU dies at once. The orphaned sleep 25 exits on its own, and --cleanup removes the run dir.
pgrep -af "qemu-system|run-vm-tests|vm-test-harness|ssh .*127.0.0.1|sleep 25", ss -ltn, and run files
## TERM-harness: before kill (harness 2100582, runner 2100598)
2100582 /bin/bash tests/vm/vm-test-harness.sh --run-tests --nodes=2
2100598 /bin/bash /home/mmattox/gcfs-vm-harness/tests/vm/scripts/run-vm-tests.sh 2
2116657 qemu-system-x86_64 -name goclusterfs-vm2 ...
2118703 sleep 25
LISTEN 0 1 127.0.0.1:2223 0.0.0.0:* 
run.eV9iR0
vm2.pid
## TERM-harness: 2s after kill
## KILL-harness: before kill (harness 2119178, runner 2119194)
2119178 /bin/bash tests/vm/vm-test-harness.sh --run-tests --nodes=2
2119194 /bin/bash /home/mmattox/gcfs-vm-harness/tests/vm/scripts/run-vm-tests.sh 2
2133595 qemu-system-x86_64 -name goclusterfs-vm2 ...
2136005 sleep 25
LISTEN 0 1 127.0.0.1:2223 0.0.0.0:* 
run.5WElX7
vm2.pid
## KILL-harness: 2s after kill
2119194 /bin/bash /home/mmattox/gcfs-vm-harness/tests/vm/scripts/run-vm-tests.sh 2
2133595 qemu-system-x86_64 -name goclusterfs-vm2 ...
2136005 sleep 25
LISTEN 0 1 127.0.0.1:2223 0.0.0.0:* 
run.5WElX7
vm2.pid
## KILL-harness: 23s after kill
run.5WElX7
## KILL-harness-and-runner: before kill (harness 2139302, runner 2139318)
2139302 /bin/bash tests/vm/vm-test-harness.sh --run-tests --nodes=2
2139318 /bin/bash /home/mmattox/gcfs-vm-harness/tests/vm/scripts/run-vm-tests.sh 2
2151234 qemu-system-x86_64 -name goclusterfs-vm2 ...
2153787 sleep 25
LISTEN 0 1 127.0.0.1:2223 0.0.0.0:* 
run.ZjOrPZ
vm2.pid
## KILL-harness-and-runner: 2s after kill
2153787 sleep 25
run.ZjOrPZ
vm2.pid
## KILL-harness-and-runner: 22s after kill
run.ZjOrPZ
vm2.pid

Not covered: the bring-up retry never fired. stop_vm still waits 30s and then kills, because the Alpine live ISO ignores ACPI power-down. That is unchanged from main.

🤖 Generated with Claude Code

mattmattox and others added 7 commits October 24, 2026 15:34
The harness boots the Alpine live ISO directly, so the Packer template,
build-vm-image.sh and the vm-entrypoint.sh they installed are dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Traps on EXIT/INT/TERM stop the run's child processes and VMs. QEMU
  runs under setpriv --pdeathsig KILL, and the runner under
  --pdeathsig TERM, so a killed harness or runner cannot leave VMs or
  test processes behind.
- Each run gets a private directory with its own SSH key and a copy of
  goclusterfs-mount taken at start. VMs accept only that key and listen
  on 127.0.0.1 ports from VM_SSH_BASE_PORT (default 2222). sshpass and
  the shared root password are gone.
- VM setup no longer types long lines on the serial console, where the
  guest line editor wrapped and mangled them. The harness waits for the
  login prompt in the console log, logs in, and runs a setup script
  from a small read-only disk. A failed bring-up is retried once.
- goclusterfs-mount writes its log to a virtio-serial port backed by a
  host file, so tests/vm/results/vmN.mount.log keeps everything up to a
  kill. Console and QEMU logs land there too, with boot, stop, kill and
  per-test markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pid file left by a killed run could name a reused PID, and kill_vm
would then SIGKILL an unrelated process. is_vm_running now checks that
the PID is still that VM's QEMU, kill_vm only kills a running VM, and
stop_vm removes stale pid files and sockets. The runner removes the run
directory when the harness that created it is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The journal tests from #15 grep the in-VM mount log. Write it with tee
and stream the same output to the host log device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- init_vm_run takes a lock on runtime/harness.lock and refuses a second
  run in the same checkout, which would share pid files and ports. QEMU
  does not inherit the lock, so debug VMs from --start-vm do not hold
  it. A new run stops VMs left in the checkout by an earlier run.
- --cleanup keeps the run directory of a run that still holds the lock.
- download_iso checks the ISO against a pinned SHA-256 and removes a
  file that does not match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	tests/vm/scripts/build-vm-image.sh
#	tests/vm/vm-test-harness.sh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 4f6d5b5 into main Oct 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant