Repository navigation
fix(vm): stop leaking VMs, isolate runs and keep VM logs - #26
Merged
Merged
Conversation
The harness boots the Alpine live ISO directly, so the Packer template, build-vm-image.sh and the vm-entrypoint.sh they installed are dead code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Traps on EXIT/INT/TERM stop the run's child processes and VMs. QEMU runs under setpriv --pdeathsig KILL, and the runner under --pdeathsig TERM, so a killed harness or runner cannot leave VMs or test processes behind. - Each run gets a private directory with its own SSH key and a copy of goclusterfs-mount taken at start. VMs accept only that key and listen on 127.0.0.1 ports from VM_SSH_BASE_PORT (default 2222). sshpass and the shared root password are gone. - VM setup no longer types long lines on the serial console, where the guest line editor wrapped and mangled them. The harness waits for the login prompt in the console log, logs in, and runs a setup script from a small read-only disk. A failed bring-up is retried once. - goclusterfs-mount writes its log to a virtio-serial port backed by a host file, so tests/vm/results/vmN.mount.log keeps everything up to a kill. Console and QEMU logs land there too, with boot, stop, kill and per-test markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pid file left by a killed run could name a reused PID, and kill_vm would then SIGKILL an unrelated process. is_vm_running now checks that the PID is still that VM's QEMU, kill_vm only kills a running VM, and stop_vm removes stale pid files and sockets. The runner removes the run directory when the harness that created it is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The journal tests from #15 grep the in-VM mount log. Write it with tee and stream the same output to the host log device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- init_vm_run takes a lock on runtime/harness.lock and refuses a second run in the same checkout, which would share pid files and ports. QEMU does not inherit the lock, so debug VMs from --start-vm do not hold it. A new run stops VMs left in the checkout by an earlier run. - --cleanup keeps the run directory of a run that still holds the lock. - download_iso checks the ISO against a pinned SHA-256 and removes a file that does not match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # tests/vm/scripts/build-vm-image.sh # tests/vm/vm-test-harness.sh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VM runs on the shared test host leaked QEMU and
run-vm-tests.shprocesses when they failed or were killed. In one incident, an orphan QEMU held port 2222 after its run's files were overwritten mid-run. Bring-up also failed now and then with "SSH verification failed after 60s".setpriv --pdeathsigstop that run's VMs and child processes.goclusterfs-mounttaken at start, and SSH ports fromVM_SSH_BASE_PORT(default 2222) bound to 127.0.0.1. sshpass and the root password are gone.tests/vm/results/vmN.mount.logover virtio-serial, with test, boot, stop and kill markers. The in-VM/var/log/goclusterfs.logstays for the fix(journal): give each node registry slot its own journal region #15 tests.--cleanupkeeps an active run's directory.build-vm-image.shandvm-entrypoint.sh.I stream the logs instead of copying them over SSH before each stop or kill. An SSH copy would delay the kill in the power-failure tests, and streaming keeps everything up to the kill.
Results on .68, under the lock
Both pass the same tests. The branch is faster because a VM boots in about 12s, against about 70s of fixed sleeps on main.
Kill tests repeated on c65734c gave the same results. After a KILL to the harness the run directory is now removed too, and a second run in the same checkout is refused with the lock error.
--start-vm,--stop-vmand--cleanupleave nothing behind.Kills during a
sleep 25in VM Kill Recovery:sleep 25exits on its own, and--cleanupremoves the run dir.pgrep -af "qemu-system|run-vm-tests|vm-test-harness|ssh .*127.0.0.1|sleep 25", ss -ltn, and run files
Not covered: the bring-up retry never fired.
stop_vmstill waits 30s and then kills, because the Alpine live ISO ignores ACPI power-down. That is unchanged from main.🤖 Generated with Claude Code