Repository navigation
fix(journal): give each node registry slot its own journal region - #15
Merged
Merged
Conversation
Node A restarts while node B keeps creating files. On main, A replays B's live journal, both nodes journal at offset 5MB, and after B crashes no node can find B's journal to replay it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every node used the journal at 5MB. A restarting node replayed and then reinitialized the journal live peers were writing, and cross-node recovery could not find a crashed node's journal. Layout v2 reserves 64 journals of 4MB, one per registry slot, and moves refcounts, snapshots and data up by 252MB. -format now writes the superblock and zeroes the metadata regions; mount reads every offset from the superblock and refuses older images with a clear message. Register records the slot's journal region, starts heartbeats, waits 100ms and re-reads the slot so two nodes racing for it do not both keep it, and skips slots a peer is recovering. Slot clears compare and swap so they never wipe a newer owner. Cross-node replay opens the region by slot and treats a region without a journal as clean. fsck phase 1 checks the journal region splits into one journal per registry slot; phase 2 checks only the bitmap sectors that cover the volume and accepts never-written sectors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FS.Close closes the device, so releasing the slot afterwards failed with EINVAL. Releasing it earlier would free the slot before the journal is marked clean. The slot frees after the 10s heartbeat timeout, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Format wrote zeros over every metadata region. On a fresh sparse image that turned the snapshot registry hole into allocated blocks, and the snapshot registry scan at mount (two passes of 7936 4KB O_DIRECT reads) went from page-cache-free hole reads to disk reads: about 2s per pass on CI runners, enough to push the integration suite past its 5m timeout. Format now writes only ranges that are not already zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Journal Crash Recovery Multiple Ops slept 4s after docker start, then called start_goclusterfs, which does not exist, and read file1.txt from the still-unmounted mount point. Both CI failures (main run 40105056198, PR run 40116360883) log "start_goclusterfs: command not found" and "cat: /mnt/goclusterfs/file1.txt: No such file or directory". Network Partition then wrote through node1 before it was mounted. Container restarts take 5-10s on CI. The tests now wait for the node to be healthy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s resumable A node that crashed and restarted inside the 10s heartbeat timeout took a new slot and served at once, while its dirty journal waited about 20s for failure detection. Replaying it then could undo newer writes. Slots now record the mount's boot ID, PID and process start time. Register first takes back the slot of a mount process on this host that has exited, so mount replays that journal before serving, and failure detection treats such slots as failed without waiting. StartFailureDetection runs one pass before returning, so mount also replays other failed slots first. A failed or interrupted peer replay left the journal Recovering and the slot Dead for good: Recover refused Recovering journals, Register skipped Dead slots, and the error was dropped. Dead slots now carry a recovery lease (owner and start time). Another node takes over once the lease lapses, Register can claim such a slot and replay it at mount, and Recover resumes a Recovering journal. Replay errors are logged, and mount stops failure detection before closing the device. Register now compares against the exact bytes it judged claimable, and mount exits when its heartbeat finds another node in its slot, so two nodes never keep journaling to one region. Superblock reads require the exact v2 layout, detect pre-v2 devices that have leftover bytes at offset 0, and format syncs the zeroed regions before writing the superblock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ock contention On ext4 images node B's create failed with EIO while restarted node A listed the root directory: the root inode lock gave up after 3 retries once the request context was done. That lock behavior is separate from the journal; the writer retries such creates so the test checks journals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ays from reapplying Containers with host networking share the hostname and boot ID, and each runs its mount as PID 1. A second container read the first one's PID 1 with another start time as a dead local mount, took its live slot and replayed its journal. Slots now record the PID namespace, and the PID check applies only within it. A slot of this host in another PID namespace, such as a restarted container's previous incarnation, is watched until it heartbeats (alive, skipped) or goes stale (claimed and replayed before serving). A resumed replay started from the head again and could re-apply frees after other nodes reused the blocks or inodes. Replay now moves the head past each transaction once it is applied. The recovery lease swap is re-checked after 100ms like a slot claim, the lease is renewed during replay, and replay checks it before every transaction and before resetting the journal, so a node that lost the lease stops instead of overwriting a newer journal header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 24, 2026
This was referenced Oct 24, 2026
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
* chore(vm): remove unused Packer template and image builder The harness boots the Alpine live ISO directly, so the Packer template, build-vm-image.sh and the vm-entrypoint.sh they installed are dead code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(vm): stop leaking VMs, isolate runs and keep VM logs - Traps on EXIT/INT/TERM stop the run's child processes and VMs. QEMU runs under setpriv --pdeathsig KILL, and the runner under --pdeathsig TERM, so a killed harness or runner cannot leave VMs or test processes behind. - Each run gets a private directory with its own SSH key and a copy of goclusterfs-mount taken at start. VMs accept only that key and listen on 127.0.0.1 ports from VM_SSH_BASE_PORT (default 2222). sshpass and the shared root password are gone. - VM setup no longer types long lines on the serial console, where the guest line editor wrapped and mangled them. The harness waits for the login prompt in the console log, logs in, and runs a setup script from a small read-only disk. A failed bring-up is retried once. - goclusterfs-mount writes its log to a virtio-serial port backed by a host file, so tests/vm/results/vmN.mount.log keeps everything up to a kill. Console and QEMU logs land there too, with boot, stop, kill and per-test markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(vm): ignore stale VM pid files and clean up after a killed harness A pid file left by a killed run could name a reused PID, and kill_vm would then SIGKILL an unrelated process. is_vm_running now checks that the PID is still that VM's QEMU, kill_vm only kills a running VM, and stop_vm removes stale pid files and sockets. The runner removes the run directory when the harness that created it is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(vm): keep /var/log/goclusterfs.log in the VM The journal tests from #15 grep the in-VM mount log. Write it with tee and stream the same output to the host log device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(vm): one run per checkout and pin the Alpine ISO hash - init_vm_run takes a lock on runtime/harness.lock and refuses a second run in the same checkout, which would share pid files and ports. QEMU does not inherit the lock, so debug VMs from --start-vm do not hold it. A new run stops VMs left in the checkout by an earlier run. - --cleanup keeps the run directory of a run that still holds the lock. - download_iso checks the ISO against a pinned SHA-256 and removes a file that does not match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(vm): install attr for the snapshot tests from #16 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every node journaled at 5 MB, so a restarting node replayed and reset live peers' journals, and no node could replay a crashed peer's journal. Each registry slot now owns a 4 MB journal. This changes the on-disk format to version 2: mount refuses older images, which must be reformatted, and the smallest device is now 264 MB.
iat5 MB + i × 4 MBFixed here because per-slot journals depend on them:
-formatkept old metadata.Tests on .68:
TestMultiNodeRestartKeepsPeerJournalfails on main and passes here.-race-short(68 pass) and full (80 pass)Not in this PR:
🤖 Generated with Claude Code