Skip to content

fix(journal): give each node registry slot its own journal region - #15

Merged
mattmattox merged 12 commits into
mainfrom
claude/per-node-journal-aca2aa
Oct 24, 2026
Merged

mattmattox merged 12 commits into
mainfrom
claude/per-node-journal-aca2aa

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Every node journaled at 5 MB, so a restarting node replayed and reset live peers' journals, and no node could replay a crashed peer's journal. Each registry slot now owns a 4 MB journal. This changes the on-disk format to version 2: mount refuses older images, which must be reformatted, and the smallest device is now 264 MB.

Offset Size Region
0 1 MB superblock
1 MB 1 MB block bitmap
2 MB 1 MB inode table
3 MB 1 MB node registry (64 slots)
4 MB 1 MB locks
5 MB 256 MB journals: slot i at 5 MB + i × 4 MB
261 MB 1 MB refcounts
262 MB 1 MB snapshot registry
263 MB rest data

Fixed here because per-slot journals depend on them:

  • Mount never sent heartbeats, so later mounts took live nodes' slots.
  • Two nodes registering at once could both keep one slot.
  • -format kept old metadata.

Tests on .68:

  • TestMultiNodeRestartKeepsPeerJournal fails on main and passes here.
  • Before rebasing onto feat(extent): store extents beyond the inline two in an overflow tree #8 and fix: data-safety bugs in fallocate, block 0 reads, and SFB #12, everything passed:
    • unit tests with -race
    • integration -short (68 pass) and full (80 pass)
    • Docker: 32 pass, 10 skipped
    • VM: 46/46
  • After the rebase, unit tests and lint pass. Integration, Docker and VM are rerunning.
  • I ran the Docker scenarios that flaked in CI 10x each (Journal Recovery Multi-Node, Journal Crash Recovery Multiple Ops, Network Partition). All 30 passed on both main and this branch. In main's node logs, 17 of node1's 20 restarts replayed another node's journal; on this branch, 0 of 14.

Not in this PR:

  • The 1 MB refcount region overflows past about 128 GB of data, on main too. Fixing it changes the layout.
  • Nothing fences a node paused for more than 10 s, which can then share its journal with a new mount.

🤖 Generated with Claude Code

mattmattox and others added 12 commits October 24, 2026 12:27
Node A restarts while node B keeps creating files. On main, A replays
B's live journal, both nodes journal at offset 5MB, and after B crashes
no node can find B's journal to replay it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every node used the journal at 5MB. A restarting node replayed and then
reinitialized the journal live peers were writing, and cross-node
recovery could not find a crashed node's journal.

Layout v2 reserves 64 journals of 4MB, one per registry slot, and moves
refcounts, snapshots and data up by 252MB. -format now writes the
superblock and zeroes the metadata regions; mount reads every offset from
the superblock and refuses older images with a clear message.

Register records the slot's journal region, starts heartbeats, waits
100ms and re-reads the slot so two nodes racing for it do not both keep
it, and skips slots a peer is recovering. Slot clears compare and swap so
they never wipe a newer owner. Cross-node replay opens the region by
slot and treats a region without a journal as clean.

fsck phase 1 checks the journal region splits into one journal per
registry slot; phase 2 checks only the bitmap sectors that cover the
volume and accepts never-written sectors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FS.Close closes the device, so releasing the slot afterwards failed with
EINVAL. Releasing it earlier would free the slot before the journal is
marked clean. The slot frees after the 10s heartbeat timeout, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Format wrote zeros over every metadata region. On a fresh sparse image
that turned the snapshot registry hole into allocated blocks, and the
snapshot registry scan at mount (two passes of 7936 4KB O_DIRECT reads)
went from page-cache-free hole reads to disk reads: about 2s per pass on
CI runners, enough to push the integration suite past its 5m timeout.
Format now writes only ranges that are not already zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Journal Crash Recovery Multiple Ops slept 4s after docker start, then
called start_goclusterfs, which does not exist, and read file1.txt from
the still-unmounted mount point. Both CI failures (main run 40105056198,
PR run 40116360883) log "start_goclusterfs: command not found" and
"cat: /mnt/goclusterfs/file1.txt: No such file or directory". Network
Partition then wrote through node1 before it was mounted. Container
restarts take 5-10s on CI. The tests now wait for the node to be healthy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s resumable

A node that crashed and restarted inside the 10s heartbeat timeout took a
new slot and served at once, while its dirty journal waited about 20s for
failure detection. Replaying it then could undo newer writes. Slots now
record the mount's boot ID, PID and process start time. Register first
takes back the slot of a mount process on this host that has exited, so
mount replays that journal before serving, and failure detection treats
such slots as failed without waiting. StartFailureDetection runs one
pass before returning, so mount also replays other failed slots first.

A failed or interrupted peer replay left the journal Recovering and the
slot Dead for good: Recover refused Recovering journals, Register skipped
Dead slots, and the error was dropped. Dead slots now carry a recovery
lease (owner and start time). Another node takes over once the lease
lapses, Register can claim such a slot and replay it at mount, and
Recover resumes a Recovering journal. Replay errors are logged, and mount
stops failure detection before closing the device.

Register now compares against the exact bytes it judged claimable, and
mount exits when its heartbeat finds another node in its slot, so two
nodes never keep journaling to one region.

Superblock reads require the exact v2 layout, detect pre-v2 devices that
have leftover bytes at offset 0, and format syncs the zeroed regions
before writing the superblock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ock contention

On ext4 images node B's create failed with EIO while restarted node A
listed the root directory: the root inode lock gave up after 3 retries
once the request context was done. That lock behavior is separate from
the journal; the writer retries such creates so the test checks journals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ays from reapplying

Containers with host networking share the hostname and boot ID, and each
runs its mount as PID 1. A second container read the first one's PID 1
with another start time as a dead local mount, took its live slot and
replayed its journal. Slots now record the PID namespace, and the PID
check applies only within it. A slot of this host in another PID
namespace, such as a restarted container's previous incarnation, is
watched until it heartbeats (alive, skipped) or goes stale (claimed and
replayed before serving).

A resumed replay started from the head again and could re-apply frees
after other nodes reused the blocks or inodes. Replay now moves the head
past each transaction once it is applied.

The recovery lease swap is re-checked after 100ms like a slot claim, the
lease is renewed during replay, and replay checks it before every
transaction and before resetting the journal, so a node that lost the
lease stops instead of overwriting a newer journal header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit a1957da into main Oct 24, 2026
6 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
* chore(vm): remove unused Packer template and image builder

The harness boots the Alpine live ISO directly, so the Packer template,
build-vm-image.sh and the vm-entrypoint.sh they installed are dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): stop leaking VMs, isolate runs and keep VM logs

- Traps on EXIT/INT/TERM stop the run's child processes and VMs. QEMU
  runs under setpriv --pdeathsig KILL, and the runner under
  --pdeathsig TERM, so a killed harness or runner cannot leave VMs or
  test processes behind.
- Each run gets a private directory with its own SSH key and a copy of
  goclusterfs-mount taken at start. VMs accept only that key and listen
  on 127.0.0.1 ports from VM_SSH_BASE_PORT (default 2222). sshpass and
  the shared root password are gone.
- VM setup no longer types long lines on the serial console, where the
  guest line editor wrapped and mangled them. The harness waits for the
  login prompt in the console log, logs in, and runs a setup script
  from a small read-only disk. A failed bring-up is retried once.
- goclusterfs-mount writes its log to a virtio-serial port backed by a
  host file, so tests/vm/results/vmN.mount.log keeps everything up to a
  kill. Console and QEMU logs land there too, with boot, stop, kill and
  per-test markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): ignore stale VM pid files and clean up after a killed harness

A pid file left by a killed run could name a reused PID, and kill_vm
would then SIGKILL an unrelated process. is_vm_running now checks that
the PID is still that VM's QEMU, kill_vm only kills a running VM, and
stop_vm removes stale pid files and sockets. The runner removes the run
directory when the harness that created it is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): keep /var/log/goclusterfs.log in the VM

The journal tests from #15 grep the in-VM mount log. Write it with tee
and stream the same output to the host log device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): one run per checkout and pin the Alpine ISO hash

- init_vm_run takes a lock on runtime/harness.lock and refuses a second
  run in the same checkout, which would share pid files and ports. QEMU
  does not inherit the lock, so debug VMs from --start-vm do not hold
  it. A new run stops VMs left in the checkout by an earlier run.
- --cleanup keeps the run directory of a run that still holds the lock.
- download_iso checks the ISO against a pinned SHA-256 and removes a
  file that does not match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): install attr for the snapshot tests from #16

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant