Skip to content

fix(snapshot): make snapshots and clones copy on write - #16

Merged
mattmattox merged 18 commits into
mainfrom
fix-snapshot-cow
Oct 24, 2026
Merged

mattmattox merged 18 commits into
mainfrom
fix-snapshot-cow

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Snapshots kept nothing and clone writes changed their source. New blocks have refcount 0, a snapshot raised them to 1, and copy-on-write needs more than 1.

Refcounts.

  • 0 and 1 both mean one owner. Sharing a block adds an owner (0 becomes 2). Every free drops one owner, and the block goes back to the bitmap only when none is left.
  • New sharing sets FlagCounted on the extents. FlagShared without it comes from older versions, which never counted owners. Those blocks are never freed, and snapshotting or cloning such a file returns ENOTSUP.
  • A copy-on-write leaves a pending change, so if two nodes copy the same block, the second inode write fails with a tree conflict. That node frees its own copy and retries; the old block is released once.

Format change. Each snapshot owns a hidden inode (FlagSnapshotData, in no directory) that shares the blocks it captured. The descriptor names it in bytes 28–31 under a new FlagHasData; those bytes held an extent tree root nothing read. There is no superblock version to bump, so the flag marks the new format. The CHANGELOG says to run fsck before the first mount after upgrading an image with snapshots or clones.

Ordering.

  • Create: claim the data inode, write the descriptor as creating, mark the file shared, share the blocks, sync, write the data inode, then activate (creating to active only).
  • Delete and crash recovery: commit the empty data inode, free its blocks, remove the descriptor, then free the inode.
  • The registry takes the mount's lock manager. Two nodes used to be able to pick the same slot or ID, so one node's delete could remove the other's snapshot.
  • A crash can leak blocks, never free one twice.
  • With FS locks on (fix(lock): stop double lock holders and take FS locks on mounts #13), a write into a shared block persists and drops the inode lock, copies the block, then takes the lock and keeps the copy only if the inode is unchanged. Old blocks are released after the final unlock.
  • A plain inode write also fails with a tree conflict when the inline extents on disk changed since the copy was read, so a stale chmod cannot undo a committed copy.

Not covered.

  • Two nodes can still both claim one free inode until lock claims are confirmed (delta-lease), so two snapshots can share a data inode. TestSnapshotConcurrentCreatesAcrossNodes skips under -short like test: skip the two-node first-write test in short mode until lock claims are confirmed #22's test; the Docker and VM versions still run.
  • Some error returns after a mid-write persist skip releasing the copied blocks: the MetaGeneration mismatches, a non-conflict inode write error, and a data write error. Committed copies' old blocks keep an extra count, and uncommitted copies leak on EIO. None of these frees a block that is still mapped.
  • Snapshots of a deleted file keep their blocks until someone deletes them through the reused inode number.
  • A mount rolls back any snapshot still creating, including one another node is making; that create then fails.

Tests on 172.25.1.68 (1a6db24 on 28fed6f).

  • golangci-lint: 0 issues. Unit tests pass with -race.
  • Integration -short: 79 passed, 0 failed. The full snapshot and clone set without -short: 21 passed.
  • make test-parity passes.
  • Docker and VM: queued; results to follow.
  • With FS locks live, the old copy-on-write failed every write into a shared block with EIO ("cannot acquire 4 while holding 8"). The new one passes the in-process tests, which now run with a lock manager per node.

🤖 Generated with Claude Code

mattmattox and others added 16 commits October 24, 2026 15:26
New blocks start at a count of 0, so a snapshot raised them to 1 and
copy-on-write, which needs more than 1, never fired. Share treats 0 and
1 alike as one owner and adds the new one, and Release drops an owner
and reports the blocks nobody holds any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Freed blocks now drop one owner in the refcount table and go back to
the bitmap only when none is left. Shared blocks used to be skipped and
leaked. A shared block with a count of 0 comes from a clone made by an
older version, so it stays allocated. A copy-on-write remap bumps
DataGeneration so readers drop cached block mappings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot stored no block list, so deleting it used the file's current
blocks and a clone of it copied the live file. Each snapshot now owns a
hidden inode that shares the blocks it captured; the descriptor names it
in bytes 28-31 under the new FlagHasData. Creation shares the blocks
before marking the file shared, so later writes copy them. Delete and
crash recovery give back the data inode's blocks. Jobs no longer drop
when the request context is canceled. fsck accepts data inodes and
treats a refcount of 0 as one owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uites

Adds Go, Docker and VM tests that read a snapshot through a clone after
in-place overwrite, append and truncate, write into a clone and its
source with dd conv=notrunc, and delete a snapshot after an overwrite.
The Docker image and VMs install attr for setfattr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If a crash hit after the data inode was freed but before the descriptor
was removed, another snapshot could reuse the slot and recovery would
empty it. Freeing last leaks at most an empty inode, which fsck reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aring

Two nodes could copy the same shared block of an inline file: neither
inode write was checked, so both released the old block and freed it
while a snapshot still mapped it. A copy-on-write now leaves a pending
change, so the second write fails with a tree conflict and retries.

New sharing sets FlagCounted. Blocks older versions marked shared
without it have unknown owner counts, so frees leave them allocated and
HasUncountedSharing lets callers refuse to share them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efcounts

- Mark the source shared before its blocks gain owners, and sync the
  refcounts before the copy maps them, so a crash leaves at most a
  leak.
- Refuse to snapshot or clone a file an older version shared
  (ENOTSUP).
- Re-check that a snapshot is still active before a clone of it maps
  its blocks.
- Only a snapshot still being created can become active.
- Snapshot IDs come from the highest ID on disk, so two nodes no longer
  reuse one.
- Refcounts stay on with -snapshots=false, so shared blocks are still
  copied before writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The registry got FS.lockMgr, which is nil, so two nodes creating
snapshots at once could take the same slot or ID: one snapshot vanished
or a delete by ID removed the other node's. InitSnapshots now takes the
mount's lock manager.

Adds a two-node registry test and Go, Docker and VM tests that snapshot
on two nodes at once. Adds tests where node A commits a copy of a shared
block between node B's copy and commit, and where an older version's
snapshot and clone of one file are written and deleted. The ENOTSUP for
files an older version shared now points to goclusterfs-fsck, and the
CHANGELOG says to run fsck before the first mount after upgrading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tive snapshots

A copy-on-write of a block an older version shared without counting its
owners released it like a counted one. With two old snapshots and two
old clones, two copies dropped the count from 2 to 0 and freed a block
the other clone mapped. Such blocks are now copied but never released.

Mount recovery marked any snapshot it saw creating as deleting, even if
another node had activated it since. It now uses a guarded
creating-to-deleting transition, and the delete job refuses snapshots
that are not marked deleting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In CI, two nodes creating snapshots at once still got the same ID even
under the registry lock, because the cross-node lock does not always
exclude them. A new ID is now a counter above the highest ID on disk,
times 64, plus the node slot. Each node fills descriptor sectors it
alone owns first, so two nodes neither reuse an ID nor overwrite each
other's sector. The registry test now runs with no lock at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With #13's FS locks, a write into a shared block held the inode lock and
then needed the refcount and bitmap locks, which come first in lock
order, so it failed with EIO. The write now persists and drops the inode
lock, copies the block on its unlocked copy, then takes the lock and
keeps the copy only if no node wrote the inode meanwhile; otherwise it
frees the copy and retries. A conflict after an earlier mid-write commit
frees only the copies that commit did not map.

The in-process snapshot tests now run with a lock manager per FS, like
separate nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers #13's release of committed copies on a failed final inode write:
each old block loses the file's ownership once across the retry, and
uncounted blocks from an older version lose none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two nodes can still both claim one free inode until lock claims are
confirmed (delta-lease), and creating 20 snapshots on cold lock records
across two nodes hits that often. The Docker and VM versions still run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox and others added 2 commits October 24, 2026 15:32
The stale-write check compared only extent trees. A copy of an inline
inode read before a copy-on-write committed, such as a slow chmod on
another node, could write the old mapping back after the old block was
released, leaving the snapshot's block undercounted. A plain write now
fails with ErrTreeConflict when the inline extents on disk differ from
the ones the copy was read with; callers already retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pted snapshot calls

A copy-on-write on an unlocked inode copy can find the tree rewritten by
another node; it now retries instead of returning EIO. Snapshot and clone
xattrs span several writes, and go-fuse cancels the context when the
kernel interrupts a request, which made a contended registry lock fail
with EIO. They now run to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 5243e17 into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* chore(vm): remove unused Packer template and image builder

The harness boots the Alpine live ISO directly, so the Packer template,
build-vm-image.sh and the vm-entrypoint.sh they installed are dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): stop leaking VMs, isolate runs and keep VM logs

- Traps on EXIT/INT/TERM stop the run's child processes and VMs. QEMU
  runs under setpriv --pdeathsig KILL, and the runner under
  --pdeathsig TERM, so a killed harness or runner cannot leave VMs or
  test processes behind.
- Each run gets a private directory with its own SSH key and a copy of
  goclusterfs-mount taken at start. VMs accept only that key and listen
  on 127.0.0.1 ports from VM_SSH_BASE_PORT (default 2222). sshpass and
  the shared root password are gone.
- VM setup no longer types long lines on the serial console, where the
  guest line editor wrapped and mangled them. The harness waits for the
  login prompt in the console log, logs in, and runs a setup script
  from a small read-only disk. A failed bring-up is retried once.
- goclusterfs-mount writes its log to a virtio-serial port backed by a
  host file, so tests/vm/results/vmN.mount.log keeps everything up to a
  kill. Console and QEMU logs land there too, with boot, stop, kill and
  per-test markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): ignore stale VM pid files and clean up after a killed harness

A pid file left by a killed run could name a reused PID, and kill_vm
would then SIGKILL an unrelated process. is_vm_running now checks that
the PID is still that VM's QEMU, kill_vm only kills a running VM, and
stop_vm removes stale pid files and sockets. The runner removes the run
directory when the harness that created it is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): keep /var/log/goclusterfs.log in the VM

The journal tests from #15 grep the in-VM mount log. Write it with tee
and stream the same output to the host log device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): one run per checkout and pin the Alpine ISO hash

- init_vm_run takes a lock on runtime/harness.lock and refuses a second
  run in the same checkout, which would share pid files and ports. QEMU
  does not inherit the lock, so debug VMs from --start-vm do not hold
  it. A new run stops VMs left in the checkout by an earlier run.
- --cleanup keeps the run directory of a run that still holds the lock.
- download_iso checks the ISO against a pinned SHA-256 and removes a
  file that does not match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vm): install attr for the snapshot tests from #16

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* docs: sync operator guides with current flags and behavior

OPERATIONS, DEPLOYMENT and TROUBLESHOOTING still used the old
goclusterfs binary name, a -format without -mount, a .snapshots
directory and snapshot API that never existed, lost+found, a
-fuse-debug flag, a mount -version flag, pprof, and read-only mounts.

They now match the code at e31fa92: format v2 and per-node journals,
-skip-recovery and damaged journal slots (#34), fsck exit codes and
-repair (#32), -attr-timeout (#24), lock renewal and takeover (#13,
#23), xattr snapshots and clones (#16), and the real log lines.

OPERATIONS.md gains a Security Model section: trust in the shared
device, single-user mounts with no permission checks, lock limits,
and unauthenticated broadcast unless -broadcast-secret is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: lock waits have no time limit

ErrLockTimeout only fires when the request context is cancelled;
lock.DefaultTimeout is never used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: make damaged-journal recovery safe and fix review findings

-skip-recovery discards the journal of whichever slot the mount takes,
which is not the damaged slot once a peer holds its recovery lease.
The steps now say when to run it and how to check the slot, and only
use it for a damaged header, not for I/O or replay errors.

Also quote the real legacy-layout and superblock errors, cover a v2
device with a wiped superblock, state that there is no HTTP API, and
correct peer-announce, replay, unicast fallback and cache lag claims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant