Skip to content

fix(lock): stop double lock holders and take FS locks on mounts - #13

Merged
mattmattox merged 22 commits into
mainfrom
claude/fix-lock-double-holders-9658e1
Oct 24, 2026
Merged

mattmattox merged 22 commits into
mainfrom
claude/fix-lock-double-holders-9658e1

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Two nodes could hold the same lock in normal operation, and mounted filesystems never took the FS-level locks at all.

  • Lock records: the region has 256 records, and a record held for one resource counted as free for any other resource that hashed to it. Any node could take over, say, the block bitmap lock. Same-node races and shared release also left locks unheld on disk.
  • Inode allocation: two nodes could claim the same free inode, because the inode lock did not exclude the other node. In fix(snapshot): make snapshots and clones copy on write #16's snapshot test both nodes got data inode 5 in 3 of 45 runs on main, and 0 of 45 with this PR.
  • Mounts: cmd/mount never passed the lock manager to the FS. Two nodes writing a new file's first block lost one node's data in 11 of 20 files. The lock branches had never run and dropped every new block once enabled; that is fixed here.

Lock order with the live inode lock. The bitmap lock comes before the inode lock, so nothing allocates or frees under the inode lock:

Same-node holds: an exclusive request waits for this node's own readers, so two nodes that both read a directory cannot both write it. To keep that from hanging, nothing holds a local mutex across Acquire: the inode table, bitmap, refcount and snapshot registry now take the distributed lock before their mutex. A key joining an exclusive hold older than 5s renews the record, so a busy node does not lose it as stale. If another node already took it over, the join fails with ErrLockLost.

Safe now: node crash and clean unmount. Concurrent first writes to a file now take the inode lock, but the non-atomic takeover below still lets both nodes win about 1 file in 150 (4 of 30 test runs on main; #22 skips the test in CI until claim confirmation lands).

Not safe:

  • A node stalled for more than 15s keeps writing after losing its lock. Limits are in docs/OPERATIONS.md.
  • Lock takeover is a non-atomic read-then-write. TestMultiNodeConcurrentDirectoryOps still fails about 15% of runs here and on main (the VM version failed 3/3 on both). A follow-up adds write, wait, re-read confirmation.
  • Records have no acquisition order. Two nodes holding crossed records wait until one is 15s old, then both proceed (about 1 in 65,000 overlapping pairs).
  • Writing a shared snapshot block (copy-on-write) takes the refcount and bitmap locks while holding the inode lock. The lock order check rejects that, so the write fails with EIO. fix(snapshot): make snapshots and clones copy on write #16 makes this path reachable and moves that work ahead of the inode lock. Whichever of fix(lock): stop double lock holders and take FS locks on mounts #13 and fix(snapshot): make snapshots and clones copy on write #16 lands second adds a test that writes into a shared block on a mounted FS.

Cost (single node, tmpfs image on .68, main → this PR): 256MB of 1MB writes 0.90s → 1.17s; 32MB of 4KB writes 1.66s → 2.55s; 300 creates of 128KB 2.58s → 2.67s. Each block write now takes a distributed extent lock.

Tests:

  • GitHub CI: all 8 checks pass on the latest head.
  • .68, at 81279c6 (later commits only merge main): unit tests with -race pass; integration passes 88 tests, skips 2, and logs no lock-order errors.
  • .68, at the reviewed lock code (7750075): Docker with concurrent tests passes 43, fails 0, skips 1.
  • VM, before the review fixes: the new first-write test passes 3/3 here and fails 2/3 on main. A full VM run on this head is still to come.

🤖 Generated with Claude Code

mattmattox and others added 5 commits October 24, 2026 11:13
…ble holders

The lock region has 256 records in the production layout, and tryAcquire
treated a record held for a different resource as free. Any node could take
over a held lock by touching a resource that hashed to the same record.

Other double-holder windows fixed here:
- two goroutines on one node acquiring the same key at once left the
  on-disk record cleared while both believed they held it
- releasing a shared lock cleared the record for all readers when the
  releasing node was the last one to join
- an exclusive request piggybacked on a same-node shared hold
- reclaiming a stale shared record kept the dead readers' count
- a release whose CAS lost a race returned nil without releasing

Records now act as lock stripes: a held record blocks every resource that
hashes to it, a node can hold several keys on one record, and a per-record
gate serializes this node's on-disk acquire and release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmd/mount built the FS with NewWithComponents, which had no lock manager
parameter, so every inode, extent and SFB lock branch in internal/goclusterfs
was dead in production. Two nodes writing a new file's first block at the same
time lost one node's data in 11 of 20 files.

The dead branches had never run and lost data once enabled: after allocating a
block without the inode lock, writeInternal and Allocate re-read the inode
under the lock and dropped the new extent. They now map the new block in the
fresh inode, or free it and retry if another node mapped that block first.
Changes made under the lock are written before the lock is dropped for the
next allocation, and the final size update builds on the inode read under the
lock.

Readers joining a shared lock no longer refresh its age. A reader count leaked
by a lost CAS update now goes stale after 15s instead of blocking writers for
as long as readers keep arriving; Docker hung on rmdir without this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VM runner deletes every file before each test, so new files reuse inode
numbers and a node that read earlier files serves stale cached extents. That
made the VM check fail even though no write was lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox force-pushed the claude/fix-lock-double-holders-9658e1 branch from e834320 to f6ddae8 Compare October 24, 2026 16:18
mattmattox and others added 14 commits October 24, 2026 11:53
… busy records

An exclusive request waited for this node's shared holders of the record.
inode.Table holds its mutex across Acquire, and a directory listing holds the
directory's shared lock while it reads the table, so creating a file while the
directory was listed hung the whole mount. Exclusive requests now join the
node's existing hold, as they did before this branch. The record on disk stays
shared, so other nodes can read but not write meanwhile.

Same-node joins never touched disk, so a record kept busy by overlapping keys
kept its first AcquireTime and was taken over as stale after 15s while still
held. A key joining a hold older than renewAfter now rewrites AcquireTime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reservations

The bitmap lock comes before the inode lock, so frees made while holding it
fail the lock order check. Tree reservations released under the inode lock
parked their blocks in an in-memory list that unmount drops: appends merging
into a full inline extent leaked one block each. writeFailed freed the write's
blocks the same way. Both now run after the inode lock is released.

writeInternal released the inode lock it took for the size update, then
released it again in its deferred cleanup. If another goroutine on the node
had joined that lock in between, the second release dropped its hold. All
releases now go through one helper that clears lockHeld.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ble holders

The lock region has 256 records in the production layout, and tryAcquire
treated a record held for a different resource as free. Any node could take
over a held lock by touching a resource that hashed to the same record.

Other double-holder windows fixed here:
- two goroutines on one node acquiring the same key at once left the
  on-disk record cleared while both believed they held it
- releasing a shared lock cleared the record for all readers when the
  releasing node was the last one to join
- an exclusive request piggybacked on a same-node shared hold
- reclaiming a stale shared record kept the dead readers' count
- a release whose CAS lost a race returned nil without releasing

Records now act as lock stripes: a held record blocks every resource that
hashes to it, a node can hold several keys on one record, and a per-record
gate serializes this node's on-disk acquire and release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmd/mount built the FS with NewWithComponents, which had no lock manager
parameter, so every inode, extent and SFB lock branch in internal/goclusterfs
was dead in production. Two nodes writing a new file's first block at the same
time lost one node's data in 11 of 20 files.

The dead branches had never run and lost data once enabled: after allocating a
block without the inode lock, writeInternal and Allocate re-read the inode
under the lock and dropped the new extent. They now map the new block in the
fresh inode, or free it and retry if another node mapped that block first.
Changes made under the lock are written before the lock is dropped for the
next allocation, and the final size update builds on the inode read under the
lock.

Readers joining a shared lock no longer refresh its age. A reader count leaked
by a lost CAS update now goes stale after 15s instead of blocking writers for
as long as readers keep arriving; Docker hung on rmdir without this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VM runner deletes every file before each test, so new files reuse inode
numbers and a node that read earlier files serves stale cached extents. That
made the VM check fail even though no write was lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… busy records

An exclusive request waited for this node's shared holders of the record.
inode.Table holds its mutex across Acquire, and a directory listing holds the
directory's shared lock while it reads the table, so creating a file while the
directory was listed hung the whole mount. Exclusive requests now join the
node's existing hold, as they did before this branch. The record on disk stays
shared, so other nodes can read but not write meanwhile.

Same-node joins never touched disk, so a record kept busy by overlapping keys
kept its first AcquireTime and was taken over as stale after 15s while still
held. A key joining a hold older than renewAfter now rewrites AcquireTime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reservations

The bitmap lock comes before the inode lock, so frees made while holding it
fail the lock order check. Tree reservations released under the inode lock
parked their blocks in an in-memory list that unmount drops: appends merging
into a full inline extent leaked one block each. writeFailed freed the write's
blocks the same way. Both now run after the inode lock is released.

writeInternal released the inode lock it took for the size update, then
released it again in its deferred cleanup. If another goroutine on the node
had joined that lock in between, the second release dropped its hold. All
releases now go through one helper that clears lockHeld.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since #12, the first write to an unwritten block clears only that block, which
can split one extent into three and allocate extent-tree blocks. Under the
live inode lock that allocation failed the lock order check, so a write into
the middle of a fallocated range returned EIO. The flag is now cleared on an
unlocked copy; the write keeps it only if the inode is unchanged once the lock
is held, and retries otherwise. The lock stays held from that check through
the zero-fill, the data write and the inode commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… across Acquire

Letting an exclusive request join this node's shared hold kept the record
shared on disk, so two nodes that both read a directory could each take it
exclusively and lose each other's entries. Exclusive requests wait for this
node's readers again.

The hang that change avoided came from local mutexes held across Acquire: a
writer held inode.Table's mutex while waiting for a reader that needed it.
inode.Table, bitmap, refcount and the snapshot registry now take the
distributed lock before their mutex.

Renewal now covers exclusive holds only, so a dead node's reader count no
longer stays fresh. A join that finds its record taken over fails with
ErrLockLost, and later joins to that hold fail too. Pending tree changes are
discarded after the inode lock is dropped on every error path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox and others added 3 commits October 24, 2026 14:06
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de write fails

Before allocating its next block, a write persists the inode, which maps the
blocks allocated so far. If the final inode write then failed, writeFailed
freed every block of the write, including the ones the inode on disk still
mapped, so the retry handed them out again. It now frees only allocations and
copy-on-write copies made after the last persist, and drops the refcounts of
copies that persist already committed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox marked this pull request as ready for review October 24, 2026 19:36
@mattmattox
mattmattox merged commit 95322f2 into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
With #13's FS locks, a write into a shared block held the inode lock and
then needed the refcount and bitmap locks, which come first in lock
order, so it failed with EIO. The write now persists and drops the inode
lock, copies the block on its unlocked copy, then takes the lock and
keeps the copy only if no node wrote the inode meanwhile; otherwise it
frees the copy and retries. A conflict after an earlier mid-write commit
frees only the copies that commit did not map.

The in-process snapshot tests now run with a lock manager per FS, like
separate nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
Covers #13's release of committed copies on a failed final inode write:
each old block loses the file's ownership once across the retry, and
uncounted blocks from an older version lose none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
…ims are confirmed (#22)

Taking a lock is still a read then a write, so two nodes that claim a free
inode lock at the same moment can both hold it. The test does that for 20
files per run and loses one node's write in about 1 file in 150: 4 of 30
runs failed on main and 2 of 30 on #13 before #18 and #20 landed, so #18 is
not the cause. CI runs integration with -short; full runs still include it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
With #13's FS locks, a write into a shared block held the inode lock and
then needed the refcount and bitmap locks, which come first in lock
order, so it failed with EIO. The write now persists and drops the inode
lock, copies the block on its unlocked copy, then takes the lock and
keeps the copy only if no node wrote the inode meanwhile; otherwise it
frees the copy and retries. A conflict after an earlier mid-write commit
frees only the copies that commit did not map.

The in-process snapshot tests now run with a lock manager per FS, like
separate nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
Covers #13's release of committed copies on a failed final inode write:
each old block loses the file's ownership once across the retry, and
uncounted blocks from an older version lose none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* fix(refcount): count the existing owner when a block is first shared

New blocks start at a count of 0, so a snapshot raised them to 1 and
copy-on-write, which needs more than 1, never fired. Share treats 0 and
1 alike as one owner and adds the new one, and Release drops an owner
and reports the blocks nobody holds any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extent): free shared blocks only after their last owner lets go

Freed blocks now drop one owner in the refcount table and go back to
the bitmap only when none is left. Shared blocks used to be skipped and
leaked. A shared block with a count of 0 comes from a clone made by an
older version, so it stays allocated. A copy-on-write remap bumps
DataGeneration so readers drop cached block mappings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): keep each snapshot's blocks in a hidden data inode

A snapshot stored no block list, so deleting it used the file's current
blocks and a clone of it copied the live file. Each snapshot now owns a
hidden inode that shares the blocks it captured; the descriptor names it
in bytes 28-31 under the new FlagHasData. Creation shares the blocks
before marking the file shared, so later writes copy them. Delete and
crash recovery give back the data inode's blocks. Jobs no longer drop
when the request context is canceled. fsck accepts data inodes and
treats a refcount of 0 as one owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): check snapshot data, clone writes and delete in all suites

Adds Go, Docker and VM tests that read a snapshot through a clone after
in-place overwrite, append and truncate, write into a clone and its
source with dd conv=notrunc, and delete a snapshot after an overwrite.
The Docker image and VMs install attr for setfattr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): free a snapshot's data inode after its descriptor is gone

If a crash hit after the data inode was freed but before the descriptor
was removed, another snapshot could reuse the slot and recovery would
empty it. Freeing last leaks at most an empty inode, which fsck reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): wait for deleted snapshots to disappear

TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extent): conflict-check copy-on-write and never free uncounted sharing

Two nodes could copy the same shared block of an inline file: neither
inode write was checked, so both released the old block and freed it
while a snapshot still mapped it. A copy-on-write now leaves a pending
change, so the second write fails with a tree conflict and retries.

New sharing sets FlagCounted. Blocks older versions marked shared
without it have unknown owner counts, so frees leave them allocated and
HasUncountedSharing lets callers refuse to share them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): order sharing safely and refuse files shared without refcounts

- Mark the source shared before its blocks gain owners, and sync the
  refcounts before the copy maps them, so a crash leaves at most a
  leak.
- Refuse to snapshot or clone a file an older version shared
  (ENOTSUP).
- Re-check that a snapshot is still active before a clone of it maps
  its blocks.
- Only a snapshot still being created can become active.
- Snapshot IDs come from the highest ID on disk, so two nodes no longer
  reuse one.
- Refcounts stay on with -snapshots=false, so shared blocks are still
  copied before writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): lock the snapshot registry across nodes

The registry got FS.lockMgr, which is nil, so two nodes creating
snapshots at once could take the same slot or ID: one snapshot vanished
or a delete by ID removed the other node's. InitSnapshots now takes the
mount's lock manager.

Adds a two-node registry test and Go, Docker and VM tests that snapshot
on two nodes at once. Adds tests where node A commits a copy of a shared
block between node B's copy and commit, and where an older version's
snapshot and clone of one file are written and deleted. The ENOTSUP for
files an older version shared now points to goclusterfs-fsck, and the
CHANGELOG says to run fsck before the first mount after upgrading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): never release uncounted blocks, and never roll back active snapshots

A copy-on-write of a block an older version shared without counting its
owners released it like a counted one. With two old snapshots and two
old clones, two copies dropped the count from 2 to 0 and freed a block
the other clone mapped. Such blocks are now copied but never released.

Mount recovery marked any snapshot it saw creating as deleting, even if
another node had activated it since. It now uses a guarded
creating-to-deleting transition, and the delete job refuses snapshots
that are not marked deleting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): keep snapshot IDs and descriptor sectors apart per node

In CI, two nodes creating snapshots at once still got the same ID even
under the registry lock, because the cross-node lock does not always
exclude them. A new ID is now a counter above the highest ID on disk,
times 64, plus the node slot. Each node fills descriptor sectors it
alone owns first, so two nodes neither reuse an ID nor overwrite each
other's sector. The registry test now runs with no lock at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): check IDs after older images and node slot reuse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): copy shared blocks before taking the inode lock

With #13's FS locks, a write into a shared block held the inode lock and
then needed the refcount and bitmap locks, which come first in lock
order, so it failed with EIO. The write now persists and drops the inode
lock, copies the block on its unlocked copy, then takes the lock and
keeps the copy only if no node wrote the inode meanwhile; otherwise it
frees the copy and retries. A conflict after an earlier mid-write commit
frees only the copies that commit did not map.

The in-process snapshot tests now run with a lock manager per FS, like
separate nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): failed final write after a mid-write copy commit

Covers #13's release of committed copies on a failed final inode write:
each old block loses the file's ownership once across the retry, and
uncounted blocks from an older version lose none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): log why a snapshot could not be added to the registry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): skip the two-node concurrent snapshot test in short mode

Two nodes can still both claim one free inode until lock claims are
confirmed (delta-lease), and creating 20 snapshots on cold lock records
across two nodes hits that often. The Docker and VM versions still run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(inode): reject stale copies whose inline extents changed on disk

The stale-write check compared only extent trees. A copy of an inline
inode read before a copy-on-write committed, such as a slow chmod on
another node, could write the old mapping back after the old block was
released, leaving the snapshot's block undercounted. A plain write now
fails with ErrTreeConflict when the inline extents on disk differ from
the ones the copy was read with; callers already retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): retry copy-on-write on a stale tree and finish interrupted snapshot calls

A copy-on-write on an unlocked inode copy can find the tree rewritten by
another node; it now retries instead of returning EIO. Snapshot and clone
xattrs span several writes, and go-fuse cancels the context when the
kernel interrupts a request, which made a contended registry lock fail
with EIO. They now run to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* docs: sync operator guides with current flags and behavior

OPERATIONS, DEPLOYMENT and TROUBLESHOOTING still used the old
goclusterfs binary name, a -format without -mount, a .snapshots
directory and snapshot API that never existed, lost+found, a
-fuse-debug flag, a mount -version flag, pprof, and read-only mounts.

They now match the code at e31fa92: format v2 and per-node journals,
-skip-recovery and damaged journal slots (#34), fsck exit codes and
-repair (#32), -attr-timeout (#24), lock renewal and takeover (#13,
#23), xattr snapshots and clones (#16), and the real log lines.

OPERATIONS.md gains a Security Model section: trust in the shared
device, single-user mounts with no permission checks, lock limits,
and unauthenticated broadcast unless -broadcast-secret is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: lock waits have no time limit

ErrLockTimeout only fires when the request context is cancelled;
lock.DefaultTimeout is never used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: make damaged-journal recovery safe and fix review findings

-skip-recovery discards the journal of whichever slot the mount takes,
which is not the damaged slot once a peer holds its recovery lease.
The steps now say when to run it and how to check the slot, and only
use it for a damaged header, not for I/O or replay errors.

Also quote the real legacy-layout and superblock errors, cover a v2
device with a wiped superblock, state that there is no HTTP API, and
correct peer-announce, replay, unicast fallback and cache lag claims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant