Skip to content

fix(lock): release a crashed node's locks once it is proven dead - #23

Merged
mattmattox merged 3 commits into
mainfrom
claude/reclaim-dead-node-locks
Oct 24, 2026
Merged

mattmattox merged 3 commits into
mainfrom
claude/reclaim-dead-node-locks

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

A crashed node's lock records stayed held until the 15 s staleness threshold, so recovery after a crash waited on them. TestJournalCrashDuringWrite failed 5/5 on ext4 images: mount-time replay waited 14.5 s for the root directory lock, past the 10 s mount timeout.

Exclusive locks. Once a node is proven dead, the node replaying its journal takes over the dead node's exclusive locks with lock.Manager.AdoptOwner, then frees them with ReleaseAdopted after the replay. Until then, no peer can lock the dead node's directories or inodes mid-replay, while the replaying node's own acquires reuse them. After the takeover, AdoptOwner re-reads the records 50 ms later to catch a racing stale takeover. This happens in two places:

  • mount, for the node's own dead previous incarnation, around its journal replay
  • failure detection, around the replay it wins the recovery lease for

Shared locks are left to the staleness rule. Their OwnerUUID names only the latest reader, who may already have released while others still read.

Proven dead means one of:

  • the mount process is gone (this host, same PID namespace)
  • the heartbeat is past the 20 s detection gate and this node's monotonic clock has seen it unchanged for 15 s

So a node paused for more than about 20 s counts as dead. It exits on its next heartbeat, and OPERATIONS.md says so.

Also in this PR:

  • Failure detection skips a slot whose heartbeat it saw advance within 10 s, unless the owner's process is gone.
  • A node whose slot is marked Dead, or whose heartbeat writes have failed for 5 s, stops.

Lock-review's delta-lease change (claim confirmation, caching) will rebase onto this.

Tests on .66 with ext4 images:

Test main (526d631) this PR
TestSurvivorReleasesCrashedNodeLocks: A crashes holding the root directory lock; B must create a file within 10 s fails (15.6 s) passes (5.3 s)
TestSlowNodeKeepsLocks: A paused 17 s, between the 15 s own-clock check and the 20 s gate; B must not take the lock while it is fresh, nor take over A's locks passes passes
TestJournalCrashDuringWrite ×5 0/5 5/5

Unit tests pass with -race. They cover adopt-and-release, a shared record whose latest reader already released, takeover only for proven-dead nodes (in order: adopt, replay, release), heartbeat write failure, and own-clock death. Integration passes: -short 78/78 and full 91/91. Lint is clean.

🤖 Generated with Claude Code

mattmattox and others added 3 commits October 24, 2026 15:30
Lock records of a crashed node stayed held until the 15s staleness
threshold. Replay after a crash therefore waited on locks the dead node
held: TestJournalCrashDuringWrite failed 5/5 on ext4 images because
mount-time replay waited 14.5s for the root directory lock, past the 10s
mount timeout.

lock.Manager.ReclaimOwner releases every record a given node holds.
internal/node calls it, through Registry.SetOwnerDeadHandler, only for a
node proven dead: its mount process is gone in this PID namespace on this
host, or its heartbeat has not changed for the heartbeat timeout plus 5s
by this node's monotonic clock. A slow node keeps its locks. Mount
releases a dead previous incarnation's locks before replaying its
journal, and failure detection does so after winning a slot's recovery
lease.

Failure detection also skips a slot whose heartbeat it saw advance within
the heartbeat timeout, whatever the wall-clock time in it says, and a node
whose own slot is marked Dead stops instead of journaling on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the first version found that freeing a dead node's locks was
unsafe for shared records and too early for exclusive ones:
- A shared record's OwnerUUID names only the latest reader, who may have
  released already, so dropping its count could take a live reader's.
  Shared records are now left to the staleness rule.
- Freeing exclusive records before the replay let peers lock the dead
  node's directories and inodes mid-replay. AdoptOwner now makes the
  replaying node their holder, re-reads them after 50ms to notice a racing
  takeover, and ReleaseAdopted frees them after the replay. The replaying
  node's own acquires reuse them.

A node whose heartbeat writes fail for 5s now stops, since peers would
soon prove it dead and take its locks. OPERATIONS.md now says a node paused
past about 20s counts as dead. TestSlowNodeKeepsLocks pauses 17s, between
the 15s own-clock check and the 20s detection gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit b56ff1f into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
…ad node's locks

#23 frees a crashed node's exclusive locks once it is proven dead, during the
journal replay a restarted node runs before it serves, so the restarted
mount no longer waits for its earlier incarnation's cached holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* docs: sync operator guides with current flags and behavior

OPERATIONS, DEPLOYMENT and TROUBLESHOOTING still used the old
goclusterfs binary name, a -format without -mount, a .snapshots
directory and snapshot API that never existed, lost+found, a
-fuse-debug flag, a mount -version flag, pprof, and read-only mounts.

They now match the code at e31fa92: format v2 and per-node journals,
-skip-recovery and damaged journal slots (#34), fsck exit codes and
-repair (#32), -attr-timeout (#24), lock renewal and takeover (#13,
#23), xattr snapshots and clones (#16), and the real log lines.

OPERATIONS.md gains a Security Model section: trust in the shared
device, single-user mounts with no permission checks, lock limits,
and unauthenticated broadcast unless -broadcast-secret is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: lock waits have no time limit

ErrLockTimeout only fires when the request context is cancelled;
lock.DefaultTimeout is never used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: make damaged-journal recovery safe and fix review findings

-skip-recovery discards the journal of whichever slot the mount takes,
which is not the damaged slot once a peer holds its recovery lease.
The steps now say when to run it and how to check the slot, and only
use it for a damaged header, not for I/O or replay errors.

Also quote the real legacy-layout and superblock errors, cover a v2
device with a wiped superblock, state that there is no HTTP API, and
correct peer-announce, replay, unicast fallback and cache lag claims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 13, 2026
#29)

* feat(lock): confirm claims after a settle wait and cache released holds

Taking a lock reads the record and then writes a claim, so two nodes that
read a free record at the same moment could both win. A node now waits
ClaimSettleDelay (20ms) after writing a claim or a renewal and re-reads the
record; it keeps the lock only if its claim is still there. A shared add on
a record that is already shared skips the wait: losing that race can only
over-count readers.

To keep the wait off hot paths, a node keeps a record for CacheGrace (250ms)
after its last local holder releases it, so the next local acquire needs no
disk I/O. A cached hold that is too weak for the request, lost, or older than
renewAfter is released and acquired afresh. FS.Close releases cached holds.
RetryMaxDelay drops to 100ms so waiters notice a released hold soon.

Writes of whole 4KB sectors to an allocated, written, unshared block no longer
take the per-block extent lock or read the sector first.

FUSE cancels a request's context whenever the caller gets a signal; Go
callers get SIGURG constantly, so lock waits now end only on a deadline or
after DefaultTimeout. Torn reads of a record another node is writing are
read again before the record is reported invalid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): document the lock settle wait, cache grace and I/O limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): benchmark an acquire that cannot reuse a cached hold

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): let a node take the locks its registry slot's previous owner held

With cached holds, a node that crashes leaves every record it used in the
last 250ms held, so its restarted mount waited 15s for its own earlier
incarnation's locks. A record held exclusively under this node's registry
slot by another UUID belongs to an earlier occupant of the slot, which the
registry hands out only after that occupant failed, so it is taken at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): yield records a busy node has held past the cache grace

A node that kept reusing a record rejoined its cached hold before the grace
ran out, so it never released it, and another node waited as long as the
first stayed busy: a restarted mount waited while a peer kept creating
files. A hold taken from disk more than CacheGrace ago is now released when
its last local key drops, and this node waits YieldGap before claiming it
again. RetryMaxDelay drops to 25ms so a waiter polls within that gap.

A block allocated by the current write stays invisible to other nodes until
the inode, whose lock the write holds, is committed, so writing it no
longer takes the per-block extent lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: un-skip the first-write test and check readers during allocating writes

Claim confirmation stops two nodes from both taking a free inode lock, so the
first-write test runs in short mode again. A new test has one node read a file
while another fills it with fresh blocks, which it writes without the extent
lock; the reader must see only holes, the old size or each block's own data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fs): check a reader on another node never sees uncommitted blocks

The mounted-FS version could not see the file grow on the second node: its
inode stays cached while the file is open when the cache broadcast is not
running. Two FS instances on one image, with their own lock managers on a
shared lock region and no inode cache, read the on-disk state every time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(lock): drop the slot-owner takeover now that #23 adopts a dead node's locks

#23 frees a crashed node's exclusive locks once it is proven dead, during the
journal replay a restarted node runs before it serves, so the restarted
mount no longer waits for its earlier incarnation's cached holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): keep shared claims that other readers join, drop slow claim writes

A node's first shared claim failed its settle check when another node joined
it during the wait, then re-joined on top, leaking one reader count until the
lock went stale. A shared claim now also holds when the record is still shared
at a newer sequence.

A claim whose write would land more than half the settle delay after its read
is dropped and retried, so a slow or queued write cannot overwrite a claim
another node has already confirmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(goclusterfs): name the extent-lock skip condition

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): check claim write deadlines after the shard wait, bound record retries

The 10ms check on a claim write ran before WriteAt took the device's I/O
shard mutex, so a wait there went unmeasured. storage.Device.WriteAtBefore
now checks the deadline once it holds the shard, right before the write. A
write that still finishes more than 10ms after its read is not held, and
the node leaves that record alone until it is stale, because the write may
have replaced a claim another node already confirmed.

Releases, renewals and dead-node takeovers no longer retry forever on
storage that keeps missing the deadline: they back off with jitter and give
up after 10 tries.

Adds the PM's repro tests: two nodes reading one directory, or the
refcounts, at once must not delay the next create, unlink, snapshot create,
copy-on-write or snapshot delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(ops): note late read claims beside a writer and the cost of a late lock write

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): close late read claims to other readers, free kept locks in parallel

A read claim whose write lands late can sit on top of a write claim another
node has just confirmed. Left as a read claim, it let further readers in
beside that writer. The node now rewrites it as its own write claim before
leaving the record to go stale.

ReleaseCached frees kept records eight at a time, so an unmount on slow
storage no longer pays each record's retries one after another.

The two CAS test hooks become one injectable record write, so tests can
model a write that passes its deadline check and then lands late.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): write the late read claim fence without a deadline

The fence that turns a late read claim into this node's write claim used the
normal 10ms write deadline, so on still-slow storage it was skipped and the
record stayed open to readers beside a writer. The fence can only make the
record exclusive to this node, so it now writes without a deadline. It
fences the record as it is now, so readers that joined before it are shut
out from then on too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(storage): check deferred close and remove errors like the rest of the file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): bound the late read claim fence to half the stale threshold

An unbounded fence write that stalled past StaleThreshold could land on a
claim another node made after taking the record over as stale. The fence
now skips a record that is already stale, and its write must start within
fenceWindow (StaleThreshold/2) of its read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): keep unit tests off the 10ms record write limit unless they test it

On the CI runner's disk a 4KB O_DIRECT lock record write took 25-27ms, so
every claim counted as late, froze its record for 15s, and 12 tests timed
out. Each Manager now carries its write window. The lock tests' TestMain
relaxes the default, and the tests of the window itself restore the real
10ms on the nodes whose writes they slow down. CI also runs unit test
images on tmpfs, checking first that tmpfs takes O_DIRECT writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run concurrent snapshot creates again now that lock claims are confirmed

#33 skipped them in Docker, VM and Go -short because two nodes could both
claim one free inode's lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): run concurrent file creation in -short mode

Its skip blamed a race in the distributed lock. Two nodes allocating inodes
and blocks at once go through the lock claims #29 now confirms, and the
test passed 23 of 23 runs on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lock): log who holds the record when an acquire gives up

Operators need to see which node holds a lock that made an operation fail.
The timeout now logs at WARN with the lock key and the record's holder,
mode and age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run snapshots during writes again now that lock claims are confirmed

#42 skipped it in Go, and its Docker and VM versions, because two nodes could
both hold the refcount lock. The merge resolution re-enables the shell ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(goclusterfs): retry a write up to 10 times when other nodes change the file

A node writing a file while another node snapshots it back to back loses
the copy-on-write check each time a snapshot lands. With the 20ms lock
claim wait those windows are wider, and 3 tries made TestSnapshotsDuringWritesAcrossNodes
fail 4 of 10 runs with EAGAIN. With 10 tries it passed 20 of 20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run integration test images on tmpfs and allow 8 minutes

Lock record writes on the runner's disk can miss their 10ms limit and stall
a lock for 15s, which pushed the -short suite past its 5 minute timeout.
Main's suite already takes about 4 minutes, and lock claims now wait 20ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lock): measure where the time of late lock record writes goes

Each skipped or late record write now logs its time split into waits inside
this process (the device's shard lock, the gap between the read and the write
call) and the read and write system calls. Each node counts them and logs the
totals at unmount. CI integration tests go back to the runner's own disk, so
these numbers show what users on ordinary disks would see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lock): log acquire counts per lock type and a write gap histogram

At unmount each node logs, per lock type, how its acquires were served
(held key, joined record, cached record, fresh claim with or without the
settle wait, blocked tries), and a histogram of the time from a record
CAS's read to its write finishing. These are the numbers ClaimSettleDelay
and CacheGrace are chosen from. Docker CI keeps each node's lock log lines
with its test results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): read claims skip the settle wait and are not kept after release

A node killed within 250ms of a read left its cached read hold on disk, and
other nodes' writers waited 15s for it to go stale (TestNetworkPartition:
3s on main, 17s here). Read claims now release as soon as their last local
reader does, and they no longer wait out ClaimSettleDelay. Only write claims
need that wait: every record write still finishes within half the settle
delay of its read, so a read claim that replaced a writer's claim lands
before the writer's confirm re-read, and the writer backs off. A read can
still run beside a writer that won the race, as on main; two writers cannot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): give up a claim write only if it finished after the settle delay

A claim write can replace a claim another node already confirmed only if it
lands more than ClaimSettleDelay after its own read: until then, the other
node's confirm re-read is still ahead. The 15s give-up now applies past the
full delay, not half of it, and the margin moves to the start: a write that
cannot start within a quarter of the delay is skipped and retried, which is
cheap. On CI, both late writes in run 42863343192/42863342671 (10ms, 16.5ms)
would have been held.

Lock record reads and writes also stop taking the device's shard lock, so a
node's own data and journal I/O cannot delay them: 2 of the 6 missed writes
on CI waited about 10ms for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): name readers with per-node bits and confirm read claims

A read lock's holder count was updated by read-modify-write from several
nodes, so racing claims and releases lost updates. A lost release left the
count too high, and writers waited 15s for the record to go stale
(TestMultiNodeConcurrentDirectoryOps, ...CreatesDoNotCollide on 9af90ae).

Each read-locked record now has a bit per registry slot in its reserved
bytes. A read claim sets this node's bit and waits out ClaimSettleDelay
like a write claim, then keeps the lock only if its bit is still set. A
release clears the bit and checks again in the background that no racing
update put it back. Read holds are cached again. AdoptOwner, given the dead
node's registry slot, clears its reader bits, so a reader killed while it
kept a hold blocks writers only until the node is proven dead. Every claim
writes the current time, since readers' bits now clear on their own.

Late claims (a claim write that finished after the settle delay) are logged
with their lock key and latency and counted at unmount.

Lock records change meaning in the reserved bytes (bits 64:72), which goes
with on-disk format v3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: say 20ms in the unit job's tmpfs comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): stop waiting on lock hint counters and retry reads past attr timeout

The lock hint cache is gone, so its counters stay 0. A restarted node can
cache the size of a peer's file it looked up mid-write for -attr-timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): tie the verifyFiles retry to the node's -attr-timeout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): keep idle holds 10ms and cap busy holds with a separate 250ms turn

An idle cached hold kept other nodes waiting up to 250ms. With the root
directory and the block bitmap both hot, two nodes waited on each other's
idle holds and concurrent creates ran 2.7x slower than main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: add the idle cached hold test to the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): address the #29 review

- Whole-sector writes to existing blocks take the extent lock again. A peer's
  read-modify-write of the same sector could otherwise write back a stale copy
  over an acknowledged write.
- Once a hold's turn is over, new local requests wait for the node's holders
  to drain, so a record whose holds keep overlapping is still yielded. Only a
  goroutine that holds other locks stops waiting after another turn.
- A record adopted in this node's own slot keeps its adopted write claim
  through the replay's own acquires and releases until ReleaseAdopted.
- Adopted claims are flagged, and every claimant defers to them, so a waiter
  in a lower slot cannot keep the replaying node out.
- ReleaseAdopted leaves a slot that a new node has written since.
- A slot that stays torn for StaleThreshold stops blocking its record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(goclusterfs): run the whole-sector lost-update test with a full lock region

With the 1MB test region the two writes' other locks share one of 4 records,
which serializes them and hides the lost update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: make the lost-update and overlapping-hold tests catch their bugs

The whole-sector test raced two writers through a gate, and on fast storage
the partial write finished before the other woke, so it passed with the bug.
A hook after the partial write's sector read now runs the whole-sector write
while the old sector is held. The overlapping-hold test gets a case with
holds longer than a turn, which only an unbounded drain wait passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): expire a torn slot only while it stays torn with the same bytes

A node kept the time it first read a slot torn until it read that slot whole
again. Long after it stopped reading the record, a torn read of a live node's
renewal in flight looked expired, and both nodes held the record. The wait now
restarts whenever the torn bytes change. They are compared by FNV-1a, since
CRC32C gives every slot torn the same way one value.

OPERATIONS.md no longer says whole-sector writes skip the block lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(goclusterfs): take the inode lock exclusive when the block's lock shares its record

A write into an existing block holds the inode lock shared and then takes the
block's extent lock exclusive. When both keys hash to one lock record, the lock
manager sees a shared-to-exclusive upgrade and the write fails with EIO. Main
hits this only on partial-sector writes; #29 makes every write to an existing
block take the extent lock, so the Docker "Snapshot During Writes" test failed
when inode 130 and block 163 both mapped to record 36.

The Docker test now prints dd's error and the node's write errors when the
writer fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant