Repository navigation
fix(lock): release a crashed node's locks once it is proven dead - #23
Merged
Merged
Conversation
Lock records of a crashed node stayed held until the 15s staleness threshold. Replay after a crash therefore waited on locks the dead node held: TestJournalCrashDuringWrite failed 5/5 on ext4 images because mount-time replay waited 14.5s for the root directory lock, past the 10s mount timeout. lock.Manager.ReclaimOwner releases every record a given node holds. internal/node calls it, through Registry.SetOwnerDeadHandler, only for a node proven dead: its mount process is gone in this PID namespace on this host, or its heartbeat has not changed for the heartbeat timeout plus 5s by this node's monotonic clock. A slow node keeps its locks. Mount releases a dead previous incarnation's locks before replaying its journal, and failure detection does so after winning a slot's recovery lease. Failure detection also skips a slot whose heartbeat it saw advance within the heartbeat timeout, whatever the wall-clock time in it says, and a node whose own slot is marked Dead stops instead of journaling on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the first version found that freeing a dead node's locks was unsafe for shared records and too early for exclusive ones: - A shared record's OwnerUUID names only the latest reader, who may have released already, so dropping its count could take a live reader's. Shared records are now left to the staleness rule. - Freeing exclusive records before the replay let peers lock the dead node's directories and inodes mid-replay. AdoptOwner now makes the replaying node their holder, re-reads them after 50ms to notice a racing takeover, and ReleaseAdopted frees them after the replay. The replaying node's own acquires reuse them. A node whose heartbeat writes fail for 5s now stops, since peers would soon prove it dead and take its locks. OPERATIONS.md now says a node paused past about 20s counts as dead. TestSlowNodeKeepsLocks pauses 17s, between the 15s own-clock check and the 20s detection gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
…ad node's locks #23 frees a crashed node's exclusive locks once it is proven dead, during the journal replay a restarted node runs before it serves, so the restarted mount no longer waits for its earlier incarnation's cached holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
* docs: sync operator guides with current flags and behavior OPERATIONS, DEPLOYMENT and TROUBLESHOOTING still used the old goclusterfs binary name, a -format without -mount, a .snapshots directory and snapshot API that never existed, lost+found, a -fuse-debug flag, a mount -version flag, pprof, and read-only mounts. They now match the code at e31fa92: format v2 and per-node journals, -skip-recovery and damaged journal slots (#34), fsck exit codes and -repair (#32), -attr-timeout (#24), lock renewal and takeover (#13, #23), xattr snapshots and clones (#16), and the real log lines. OPERATIONS.md gains a Security Model section: trust in the shared device, single-user mounts with no permission checks, lock limits, and unauthenticated broadcast unless -broadcast-secret is set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: lock waits have no time limit ErrLockTimeout only fires when the request context is cancelled; lock.DefaultTimeout is never used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: make damaged-journal recovery safe and fix review findings -skip-recovery discards the journal of whichever slot the mount takes, which is not the damaged slot once a peer holds its recovery lease. The steps now say when to run it and how to check the slot, and only use it for a damaged header, not for I/O or replay errors. Also quote the real legacy-layout and superblock errors, cover a v2 device with a wiped superblock, state that there is no HTTP API, and correct peer-announce, replay, unicast fallback and cache lag claims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
#29) * feat(lock): confirm claims after a settle wait and cache released holds Taking a lock reads the record and then writes a claim, so two nodes that read a free record at the same moment could both win. A node now waits ClaimSettleDelay (20ms) after writing a claim or a renewal and re-reads the record; it keeps the lock only if its claim is still there. A shared add on a record that is already shared skips the wait: losing that race can only over-count readers. To keep the wait off hot paths, a node keeps a record for CacheGrace (250ms) after its last local holder releases it, so the next local acquire needs no disk I/O. A cached hold that is too weak for the request, lost, or older than renewAfter is released and acquired afresh. FS.Close releases cached holds. RetryMaxDelay drops to 100ms so waiters notice a released hold soon. Writes of whole 4KB sectors to an allocated, written, unshared block no longer take the per-block extent lock or read the sector first. FUSE cancels a request's context whenever the caller gets a signal; Go callers get SIGURG constantly, so lock waits now end only on a deadline or after DefaultTimeout. Torn reads of a record another node is writing are read again before the record is reported invalid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(operations): document the lock settle wait, cache grace and I/O limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(lock): benchmark an acquire that cannot reuse a cached hold Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): let a node take the locks its registry slot's previous owner held With cached holds, a node that crashes leaves every record it used in the last 250ms held, so its restarted mount waited 15s for its own earlier incarnation's locks. A record held exclusively under this node's registry slot by another UUID belongs to an earlier occupant of the slot, which the registry hands out only after that occupant failed, so it is taken at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): yield records a busy node has held past the cache grace A node that kept reusing a record rejoined its cached hold before the grace ran out, so it never released it, and another node waited as long as the first stayed busy: a restarted mount waited while a peer kept creating files. A hold taken from disk more than CacheGrace ago is now released when its last local key drops, and this node waits YieldGap before claiming it again. RetryMaxDelay drops to 25ms so a waiter polls within that gap. A block allocated by the current write stays invisible to other nodes until the inode, whose lock the write holds, is committed, so writing it no longer takes the per-block extent lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: un-skip the first-write test and check readers during allocating writes Claim confirmation stops two nodes from both taking a free inode lock, so the first-write test runs in short mode again. A new test has one node read a file while another fills it with fresh blocks, which it writes without the extent lock; the reader must see only holes, the old size or each block's own data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fs): check a reader on another node never sees uncommitted blocks The mounted-FS version could not see the file grow on the second node: its inode stays cached while the file is open when the cache broadcast is not running. Two FS instances on one image, with their own lock managers on a shared lock region and no inode cache, read the on-disk state every time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(lock): drop the slot-owner takeover now that #23 adopts a dead node's locks #23 frees a crashed node's exclusive locks once it is proven dead, during the journal replay a restarted node runs before it serves, so the restarted mount no longer waits for its earlier incarnation's cached holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): keep shared claims that other readers join, drop slow claim writes A node's first shared claim failed its settle check when another node joined it during the wait, then re-joined on top, leaking one reader count until the lock went stale. A shared claim now also holds when the record is still shared at a newer sequence. A claim whose write would land more than half the settle delay after its read is dropped and retried, so a slow or queued write cannot overwrite a claim another node has already confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(goclusterfs): name the extent-lock skip condition Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): check claim write deadlines after the shard wait, bound record retries The 10ms check on a claim write ran before WriteAt took the device's I/O shard mutex, so a wait there went unmeasured. storage.Device.WriteAtBefore now checks the deadline once it holds the shard, right before the write. A write that still finishes more than 10ms after its read is not held, and the node leaves that record alone until it is stale, because the write may have replaced a claim another node already confirmed. Releases, renewals and dead-node takeovers no longer retry forever on storage that keeps missing the deadline: they back off with jitter and give up after 10 tries. Adds the PM's repro tests: two nodes reading one directory, or the refcounts, at once must not delay the next create, unlink, snapshot create, copy-on-write or snapshot delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(ops): note late read claims beside a writer and the cost of a late lock write Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): close late read claims to other readers, free kept locks in parallel A read claim whose write lands late can sit on top of a write claim another node has just confirmed. Left as a read claim, it let further readers in beside that writer. The node now rewrites it as its own write claim before leaving the record to go stale. ReleaseCached frees kept records eight at a time, so an unmount on slow storage no longer pays each record's retries one after another. The two CAS test hooks become one injectable record write, so tests can model a write that passes its deadline check and then lands late. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): write the late read claim fence without a deadline The fence that turns a late read claim into this node's write claim used the normal 10ms write deadline, so on still-slow storage it was skipped and the record stayed open to readers beside a writer. The fence can only make the record exclusive to this node, so it now writes without a deadline. It fences the record as it is now, so readers that joined before it are shut out from then on too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(storage): check deferred close and remove errors like the rest of the file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): bound the late read claim fence to half the stale threshold An unbounded fence write that stalled past StaleThreshold could land on a claim another node made after taking the record over as stale. The fence now skips a record that is already stale, and its write must start within fenceWindow (StaleThreshold/2) of its read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(lock): keep unit tests off the 10ms record write limit unless they test it On the CI runner's disk a 4KB O_DIRECT lock record write took 25-27ms, so every claim counted as late, froze its record for 15s, and 12 tests timed out. Each Manager now carries its write window. The lock tests' TestMain relaxes the default, and the tests of the window itself restore the real 10ms on the nodes whose writes they slow down. CI also runs unit test images on tmpfs, checking first that tmpfs takes O_DIRECT writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run concurrent snapshot creates again now that lock claims are confirmed #33 skipped them in Docker, VM and Go -short because two nodes could both claim one free inode's lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): run concurrent file creation in -short mode Its skip blamed a race in the distributed lock. Two nodes allocating inodes and blocks at once go through the lock claims #29 now confirms, and the test passed 23 of 23 runs on this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lock): log who holds the record when an acquire gives up Operators need to see which node holds a lock that made an operation fail. The timeout now logs at WARN with the lock key and the record's holder, mode and age. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run snapshots during writes again now that lock claims are confirmed #42 skipped it in Go, and its Docker and VM versions, because two nodes could both hold the refcount lock. The merge resolution re-enables the shell ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(goclusterfs): retry a write up to 10 times when other nodes change the file A node writing a file while another node snapshots it back to back loses the copy-on-write check each time a snapshot lands. With the 20ms lock claim wait those windows are wider, and 3 tries made TestSnapshotsDuringWritesAcrossNodes fail 4 of 10 runs with EAGAIN. With 10 tries it passed 20 of 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run integration test images on tmpfs and allow 8 minutes Lock record writes on the runner's disk can miss their 10ms limit and stall a lock for 15s, which pushed the -short suite past its 5 minute timeout. Main's suite already takes about 4 minutes, and lock claims now wait 20ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lock): measure where the time of late lock record writes goes Each skipped or late record write now logs its time split into waits inside this process (the device's shard lock, the gap between the read and the write call) and the read and write system calls. Each node counts them and logs the totals at unmount. CI integration tests go back to the runner's own disk, so these numbers show what users on ordinary disks would see. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lock): log acquire counts per lock type and a write gap histogram At unmount each node logs, per lock type, how its acquires were served (held key, joined record, cached record, fresh claim with or without the settle wait, blocked tries), and a histogram of the time from a record CAS's read to its write finishing. These are the numbers ClaimSettleDelay and CacheGrace are chosen from. Docker CI keeps each node's lock log lines with its test results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): read claims skip the settle wait and are not kept after release A node killed within 250ms of a read left its cached read hold on disk, and other nodes' writers waited 15s for it to go stale (TestNetworkPartition: 3s on main, 17s here). Read claims now release as soon as their last local reader does, and they no longer wait out ClaimSettleDelay. Only write claims need that wait: every record write still finishes within half the settle delay of its read, so a read claim that replaced a writer's claim lands before the writer's confirm re-read, and the writer backs off. A read can still run beside a writer that won the race, as on main; two writers cannot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): give up a claim write only if it finished after the settle delay A claim write can replace a claim another node already confirmed only if it lands more than ClaimSettleDelay after its own read: until then, the other node's confirm re-read is still ahead. The 15s give-up now applies past the full delay, not half of it, and the margin moves to the start: a write that cannot start within a quarter of the delay is skipped and retried, which is cheap. On CI, both late writes in run 42863343192/42863342671 (10ms, 16.5ms) would have been held. Lock record reads and writes also stop taking the device's shard lock, so a node's own data and journal I/O cannot delay them: 2 of the 6 missed writes on CI waited about 10ms for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): name readers with per-node bits and confirm read claims A read lock's holder count was updated by read-modify-write from several nodes, so racing claims and releases lost updates. A lost release left the count too high, and writers waited 15s for the record to go stale (TestMultiNodeConcurrentDirectoryOps, ...CreatesDoNotCollide on 9af90ae). Each read-locked record now has a bit per registry slot in its reserved bytes. A read claim sets this node's bit and waits out ClaimSettleDelay like a write claim, then keeps the lock only if its bit is still set. A release clears the bit and checks again in the background that no racing update put it back. Read holds are cached again. AdoptOwner, given the dead node's registry slot, clears its reader bits, so a reader killed while it kept a hold blocks writers only until the node is proven dead. Every claim writes the current time, since readers' bits now clear on their own. Late claims (a claim write that finished after the settle delay) are logged with their lock key and latency and counted at unmount. Lock records change meaning in the reserved bytes (bits 64:72), which goes with on-disk format v3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: say 20ms in the unit job's tmpfs comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): stop waiting on lock hint counters and retry reads past attr timeout The lock hint cache is gone, so its counters stay 0. A restarted node can cache the size of a peer's file it looked up mid-write for -attr-timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): tie the verifyFiles retry to the node's -attr-timeout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): keep idle holds 10ms and cap busy holds with a separate 250ms turn An idle cached hold kept other nodes waiting up to 250ms. With the root directory and the block bitmap both hot, two nodes waited on each other's idle holds and concurrent creates ran 2.7x slower than main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: add the idle cached hold test to the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): address the #29 review - Whole-sector writes to existing blocks take the extent lock again. A peer's read-modify-write of the same sector could otherwise write back a stale copy over an acknowledged write. - Once a hold's turn is over, new local requests wait for the node's holders to drain, so a record whose holds keep overlapping is still yielded. Only a goroutine that holds other locks stops waiting after another turn. - A record adopted in this node's own slot keeps its adopted write claim through the replay's own acquires and releases until ReleaseAdopted. - Adopted claims are flagged, and every claimant defers to them, so a waiter in a lower slot cannot keep the replaying node out. - ReleaseAdopted leaves a slot that a new node has written since. - A slot that stays torn for StaleThreshold stops blocking its record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(goclusterfs): run the whole-sector lost-update test with a full lock region With the 1MB test region the two writes' other locks share one of 4 records, which serializes them and hides the lost update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: make the lost-update and overlapping-hold tests catch their bugs The whole-sector test raced two writers through a gate, and on fast storage the partial write finished before the other woke, so it passed with the bug. A hook after the partial write's sector read now runs the whole-sector write while the old sector is held. The overlapping-hold test gets a case with holds longer than a turn, which only an unbounded drain wait passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): expire a torn slot only while it stays torn with the same bytes A node kept the time it first read a slot torn until it read that slot whole again. Long after it stopped reading the record, a torn read of a live node's renewal in flight looked expired, and both nodes held the record. The wait now restarts whenever the torn bytes change. They are compared by FNV-1a, since CRC32C gives every slot torn the same way one value. OPERATIONS.md no longer says whole-sector writes skip the block lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(goclusterfs): take the inode lock exclusive when the block's lock shares its record A write into an existing block holds the inode lock shared and then takes the block's extent lock exclusive. When both keys hash to one lock record, the lock manager sees a shared-to-exclusive upgrade and the write fails with EIO. Main hits this only on partial-sector writes; #29 makes every write to an existing block take the extent lock, so the Docker "Snapshot During Writes" test failed when inode 130 and block 163 both mapped to record 36. The Docker test now prints dd's error and the node's write errors when the writer fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A crashed node's lock records stayed held until the 15 s staleness threshold, so recovery after a crash waited on them.
TestJournalCrashDuringWritefailed 5/5 on ext4 images: mount-time replay waited 14.5 s for the root directory lock, past the 10 s mount timeout.Exclusive locks. Once a node is proven dead, the node replaying its journal takes over the dead node's exclusive locks with
lock.Manager.AdoptOwner, then frees them withReleaseAdoptedafter the replay. Until then, no peer can lock the dead node's directories or inodes mid-replay, while the replaying node's own acquires reuse them. After the takeover,AdoptOwnerre-reads the records 50 ms later to catch a racing stale takeover. This happens in two places:Shared locks are left to the staleness rule. Their
OwnerUUIDnames only the latest reader, who may already have released while others still read.Proven dead means one of:
So a node paused for more than about 20 s counts as dead. It exits on its next heartbeat, and OPERATIONS.md says so.
Also in this PR:
Lock-review's delta-lease change (claim confirmation, caching) will rebase onto this.
Tests on .66 with ext4 images:
TestSurvivorReleasesCrashedNodeLocks: A crashes holding the root directory lock; B must create a file within 10 sTestSlowNodeKeepsLocks: A paused 17 s, between the 15 s own-clock check and the 20 s gate; B must not take the lock while it is fresh, nor take over A's locksTestJournalCrashDuringWrite×5Unit tests pass with
-race. They cover adopt-and-release, a shared record whose latest reader already released, takeover only for proven-dead nodes (in order: adopt, replay, release), heartbeat write failure, and own-clock death. Integration passes:-short78/78 and full 91/91. Lint is clean.🤖 Generated with Claude Code