Repository navigation
fix(lock): take over a lock only after its holder is released - #90
Merged
Merged
Conversation
Other nodes ignored a claim once it was 15s old, so a paused or stalled holder lost its locks while it could still write, and a waiter got EIO after 30s even while the holder was failing over. A claim in another node's slot now counts until the registry slot with the same number reads empty. A wait for a failing holder has no time limit: it ends when the holder's replay does, and fails only once that replay's lease lapses. A goroutine that already holds a lock keeps the 30s limit, because the replay may need that lock; the replay's own waits have no limit. Failure detection replays only nodes proven dead and takes over their locks first. A failed replay keeps them held until one succeeds. A mount that takes a failed node's slot keeps that node's claims through its replay, from any host, and retries a transaction it cannot apply yet. Two replays sharing a record take turns. A node warns about each lock it has held for over a minute, since a hold that is never released now lasts until unmount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a lock The mount retried any failure to apply its own journal, so a record that does not decode or an I/O error retried forever instead of failing with the -skip-recovery hint. It now retries only a replay that gave up waiting for another node's lock, and a lock timeout names the holder, so each retry logs who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
…leftover claims A node replaying a dead node passed over every adopted claim in a higher slot, including a live node's claims while it replays its own slot at mount, so both could hold a record. It now passes over only adopted claims in a slot marked dead, which the registry now reports apart from a stale heartbeat. Claims an earlier occupant left in a slot that no replay adopted, for example after fsck released the slot, counted as the new occupant's. A mount now clears the claims in its slot that it did not write after its replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s match recoverOwnJournal returned the fatal message itself, which started with a lowercase "journal". Mount prints the original line again, with the -skip-recovery hint, and the helper returns the replay error as is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
Since #90 no node takes over a live node's lock. Release returns ErrNotLockOwner only when this node's own bookkeeping lacks the lock, a double release that may have dropped another holder's lock early. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
… there (#95) * fix: check ignored errors in lock, inode and extent; enforce errcheck there - inode table: a failed lock release after an inode or sector update was dropped. If another node had taken the lock over, both nodes may have written the same inode table sector. Now logged as an error with that reason; other release failures log a warning. - extent: when allocate or MapCopy fail, the blocks they took are freed again. A failed free left them allocated with no owner and no trace; the returned error now names each one. - lock.WithInodeLock now returns the release error. - Tests check setup steps and lock releases; lock tests use a release helper that reports failures with Errorf, so goroutines can use it. errcheck is now enforced in lock, inode and extent too; 438 findings remain elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(inode): name the real cause of ErrNotLockOwner on release Since #90 no node takes over a live node's lock. Release returns ErrNotLockOwner only when this node's own bookkeeping lacks the lock, a double release that may have dropped another holder's lock early. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of the lock follow-ups. #83 (background renewal) is closed: under this PR a claim never goes stale, so renewing it protects nothing.
Before this PR, a claim older than 15s stopped counting. So a stalled holder lost its locks while it could still write, and a waiter got EIO after 30s even during a failover.
-skip-recoveryhint.Failover times: 4.9s for a crash that a survivor on the same host can see. 34.9s, with no EIO, in the new integration test, where node A holds a lock alive for 13s and then stops. Without this PR that test fails, because B takes A's lock at 15s, before A is proven dead.
Behavior change: after a whole-cluster crash, the first mount waits up to about 20s for locks the dead nodes held, while it proves them dead.
Tests: lock tests against a fake registry, including the shepherd's replay-vs-peer example; registry tests; mount tests for a lock-wait retry and for a bad record that stops at once; and the integration test. Each rule has a mutation that fails its test. On .68, the unit suite passes, lock, node and journal pass under -race, and the lock reclaim integration tests pass.
🤖 Generated with Claude Code