Repository navigation
fix(goclusterfs): let an overwrite step aside for a failed node's replay - #93
Merged
Merged
Conversation
Other nodes ignored a claim once it was 15s old, so a paused or stalled holder lost its locks while it could still write, and a waiter got EIO after 30s even while the holder was failing over. A claim in another node's slot now counts until the registry slot with the same number reads empty. A wait for a failing holder has no time limit: it ends when the holder's replay does, and fails only once that replay's lease lapses. A goroutine that already holds a lock keeps the 30s limit, because the replay may need that lock; the replay's own waits have no limit. Failure detection replays only nodes proven dead and takes over their locks first. A failed replay keeps them held until one succeeds. A mount that takes a failed node's slot keeps that node's claims through its replay, from any host, and retries a transaction it cannot apply yet. Two replays sharing a record take turns. A node warns about each lock it has held for over a minute, since a hold that is never released now lasts until unmount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a lock The mount retried any failure to apply its own journal, so a record that does not decode or an I/O error retried forever instead of failing with the -skip-recovery hint. It now retries only a replay that gave up waiting for another node's lock, and a lock timeout names the holder, so each retry logs who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A write that held a file's inode lock and waited for a block lock a failed node held got EIO after 30s, so the replay could take the inode lock. An overwrite holding only the shared inode lock it took for that block has changed nothing a retry would not redo. It now gets ErrReplayBlocked at once, lets go of the inode lock, waits for the block holding no lock, and retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leftover claims A node replaying a dead node passed over every adopted claim in a higher slot, including a live node's claims while it replays its own slot at mount, so both could hold a record. It now passes over only adopted claims in a slot marked dead, which the registry now reports apart from a stale heartbeat. Claims an earlier occupant left in a slot that no replay adopted, for example after fsck released the slot, counted as the new occupant's. A mount now clears the claims in its slot that it did not write after its replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry-after-replay # Conflicts: # internal/lock/lock.go # internal/lock/owner_test.go
…s match recoverOwnJournal returned the fatal message itself, which started with a lowercase "journal". Mount prints the original line again, with the -skip-recovery hint, and the helper returns the replay error as is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry-after-replay
…ter-replay # Conflicts: # CHANGELOG.md # docs/OPERATIONS.md # internal/lock/lock.go # internal/lock/owner.go # internal/lock/owner_test.go # tests/test-manifest.yaml
mattmattox
changed the base branch from
claude/lock-takeover-after-release
to
main
November 13, 2026 19:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #90 the user asked for, stacked on it. Review the last commit.
Under #90, a goroutine that holds a lock and waits for a failed node's lock gets EIO after 30s, so the failed node's replay can take the lock the goroutine holds. This PR lets the most common case retry instead of failing.
lock.Retryable(ctx)marks a caller that can drop its locks and retry. Such a caller getsErrReplayBlockedat once, so the replay doesn't wait 30s for it.ErrReplayBlockedis anErrLockTimeout, so other callers handle it as before.Which ops qualify:
Write)errRetryWriteloop.Writethat allocated or copied blocks earlier in the same callTests: a lock test (the error comes at once and is an
ErrLockTimeout) and a filesystem test. In the filesystem test, the writer waits with no locks held, a replay takes the file's inode lock, and the write then completes. Each fails with its piece removed. On .68 the full unit suite passes, and the new paths pass under -race.🤖 Generated with Claude Code