Skip to content

fix(goclusterfs): let an overwrite step aside for a failed node's replay - #93

Merged
mattmattox merged 9 commits into
mainfrom
claude/write-retry-after-replay
Nov 13, 2026
Merged

mattmattox merged 9 commits into
mainfrom
claude/write-retry-after-replay

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Follow-up to #90 the user asked for, stacked on it. Review the last commit.

Under #90, a goroutine that holds a lock and waits for a failed node's lock gets EIO after 30s, so the failed node's replay can take the lock the goroutine holds. This PR lets the most common case retry instead of failing.

lock.Retryable(ctx) marks a caller that can drop its locks and retry. Such a caller gets ErrReplayBlocked at once, so the replay doesn't wait 30s for it. ErrReplayBlocked is an ErrLockTimeout, so other callers handle it as before.

Which ops qualify:

Operation Nested wait Retries?
Overwrite of an existing block (in-place Write) shared inode lock, then the block's extent lock Yes. It has changed nothing yet. It drops the inode lock, waits for the block holding nothing, then retries through the existing errRetryWrite loop.
Write that allocated or copied blocks earlier in the same call exclusive inode lock with unsaved changes, then an extent lock No, EIO after 30s. Its changes aren't on disk yet.
Create, mkdir, unlink, setattr, truncate, snapshot and clone claims an inode lock, then the inode table sector lock No, EIO after 30s. They may already have changed a directory or another inode. The sector lock is also taken without the caller's context.
The one bitmap-then-inode nesting measured in #85's work bitmap, then inode No.

Tests: a lock test (the error comes at once and is an ErrLockTimeout) and a filesystem test. In the filesystem test, the writer waits with no locks held, a replay takes the file's inode lock, and the write then completes. Each fails with its piece removed. On .68 the full unit suite passes, and the new paths pass under -race.

🤖 Generated with Claude Code

mattmattox and others added 8 commits November 13, 2026 12:06
Other nodes ignored a claim once it was 15s old, so a paused or stalled
holder lost its locks while it could still write, and a waiter got EIO
after 30s even while the holder was failing over. A claim in another
node's slot now counts until the registry slot with the same number reads
empty. A wait for a failing holder has no time limit: it ends when the
holder's replay does, and fails only once that replay's lease lapses. A
goroutine that already holds a lock keeps the 30s limit, because the
replay may need that lock; the replay's own waits have no limit.

Failure detection replays only nodes proven dead and takes over their
locks first. A failed replay keeps them held until one succeeds. A mount
that takes a failed node's slot keeps that node's claims through its
replay, from any host, and retries a transaction it cannot apply yet. Two
replays sharing a record take turns. A node warns about each lock it has
held for over a minute, since a hold that is never released now lasts
until unmount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a lock

The mount retried any failure to apply its own journal, so a record that
does not decode or an I/O error retried forever instead of failing with
the -skip-recovery hint. It now retries only a replay that gave up waiting
for another node's lock, and a lock timeout names the holder, so each
retry logs who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A write that held a file's inode lock and waited for a block lock a
failed node held got EIO after 30s, so the replay could take the inode
lock. An overwrite holding only the shared inode lock it took for that
block has changed nothing a retry would not redo. It now gets
ErrReplayBlocked at once, lets go of the inode lock, waits for the block
holding no lock, and retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leftover claims

A node replaying a dead node passed over every adopted claim in a higher
slot, including a live node's claims while it replays its own slot at
mount, so both could hold a record. It now passes over only adopted claims
in a slot marked dead, which the registry now reports apart from a stale
heartbeat.

Claims an earlier occupant left in a slot that no replay adopted, for
example after fsck released the slot, counted as the new occupant's. A
mount now clears the claims in its slot that it did not write after its
replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry-after-replay

# Conflicts:
#	internal/lock/lock.go
#	internal/lock/owner_test.go
…s match

recoverOwnJournal returned the fatal message itself, which started with a
lowercase "journal". Mount prints the original line again, with the
-skip-recovery hint, and the helper returns the replay error as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter-replay

# Conflicts:
#	CHANGELOG.md
#	docs/OPERATIONS.md
#	internal/lock/lock.go
#	internal/lock/owner.go
#	internal/lock/owner_test.go
#	tests/test-manifest.yaml
@mattmattox
mattmattox changed the base branch from claude/lock-takeover-after-release to main November 13, 2026 19:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 30500b7 into main Nov 13, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant