Skip to content

fix(lock): take over a lock only after its holder is released - #90

Merged
mattmattox merged 4 commits into
mainfrom
claude/lock-takeover-after-release
Nov 13, 2026
Merged

mattmattox merged 4 commits into
mainfrom
claude/lock-takeover-after-release

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Second of the lock follow-ups. #83 (background renewal) is closed: under this PR a claim never goes stale, so renewing it protects nothing.

Before this PR, a claim older than 15s stopped counting. So a stalled holder lost its locks while it could still write, and a waiter got EIO after 30s even during a failover.

  • When a claim frees: another node's claim counts until the registry slot with the same number reads empty. The node reads that slot from the device after it reads the lock record. Claim age no longer matters.
  • Waits:
    • A wait for a failing holder has no time limit. A failing holder is one with a stale heartbeat, dead, or being replayed. The wait logs a WARN that names that node, and fails with EIO only once the replay's recovery lease lapses.
    • The 30s EIO stays for live holders.
    • It also stays for a goroutine that already holds a lock and then waits on a failing holder, because the replay may need the lock it holds. The replay's own waits have no limit, so that wait cycle ends with the peer giving up.
  • Replay:
    • Failure detection replays only nodes proven dead, and takes over their locks first.
    • A failed replay keeps the dead node's locks held until a replay succeeds.
    • A mount that takes a failed slot, from any host, keeps the previous owner's claims through its replay. It used to do that only for an exited process on the same host.
    • Mount retries its own replay only when the replay gave up waiting for another node's lock, and logs the holder each time. Any other failure, such as a record that doesn't decode or an I/O error, fails the mount at once with the -skip-recovery hint.
    • After its replay, a mount clears the claims in its own slot that an earlier occupant left, for example after fsck released the slot.
  • Other:
    • Two replays whose dead nodes' claims share a record take turns. Only claims in a slot marked dead take turns; a live node's own-slot replay claims never do. The registry now reports a dead slot apart from a stale heartbeat.
    • A lock held over 60s logs a WARN, since a hold that is never released now lasts until unmount.

Failover times: 4.9s for a crash that a survivor on the same host can see. 34.9s, with no EIO, in the new integration test, where node A holds a lock alive for 13s and then stops. Without this PR that test fails, because B takes A's lock at 15s, before A is proven dead.

Behavior change: after a whole-cluster crash, the first mount waits up to about 20s for locks the dead nodes held, while it proves them dead.

Tests: lock tests against a fake registry, including the shepherd's replay-vs-peer example; registry tests; mount tests for a lock-wait retry and for a bad record that stops at once; and the integration test. Each rule has a mutation that fails its test. On .68, the unit suite passes, lock, node and journal pass under -race, and the lock reclaim integration tests pass.

🤖 Generated with Claude Code

Other nodes ignored a claim once it was 15s old, so a paused or stalled
holder lost its locks while it could still write, and a waiter got EIO
after 30s even while the holder was failing over. A claim in another
node's slot now counts until the registry slot with the same number reads
empty. A wait for a failing holder has no time limit: it ends when the
holder's replay does, and fails only once that replay's lease lapses. A
goroutine that already holds a lock keeps the 30s limit, because the
replay may need that lock; the replay's own waits have no limit.

Failure detection replays only nodes proven dead and takes over their
locks first. A failed replay keeps them held until one succeeds. A mount
that takes a failed node's slot keeps that node's claims through its
replay, from any host, and retries a transaction it cannot apply yet. Two
replays sharing a record take turns. A node warns about each lock it has
held for over a minute, since a hold that is never released now lasts
until unmount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a lock

The mount retried any failure to apply its own journal, so a record that
does not decode or an I/O error retried forever instead of failing with
the -skip-recovery hint. It now retries only a replay that gave up waiting
for another node's lock, and a lock timeout names the holder, so each
retry logs who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox and others added 2 commits November 13, 2026 12:25
…leftover claims

A node replaying a dead node passed over every adopted claim in a higher
slot, including a live node's claims while it replays its own slot at
mount, so both could hold a record. It now passes over only adopted claims
in a slot marked dead, which the registry now reports apart from a stale
heartbeat.

Claims an earlier occupant left in a slot that no replay adopted, for
example after fsck released the slot, counted as the new occupant's. A
mount now clears the claims in its slot that it did not write after its
replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s match

recoverOwnJournal returned the fatal message itself, which started with a
lowercase "journal". Mount prints the original line again, with the
-skip-recovery hint, and the helper returns the replay error as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 53ef02a into main Nov 13, 2026
13 checks passed
mattmattox added a commit that referenced this pull request Nov 13, 2026
Since #90 no node takes over a live node's lock. Release returns
ErrNotLockOwner only when this node's own bookkeeping lacks the lock, a
double release that may have dropped another holder's lock early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 13, 2026
… there (#95)

* fix: check ignored errors in lock, inode and extent; enforce errcheck there

- inode table: a failed lock release after an inode or sector update
  was dropped. If another node had taken the lock over, both nodes may
  have written the same inode table sector. Now logged as an error with
  that reason; other release failures log a warning.
- extent: when allocate or MapCopy fail, the blocks they took are freed
  again. A failed free left them allocated with no owner and no trace;
  the returned error now names each one.
- lock.WithInodeLock now returns the release error.
- Tests check setup steps and lock releases; lock tests use a release
  helper that reports failures with Errorf, so goroutines can use it.

errcheck is now enforced in lock, inode and extent too; 438 findings
remain elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(inode): name the real cause of ErrNotLockOwner on release

Since #90 no node takes over a live node's lock. Release returns
ErrNotLockOwner only when this node's own bookkeeping lacks the lock, a
double release that may have dropped another holder's lock early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant