Repository navigation
fix(node): refuse device writes once this node no longer owns its registry slot - #92
Merged
Merged
Conversation
Other nodes ignored a claim once it was 15s old, so a paused or stalled holder lost its locks while it could still write, and a waiter got EIO after 30s even while the holder was failing over. A claim in another node's slot now counts until the registry slot with the same number reads empty. A wait for a failing holder has no time limit: it ends when the holder's replay does, and fails only once that replay's lease lapses. A goroutine that already holds a lock keeps the 30s limit, because the replay may need that lock; the replay's own waits have no limit. Failure detection replays only nodes proven dead and takes over their locks first. A failed replay keeps them held until one succeeds. A mount that takes a failed node's slot keeps that node's claims through its replay, from any host, and retries a transaction it cannot apply yet. Two replays sharing a record take turns. A node warns about each lock it has held for over a minute, since a hold that is never released now lasts until unmount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a lock The mount retried any failure to apply its own journal, so a record that does not decode or an I/O error retried forever instead of failing with the -skip-recovery hint. It now retries only a replay that gave up waiting for another node's lock, and a lock timeout names the holder, so each retry logs who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…istry slot A node that other nodes proved dead while it was stalled kept writing when it resumed, until its next heartbeat noticed. Every device write and discard outside the registry now checks that this node wrote a heartbeat in the last 5s, or writes one while its slot is still its own. Otherwise it refuses the write and stops the node. A write that passed the check before the stall can still land; only storage fencing closes that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-v2 # Conflicts: # CHANGELOG.md # cmd/mount/main.go # cmd/mount/recovery_test.go # docs/OPERATIONS.md # internal/lock/lock.go # internal/lock/owner.go # internal/lock/owner_test.go # internal/lock/reclaim.go # internal/lock/reclaim_test.go # internal/lock/slots.go # internal/node/liveness.go # internal/node/registry.go # internal/node/registry_test.go # tests/test-manifest.yaml
mattmattox
changed the base branch from
claude/lock-takeover-after-release
to
main
November 13, 2026 19:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-v2 # Conflicts: # CHANGELOG.md
…rtbeat A write that finds the last heartbeat over 5s old writes a heartbeat first. It now logs that, so a node whose heartbeats fall behind under load shows up before it ever loses its slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third lock follow-up, stacked on #90. Review the last commit; the base becomes main once #90 merges.
Before this, a node that other nodes proved dead while it was stalled kept writing when it resumed, until its next heartbeat noticed.
The check: every device write and every discard outside the registry checks ownership first.
ErrSlotLostand stops, through the existing slot-lost handler.What still gets through, now described in OPERATIONS.md and the README:
Only storage fencing, such as SCSI-3 or NVMe reservations, closes that gap.
Tests: device, trim and registry tests, including the registry's gate on a real device after peers marked the slot dead. Each fails with its check removed. I dropped an end-to-end SIGSTOP test, because it also passed with the gate off: after a resume, the heartbeat usually stops the node before its write lands.
On .68: the unit suite passes, node, storage and trim pass under -race, and the lock reclaim integration tests pass.
🤖 Generated with Claude Code