Skip to content

fix(node): refuse device writes once this node no longer owns its registry slot - #92

Merged
mattmattox merged 8 commits into
mainfrom
claude/lock-write-gate-v2
Nov 13, 2026
Merged

mattmattox merged 8 commits into
mainfrom
claude/lock-write-gate-v2

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Third lock follow-up, stacked on #90. Review the last commit; the base becomes main once #90 merges.

Before this, a node that other nodes proved dead while it was stalled kept writing when it resumed, until its next heartbeat noticed.

  • The check: every device write and every discard outside the registry checks ownership first.

    • If this node wrote a heartbeat in the last 5s, the write passes without I/O. Peers declare a node dead only after 20s of silence, so that leaves margin.
    • Otherwise the node writes a heartbeat if its slot is still its own.
    • If the slot isn't its own, the node refuses the write with ErrSlotLost and stops, through the existing slot-lost handler.
  • What still gets through, now described in OPERATIONS.md and the README:

    • a write that passed the check just before the stall;
    • I/O the kernel or the storage queued during the stall;
    • writes in the second after a takeover that raced the node's resume.

    Only storage fencing, such as SCSI-3 or NVMe reservations, closes that gap.

  • Tests: device, trim and registry tests, including the registry's gate on a real device after peers marked the slot dead. Each fails with its check removed. I dropped an end-to-end SIGSTOP test, because it also passed with the gate off: after a resume, the heartbeat usually stops the node before its write lands.

  • On .68: the unit suite passes, node, storage and trim pass under -race, and the lock reclaim integration tests pass.

🤖 Generated with Claude Code

mattmattox and others added 4 commits November 13, 2026 12:06
Other nodes ignored a claim once it was 15s old, so a paused or stalled
holder lost its locks while it could still write, and a waiter got EIO
after 30s even while the holder was failing over. A claim in another
node's slot now counts until the registry slot with the same number reads
empty. A wait for a failing holder has no time limit: it ends when the
holder's replay does, and fails only once that replay's lease lapses. A
goroutine that already holds a lock keeps the 30s limit, because the
replay may need that lock; the replay's own waits have no limit.

Failure detection replays only nodes proven dead and takes over their
locks first. A failed replay keeps them held until one succeeds. A mount
that takes a failed node's slot keeps that node's claims through its
replay, from any host, and retries a transaction it cannot apply yet. Two
replays sharing a record take turns. A node warns about each lock it has
held for over a minute, since a hold that is never released now lasts
until unmount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a lock

The mount retried any failure to apply its own journal, so a record that
does not decode or an I/O error retried forever instead of failing with
the -skip-recovery hint. It now retries only a replay that gave up waiting
for another node's lock, and a lock timeout names the holder, so each
retry logs who holds the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…istry slot

A node that other nodes proved dead while it was stalled kept writing when
it resumed, until its next heartbeat noticed. Every device write and
discard outside the registry now checks that this node wrote a heartbeat
in the last 5s, or writes one while its slot is still its own. Otherwise
it refuses the write and stops the node. A write that passed the check
before the stall can still land; only storage fencing closes that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-v2

# Conflicts:
#	CHANGELOG.md
#	cmd/mount/main.go
#	cmd/mount/recovery_test.go
#	docs/OPERATIONS.md
#	internal/lock/lock.go
#	internal/lock/owner.go
#	internal/lock/owner_test.go
#	internal/lock/reclaim.go
#	internal/lock/reclaim_test.go
#	internal/lock/slots.go
#	internal/node/liveness.go
#	internal/node/registry.go
#	internal/node/registry_test.go
#	tests/test-manifest.yaml
@mattmattox
mattmattox changed the base branch from claude/lock-takeover-after-release to main November 13, 2026 19:37
mattmattox and others added 4 commits November 13, 2026 13:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rtbeat

A write that finds the last heartbeat over 5s old writes a heartbeat
first. It now logs that, so a node whose heartbeats fall behind under
load shows up before it ever loses its slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit eca5deb into main Nov 13, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant