Repository navigation
fix(node): judge peers on this node's own clock, and keep registry scans off the heartbeat - #151
Merged
Merged
Conversation
…ans off the heartbeat Nodes judged each other's heartbeats and recovery leases by wall clock: - a node mounting with its clock more than 10s ahead took a live node's registry slot, and that node stopped; - a crashed node whose clock ran ahead was replayed only once its lead ran out, so lock waits on its records failed with EIO meanwhile; - a replay whose node's clock was behind looked abandoned. A node now records each slot it reads and counts the slot's node as failing, dead or abandoned only once it has read the same heartbeat or lease for long enough by its own monotonic clock. A replay renews its lease on its own clock, to a value that always changes. Register takes an empty slot first and takes another host's slot only after watching its heartbeat not change. Under slow storage a node was declared dead while only slow: registry scans held the heartbeat's lock across 64 reads, so at 1s per I/O a peer replayed the node. Scans now read the registry in one I/O without that lock, and registry I/O skips the per-region shard lock, as lock slots already do. storage.Device gains SetReadFault, a read-side twin of SetWriteGate for tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t leases as replays do fsck's release tests backdated a lease or heartbeat on disk and expected a peer to act on it at once; peers now act only on what they watch stay unchanged. The crash test lets a 1s lease lapse for real through node.SetRecoveryLease, and the late mount comes back as the same host, the one way it still takes a stale slot while another is empty. The test helper that renews a lease now always changes it, as a replay's renewal does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it Taking empty slots first removed the way a mount on another host takes a crashed node's slot and replays its journal, and with it -skip-recovery for that journal. A mount now again takes another host's slot whose heartbeat looks old by its clock, but only after watching the heartbeat stay unchanged for HeartbeatTimeout plus two beats; a live node whose clock is behind beats within that time and keeps its slot. Integration mounts may now take up to 30s to become ready. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed, and take empty slots first Register claimed a slot it had watched stay silent with a compare-and-swap against a fresh read, so a slot a peer marked dead to replay it, or whose node beat again, in between was taken anyway. It now swaps against the bytes it watched, so any change makes the claim fail. A mount takes an empty slot first, so it never waits while one is free. Another host's silent slot is taken only when -skip-recovery names it, before any other slot, or no slot is empty. A named slot under another node's recovery is taken once its lease stays unrenewed for the lease term. Failed replays now name -skip-recovery in their log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WaitForMount saw an exit only through ProcessState, which stays nil until someone calls Wait, so a mount meant to fail always waited the full timeout. It now also checks whether the process is a zombie. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tbeat is old A restarted container keeps its hostname but gets a new PID namespace, so its mount watched its old slot. The watch always ran its full 12s; it now ends, as before, once the slot's heartbeat is older than HeartbeatTimeout by this host's clock, which wrote it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it After a crash and reboot a host has a new boot ID, so its mount no longer counted its old slot as its own and took an empty one, leaving the old slot and its dirty journal for peers. Until a peer replayed it, the old node's locks blocked the new mount, and an fsck run in between refused to repair. A mount now watches the slots its hostname held in an earlier boot, whose heartbeat looks old, before taking an empty one, and takes back one that stays unchanged, replaying its journal before serving. Another host with the same name beats during the watch and keeps its slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The registry fuzz test from #153 called failed(), which this branch replaces. It now checks that a slot seen for the first time is never stale, that an active slot's heartbeat age agrees with LeaseExpiry by this host's clock, and that a dead slot's lease ends recoveryLease after it started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 16, 2026
…d it The grace period used the peer's MountTime from the registry, written by the peer's clock, against this node's clock, so a peer with a clock ahead went unreported for up to the skew. Silence now counts from when this node first saw the peer in the registry, by its own clock, as #151 did for liveness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 16, 2026
…rom (#166) * feat(broadcast): report mounted nodes this node hears no broadcasts from A network that drops multicast was invisible: a node that hears nothing from a peer finds no gap in its sequence numbers, so #96's loss check never fires, and only directory lookups lagging by -cache-ttl showed it. Every -broadcast-heartbeat interval (30s by default, now a flag), the broadcaster compares the nodes the registry shows mounted with the ones it has heard. A node silent for 3 intervals since it mounted, or since this node started, gets a WARN naming it, a degraded broadcast-peers check on /health, and a count in goclusterfs_broadcast_silent_peers. Hearing it again logs that and clears it. The Docker nodes run with a 2s heartbeat, so the new Docker test sees node 2 report its three peers within seconds of dropping multicast and recover after the fault is healed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(broadcast): time a peer's silence from when this node first listed it The grace period used the peer's MountTime from the registry, written by the peer's clock, against this node's clock, so a peer with a clock ahead went unreported for up to the skew. Silence now counts from when this node first saw the peer in the registry, by its own clock, as #151 did for liveness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes hostile-conditions findings 1, 2, 3 and 6 (shared notes, hostile-conditions/findings.md).
-skip-recoverynames comes first: empty, or this host's exited mount, at once; another host's silent node after a 12 s watch; a slot under another node's failing replay once its lease stays unrenewed for the lease term.r.mu, and registry I/O skips the shard lock.Tests come from the repros, and each fails with its part reverted. The latency and clock VM scenarios (which failed on main) are added.
node.SetRecoveryLeaseis exported so fsck's tests can watch a lease lapse. Integration mounts may take up to 30 s (only a named-skip-recoverymount waits). Docs drop "clocks must agree" for failover and rewrite the Damaged Journal Slot steps.🤖 Generated with Claude Code