Skip to content

fix(node): judge peers on this node's own clock, and keep registry scans off the heartbeat - #151

Merged
mattmattox merged 11 commits into
mainfrom
claude/monotonic-liveness
Nov 14, 2026
Merged

mattmattox merged 11 commits into
mainfrom
claude/monotonic-liveness

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes hostile-conditions findings 1, 2, 3 and 6 (shared notes, hostile-conditions/findings.md).

  • Liveness: each node records the slots it reads. A peer counts as failing, dead or abandoned only after this node has read the same heartbeat (10 s or 20 s) or lease (60 s) unchanged on its own monotonic clock. Wall time only compares timestamps from this same host. Lease renewal is timed on the replayer's clock and always changes the value.
  • Register:
    • The slot -skip-recovery names comes first: empty, or this host's exited mount, at once; another host's silent node after a 12 s watch; a slot under another node's failing replay once its lease stays unrenewed for the lease term.
    • Otherwise empty slots come first. Another host's silent slot is taken only when none is empty.
    • Claims swap against the exact bytes watched, so a slot that changed since is left alone.
  • Heartbeat starvation: scans read the registry in one I/O without r.mu, and registry I/O skips the shard lock.

Tests come from the repros, and each fails with its part reverted. The latency and clock VM scenarios (which failed on main) are added. node.SetRecoveryLease is exported so fsck's tests can watch a lease lapse. Integration mounts may take up to 30 s (only a named -skip-recovery mount waits). Docs drop "clocks must agree" for failover and rewrite the Damaged Journal Slot steps.

🤖 Generated with Claude Code

mattmattox and others added 11 commits November 14, 2026 10:54
…ans off the heartbeat

Nodes judged each other's heartbeats and recovery leases by wall clock:
- a node mounting with its clock more than 10s ahead took a live node's
  registry slot, and that node stopped;
- a crashed node whose clock ran ahead was replayed only once its lead ran
  out, so lock waits on its records failed with EIO meanwhile;
- a replay whose node's clock was behind looked abandoned.

A node now records each slot it reads and counts the slot's node as failing,
dead or abandoned only once it has read the same heartbeat or lease for long
enough by its own monotonic clock. A replay renews its lease on its own clock,
to a value that always changes. Register takes an empty slot first and takes
another host's slot only after watching its heartbeat not change.

Under slow storage a node was declared dead while only slow: registry scans
held the heartbeat's lock across 64 reads, so at 1s per I/O a peer replayed
the node. Scans now read the registry in one I/O without that lock, and
registry I/O skips the per-region shard lock, as lock slots already do.

storage.Device gains SetReadFault, a read-side twin of SetWriteGate for tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t leases as replays do

fsck's release tests backdated a lease or heartbeat on disk and expected a
peer to act on it at once; peers now act only on what they watch stay
unchanged. The crash test lets a 1s lease lapse for real through
node.SetRecoveryLease, and the late mount comes back as the same host, the
one way it still takes a stale slot while another is empty. The test helper
that renews a lease now always changes it, as a replay's renewal does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it

Taking empty slots first removed the way a mount on another host takes a
crashed node's slot and replays its journal, and with it -skip-recovery for
that journal. A mount now again takes another host's slot whose heartbeat
looks old by its clock, but only after watching the heartbeat stay unchanged
for HeartbeatTimeout plus two beats; a live node whose clock is behind beats
within that time and keeps its slot. Integration mounts may now take up to
30s to become ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed, and take empty slots first

Register claimed a slot it had watched stay silent with a compare-and-swap
against a fresh read, so a slot a peer marked dead to replay it, or whose node
beat again, in between was taken anyway. It now swaps against the bytes it
watched, so any change makes the claim fail.

A mount takes an empty slot first, so it never waits while one is free.
Another host's silent slot is taken only when -skip-recovery names it, before
any other slot, or no slot is empty. A named slot under another node's
recovery is taken once its lease stays unrenewed for the lease term. Failed
replays now name -skip-recovery in their log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WaitForMount saw an exit only through ProcessState, which stays nil until
someone calls Wait, so a mount meant to fail always waited the full timeout.
It now also checks whether the process is a zombie.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tbeat is old

A restarted container keeps its hostname but gets a new PID namespace, so
its mount watched its old slot. The watch always ran its full 12s; it now
ends, as before, once the slot's heartbeat is older than HeartbeatTimeout by
this host's clock, which wrote it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it

After a crash and reboot a host has a new boot ID, so its mount no longer
counted its old slot as its own and took an empty one, leaving the old slot
and its dirty journal for peers. Until a peer replayed it, the old node's
locks blocked the new mount, and an fsck run in between refused to repair.

A mount now watches the slots its hostname held in an earlier boot, whose
heartbeat looks old, before taking an empty one, and takes back one that
stays unchanged, replaying its journal before serving. Another host with the
same name beats during the watch and keeps its slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The registry fuzz test from #153 called failed(), which this branch
replaces. It now checks that a slot seen for the first time is never
stale, that an active slot's heartbeat age agrees with LeaseExpiry by this
host's clock, and that a dead slot's lease ends recoveryLease after it
started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 76d34d9 into main Nov 14, 2026
17 checks passed
mattmattox added a commit that referenced this pull request Nov 16, 2026
…d it

The grace period used the peer's MountTime from the registry, written by
the peer's clock, against this node's clock, so a peer with a clock ahead
went unreported for up to the skew. Silence now counts from when this node
first saw the peer in the registry, by its own clock, as #151 did for
liveness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 16, 2026
…rom (#166)

* feat(broadcast): report mounted nodes this node hears no broadcasts from

A network that drops multicast was invisible: a node that hears nothing
from a peer finds no gap in its sequence numbers, so #96's loss check never
fires, and only directory lookups lagging by -cache-ttl showed it.

Every -broadcast-heartbeat interval (30s by default, now a flag), the
broadcaster compares the nodes the registry shows mounted with the ones it
has heard. A node silent for 3 intervals since it mounted, or since this
node started, gets a WARN naming it, a degraded broadcast-peers check on
/health, and a count in goclusterfs_broadcast_silent_peers. Hearing it
again logs that and clears it.

The Docker nodes run with a 2s heartbeat, so the new Docker test sees
node 2 report its three peers within seconds of dropping multicast and
recover after the fault is healed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(broadcast): time a peer's silence from when this node first listed it

The grace period used the peer's MountTime from the registry, written by
the peer's clock, against this node's clock, so a peer with a clock ahead
went unreported for up to the skew. Silence now counts from when this node
first saw the peer in the registry, by its own clock, as #151 did for
liveness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant