Skip to content

feat(broadcast): report mounted nodes this node hears no broadcasts from - #166

Merged
mattmattox merged 2 commits into
mainfrom
claude/silent-broadcast-peers
Nov 16, 2026
Merged

mattmattox merged 2 commits into
mainfrom
claude/silent-broadcast-peers

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Follow-up to #147: a network that drops multicast is now visible.

How it works:

  • Every -broadcast-heartbeat interval (default 30s, now a flag), the broadcaster compares the nodes the registry shows mounted with the senders it has heard.
  • A node unheard for 3 intervals, counted from its mount or from this node's start, is reported in three ways:
    • a WARN log line naming it;
    • a degraded broadcast-peers check on /health, which is new and separate from the broadcast check because the consequence runs the other way (this node sees that peer's changes late);
    • a count in goclusterfs_broadcast_silent_peers.
  • When the node is heard again, the mount logs it and the report clears.

Tests on .68:

  • TestBroadcastReportsSilentPeer, next to a -broadcast=false node: 3/3, reported 1.2–1.7s after that node mounted.
  • Docker test, with multicast dropped on node 2: degraded with 3 silent peers after 8s, back to ok 4s after healing. Docker nodes now run a 2s heartbeat.
  • Docker group 2, the -short suite, unit tests with -race, lint and parity: all pass.

A node mounted with -broadcast=false is reported too. That's deliberate: its peers see its changes late as well.

🤖 Generated with Claude Code

mattmattox and others added 2 commits November 16, 2026 13:28
A network that drops multicast was invisible: a node that hears nothing
from a peer finds no gap in its sequence numbers, so #96's loss check never
fires, and only directory lookups lagging by -cache-ttl showed it.

Every -broadcast-heartbeat interval (30s by default, now a flag), the
broadcaster compares the nodes the registry shows mounted with the ones it
has heard. A node silent for 3 intervals since it mounted, or since this
node started, gets a WARN naming it, a degraded broadcast-peers check on
/health, and a count in goclusterfs_broadcast_silent_peers. Hearing it
again logs that and clears it.

The Docker nodes run with a 2s heartbeat, so the new Docker test sees
node 2 report its three peers within seconds of dropping multicast and
recover after the fault is healed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d it

The grace period used the peer's MountTime from the registry, written by
the peer's clock, against this node's clock, so a peer with a clock ahead
went unreported for up to the skew. Silence now counts from when this node
first saw the peer in the registry, by its own clock, as #151 did for
liveness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit e2a7bee into main Nov 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant