Skip to content

fix(broadcast): notice lost invalidations and drop caches instead of waiting for the TTL - #96

Merged
mattmattox merged 2 commits into
mainfrom
claude/broadcast-loss
Nov 13, 2026
Merged

mattmattox merged 2 commits into
mainfrom
claude/broadcast-loss

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

There are no batches to fragment: each invalidation is one packet of 62 bytes, or 94 with HMAC. Under load, the loss happens in the receiver's socket buffer. Before this PR it was silent, and the peer kept stale inodes until -cache-ttl.

Measured on .68 (3000 messages):

  • Receiver at 100µs per message: 12.5% lost over multicast and 82% over unicast. Nothing was logged and no cache was cleared.
  • After this PR, TestBurstOfInvalidationsReachesPeer shows every inode covered 190–260 ms after the burst, within a 1s limit, in every run.
  • The old dedup map held 224 MB for 2.4M messages (5 minutes of 8k writes/s). Its cleanup locked the receive path for up to 200 ms.

Mutation checks:

  • Without the send lock, TestConcurrentSendsArriveInOrder fails.
  • Without trailer heartbeats, the burst test leaves about 600 inodes uncovered.

Also on .68: broadcast and cache packages (5×, -race), lint, parity, and 13 broadcast and multinode integration tests.

Follow-up, not in this PR: cache.Manager.Clear takes about 1.5 ms even when empty, because LRU.Clear reallocates every shard map.

🤖 Generated with Claude Code

mattmattox and others added 2 commits November 13, 2026 13:54
…waiting for the TTL

A receiver that fell behind dropped invalidations silently and kept stale
inodes until -cache-ttl. Peers now track each sender's sequence numbers in
a 64-message window: a gap clears the inode and extent caches, counts the
loss and logs it at most every 10s. Senders number and send under one lock
so concurrent writers do not look like loss, and send heartbeats every 50ms
for 500ms after an invalidation so a lost end of a burst shows up too.

The window replaces the 5-minute dedup map, which grew by one entry per
message (224 MB for 2.4M) and held the receive lock for up to 200 ms while
it was cleaned. Clears run off the receive path and coalesce, since one
takes about 1.5 ms. The unicast socket gets the same 1 MB read buffer as
the multicast one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing tracking

A sequence number more than 64 behind the highest one is either a
duplicate or a message a gap already counted, since each mount has its own
UUID and sequences only go up. Resetting to it made the next message look
like a gap of 100 or more: a second cache clear and a bogus loss count.

Also adds the CHANGELOG lines for lost-message handling and the smaller
dedup state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit c3d2543 into main Nov 13, 2026
13 checks passed
mattmattox added a commit that referenced this pull request Nov 16, 2026
…rom (#166)

* feat(broadcast): report mounted nodes this node hears no broadcasts from

A network that drops multicast was invisible: a node that hears nothing
from a peer finds no gap in its sequence numbers, so #96's loss check never
fires, and only directory lookups lagging by -cache-ttl showed it.

Every -broadcast-heartbeat interval (30s by default, now a flag), the
broadcaster compares the nodes the registry shows mounted with the ones it
has heard. A node silent for 3 intervals since it mounted, or since this
node started, gets a WARN naming it, a degraded broadcast-peers check on
/health, and a count in goclusterfs_broadcast_silent_peers. Hearing it
again logs that and clears it.

The Docker nodes run with a 2s heartbeat, so the new Docker test sees
node 2 report its three peers within seconds of dropping multicast and
recover after the fault is healed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(broadcast): time a peer's silence from when this node first listed it

The grace period used the peer's MountTime from the registry, written by
the peer's clock, against this node's clock, so a peer with a clock ahead
went unreported for up to the skew. Silence now counts from when this node
first saw the peer in the registry, by its own clock, as #151 did for
liveness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant