Skip to content

fix(broadcast): give unicast its own port so broadcast actually starts - #59

Merged
mattmattox merged 2 commits into
mainfrom
claude/fix-broadcast-ports
Nov 10, 2026
Merged

mattmattox merged 2 commits into
mainfrom
claude/fix-broadcast-ports

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Broadcast never started with default flags. The multicast socket holds -mcast-port, and the unicast socket then bound the same port without address reuse, so Start failed with EADDRINUSE on every node. The mount went on without broadcast, so peers caught up only after the 5s cache TTL. main's -short run on .68 logs 151 Failed to start cache broadcast lines.

  • Ports: unicast gets its own port, -unicast-port (default 31338). Start rejects equal ports. Firewalls must now allow UDP 31338 too.
  • Same-host delivery: multicast is sent from the unicast socket. Go's ListenMulticastUDP turns off multicast loopback on its socket, so even with the bind fixed, nodes on one host never heard each other. The send socket uses -broadcast-iface, loopback on, and the configured TTL.
  • Failure handling: a mount with -broadcast on fails when broadcast cannot start, naming both ports and -broadcast=false. The silent fallback is what hid this bug.
  • Tests: integration nodes get their own unicast port and a multicast port per cluster. TestBroadcastInvalidatesPeerCache gives node 1 a 1h cache TTL; node 1 must still see node 0's append through an open file, which only a broadcast invalidation allows.

On .68:

Suite main 55ca3cd this branch
Go -short 99 passed, 151 broadcast start failures, 0 peer announcements, 175s 100 passed (+ new test), 0 failures, 43 announcements, 177s
Docker (full) 42 passed, 0 failed, 605s; broadcast failed at every node start 42 passed, 0 failed, 557s; every node started broadcast and received thousands of invalidations
VM — 55 passed, 1 failed (Concurrent Directory Ops: two files lost from both VMs, the #29 lock race); broadcast started on every VM

In Go, no other test changed result or speed by 2s or more. In Docker, per-test times moved 10-40s in both directions between single runs on a shared host. The VM failure cannot come from this PR: VMs exchanged no broadcast messages. VMs use QEMU user networking, so they cannot reach each other: broadcast starts but has no peers there.

Shared-lock leak, relevant to #29: with broadcast live, TestMultiNodeConcurrentCreatesDoNotCollide took 16s instead of 1s in 2 of 4 runs. Two nodes "acquired" the root directory lock at the same moment (one free, one shared add), the shared count never dropped, and the other node waited 15s to reclaim it as stale. main's run also had a stale reclaim. Broadcast adds disk reads after invalidations, so lock traffic, and these leaks, go up.

PR 2 gives test nodes their own API ports and reports the harness changes.

🤖 Generated with Claude Code

mattmattox and others added 2 commits November 10, 2026 16:37
Broadcast never started with default flags. The multicast socket holds
-mcast-port, and the unicast socket then bound the same port without
address reuse, so Start failed with EADDRINUSE on every node. The mount
logged a warning and went on, and peers only caught up after the 5s
cache TTL.

- Unicast gets its own port, -unicast-port (default 31338). Start
  rejects a unicast port equal to the multicast port.
- Multicast now goes out on the unicast socket. ListenMulticastUDP
  turns off multicast loopback on its socket, so nodes on the same host
  never received each other's messages even when both sockets bound.
  The send socket uses the joined interface (-broadcast-iface), loopback
  on and the configured TTL.
- A mount with -broadcast on now fails when broadcast cannot start, and
  names both ports and -broadcast=false. The silent fallback is what hid
  this bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…very

A mount now fails when it cannot bind its broadcast ports, so every
test node gets its own unicast port, and the nodes of a cluster share a
multicast port no other test uses. TestBroadcastInvalidatesPeerCache
gives node 1 a 1h cache TTL and checks that its open file sees data
node 0 appended, which only a broadcast invalidation can cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant