Skip to content

fix(goclusterfs): refresh the cached inode when getting a file's attributes - #147

Merged
mattmattox merged 3 commits into
mainfrom
claude/netfault-coherence
Nov 14, 2026
Merged

mattmattox merged 3 commits into
mainfrom
claude/netfault-coherence

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Measures cache coherence under network faults, and fixes the one read I found outside the bound.

How I measured (Docker on .68, defaults -cache-ttl 5s and -attr-timeout 1s):

  • Node 1 rewrites a file 60 times, alternating truncate and grow.
  • Three readers poll every 10 ms, one way of looking per node: node 2 through an open handle, node 3 with stat on the path, node 4 by opening the file fresh each time.
  • Faults: tc netem on the writer (loss, duplication, reordering, delay); iptables on the reader (one-way partition, multicast dropped, ingress loss); multicast blocked at the writer; plus a flood from node 4 under loss.

Results:

  • A fresh open saw each version within about 10 ms, and stat within 1.0s, in every scenario.
  • The open handle stayed within 2.0s for everything except a reader that hears nothing from the writer: multicast dropped, or a one-way partition. There it reached 9.4s, and 4 of 10 runs went over the 6s bound.
  • The cause: fstat reads the new size from disk, but Read used the cached inode, so a read at the new end of file came back short until -cache-ttl passed.

With Getattr also caching the in-use inode it read, the open handle's worst case in those runs is 0.9–2.0s.

Tests:

  • New TestMultiNodeOpenFileSeesGrowthWithoutBroadcast fails without the fix.
  • Docker group 2, which includes the six new staleness tests, passes.
  • The -short integration suite, goclusterfs unit tests, lint and parity pass.

Not fixed: if all multicast is silently dropped, a node hears nothing from its peers, so #96 has no gap to detect and nothing is logged. Details are in the shared notes.

🤖 Generated with Claude Code

mattmattox and others added 3 commits November 14, 2026 10:02
…ibutes

On a node whose broadcast messages from a writer did not arrive, a file
held open could return short reads for up to 9 seconds after the writer
grew it. Getattr read the new size from disk, so fstat showed it within
-attr-timeout, but reads used the cached inode until -cache-ttl passed;
with the writer still changing the file, each cache refresh caught a
version that was already old, and the two kept disagreeing. Getattr now
stores the inode it read, so reads catch up whenever the kernel rechecks
the attributes, about every -attr-timeout while the file is read.

Measured in Docker with tc netem and iptables (writer on one node,
readers through an open handle, stat and a fresh open on three others):
with multicast dropped or a one-way partition, the open handle's worst
lag went from 4.7-9.4 s (4 of 10 runs over the 6 s bound) to 0.9-2.0 s.
Loss up to 30%, duplication, reordering, 2 s delay and a flood under loss
stayed within 2 s before and after.

tests/docker/staleness is the measuring tool; six Docker tests run it
under each fault, and TestMultiNodeOpenFileSeesGrowthWithoutBroadcast
checks the fix without any network fault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader frozen at an early version passed: versions it never saw were
skipped, and a reader that logged nothing dropped out of the report.
Readers watch until well after the writer's last version, so each must
reach every version, and every reader log must have looks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 64e5c1e into main Nov 14, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant