Repository navigation
fix(goclusterfs): refresh the cached inode when getting a file's attributes - #147
Merged
Merged
Conversation
…ibutes On a node whose broadcast messages from a writer did not arrive, a file held open could return short reads for up to 9 seconds after the writer grew it. Getattr read the new size from disk, so fstat showed it within -attr-timeout, but reads used the cached inode until -cache-ttl passed; with the writer still changing the file, each cache refresh caught a version that was already old, and the two kept disagreeing. Getattr now stores the inode it read, so reads catch up whenever the kernel rechecks the attributes, about every -attr-timeout while the file is read. Measured in Docker with tc netem and iptables (writer on one node, readers through an open handle, stat and a fresh open on three others): with multicast dropped or a one-way partition, the open handle's worst lag went from 4.7-9.4 s (4 of 10 runs over the 6 s bound) to 0.9-2.0 s. Loss up to 30%, duplication, reordering, 2 s delay and a flood under loss stayed within 2 s before and after. tests/docker/staleness is the measuring tool; six Docker tests run it under each fault, and TestMultiNodeOpenFileSeesGrowthWithoutBroadcast checks the fix without any network fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader frozen at an early version passed: versions it never saw were skipped, and a reader that logged nothing dropped out of the report. Readers watch until well after the writer's last version, so each must reach every version, and every reader log must have looks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measures cache coherence under network faults, and fixes the one read I found outside the bound.
How I measured (Docker on .68, defaults
-cache-ttl 5sand-attr-timeout 1s):staton the path, node 4 by opening the file fresh each time.Results:
statwithin 1.0s, in every scenario.fstatreads the new size from disk, butReadused the cached inode, so a read at the new end of file came back short until-cache-ttlpassed.With
Getattralso caching the in-use inode it read, the open handle's worst case in those runs is 0.9–2.0s.Tests:
TestMultiNodeOpenFileSeesGrowthWithoutBroadcastfails without the fix.-shortintegration suite,goclusterfsunit tests, lint and parity pass.Not fixed: if all multicast is silently dropped, a node hears nothing from its peers, so #96 has no gap to detect and nothing is logged. Details are in the shared notes.
🤖 Generated with Claude Code