Repository navigation
fix(cache): keep extent mappings and inodes coherent across nodes - #24
Merged
Merged
Conversation
Cached extent mappings were validated only against DataGeneration, which ordinary writes never change, so a node kept using a mapping after another node truncated, rewrote, hole-filled or CoW-copied the file. Every mapping change is committed by an inode write, which bumps MetaGeneration, so mappings now carry both generations and miss for any later inode version. The extent invalidation hooks, which were no-ops, now drop the cached inode so the next lookup reads the current version. CacheInode no longer replaces a newer cached copy, so a disk read in File.Open that finishes after a local write cannot install the older inode. The kernel attribute and entry timeouts are now 0. go-fuse replaced the AttrValid/EntryValid of 0 that GoClusterFS returns with the 1s mount option, so a node could serve a stale size for up to 1s after another node wrote the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # internal/cache/extent_cache.go
# Conflicts: # tests/docker/scripts/run-tests.sh # tests/vm/scripts/run-vm-tests.sh
- Keep the kernel attribute and entry timeouts at 1s. At 0, every stat and path step runs a directory lookup with a cluster lock; a stat loop on one file went from 0.01s to 13.9s. Document the 1s window and what an open file can read within -cache-ttl. - Broadcast the invalidation before caching the new inode in the write paths, so a concurrent read cannot reinsert an older inode after the local invalidation. - Count a stale extent lookup as a miss instead of a hit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default stays 1s. Document the cross-node staleness window it allows and the cost of 0, and fix the fs.go comments that said kernel caching was disabled: go-fuse replaces a zero AttrValid/EntryValid with this timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
* docs: sync operator guides with current flags and behavior OPERATIONS, DEPLOYMENT and TROUBLESHOOTING still used the old goclusterfs binary name, a -format without -mount, a .snapshots directory and snapshot API that never existed, lost+found, a -fuse-debug flag, a mount -version flag, pprof, and read-only mounts. They now match the code at e31fa92: format v2 and per-node journals, -skip-recovery and damaged journal slots (#34), fsck exit codes and -repair (#32), -attr-timeout (#24), lock renewal and takeover (#13, #23), xattr snapshots and clones (#16), and the real log lines. OPERATIONS.md gains a Security Model section: trust in the shared device, single-user mounts with no permission checks, lock limits, and unauthenticated broadcast unless -broadcast-secret is set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: lock waits have no time limit ErrLockTimeout only fires when the request context is cancelled; lock.DefaultTimeout is never used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: make damaged-journal recovery safe and fix review findings -skip-recovery discards the journal of whichever slot the mount takes, which is not the damaged slot once a peer holds its recovery lease. The steps now say when to run it and how to check the slot, and only use it for a damaged header, not for I/O or replay errors. Also quote the real legacy-layout and superblock errors, cover a v2 device with a wiped superblock, state that there is no HTTP API, and correct peer-announce, replay, unicast fallback and cache lag claims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cached extent mappings now go stale on any inode write, not only when
DataGenerationchanges, which ordinary writes never did. Every mapping change, including #8's tree commits, goes throughTable.Write, which bumpsMetaGeneration. So no new bump sites are needed inextentorfs.go.There's a new
-attr-timeoutflag for the kernel attribute and entry cache. Its default stays 1s, per the PM's decision. With 0, every stat and path step reads the directory under a cluster lock. The reviewer measured, at 0 vs 1s:ls -lof 300 files: 11.8s vs 3.7s.README Limits and
docs/API-CONFIG.mddocument the window:stat/lsand reads right afteropencan lag up to-attr-timeout, and open files up to-cache-ttl. Thefs.gocomments that said kernel caching was disabled were wrong, because go-fuse replaces 0 with the mount option. I didn't changedocs/DESIGN-cache.md, which is author-owned.The write paths now broadcast the invalidation before caching the new inode. A stale extent lookup now counts as a miss instead of a hit.
Tested on .68:
-racefor the cache package, and-shortintegration (104 pass, 16 skip, 0 fail).TestMultiNodeAttrTimeoutZerofails when its reader mounts with-attr-timeout=1s.🤖 Generated with Claude Code