Skip to content

fix(cache): keep extent mappings and inodes coherent across nodes - #24

Merged
mattmattox merged 9 commits into
mainfrom
fix/cache-coherency
Oct 24, 2026
Merged

mattmattox merged 9 commits into
mainfrom
fix/cache-coherency

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Cached extent mappings now go stale on any inode write, not only when DataGeneration changes, which ordinary writes never did. Every mapping change, including #8's tree commits, goes through Table.Write, which bumps MetaGeneration. So no new bump sites are needed in extent or fs.go.

There's a new -attr-timeout flag for the kernel attribute and entry cache. Its default stays 1s, per the PM's decision. With 0, every stat and path step reads the directory under a cluster lock. The reviewer measured, at 0 vs 1s:

  • 2000 stats of one file: 13.9s vs 0.01s.
  • ls -l of 300 files: 11.8s vs 3.7s.

README Limits and docs/API-CONFIG.md document the window: stat/ls and reads right after open can lag up to -attr-timeout, and open files up to -cache-ttl. The fs.go comments that said kernel caching was disabled were wrong, because go-fuse replaces 0 with the mount option. I didn't change docs/DESIGN-cache.md, which is author-owned.

The write paths now broadcast the invalidation before caching the new inode. A stale extent lookup now counts as a miss instead of a hit.

Tested on .68:

  • The new two-node tests fail on cd52194.
  • On c44147f: unit, -race for the cache package, and -short integration (104 pass, 16 skip, 0 fail). TestMultiNodeAttrTimeoutZero fails when its reader mounts with -attr-timeout=1s.
  • Docker (43 pass, 1 skip) and VM (48 pass) on fcba4f3.

🤖 Generated with Claude Code

mattmattox and others added 9 commits October 24, 2026 11:12
Cached extent mappings were validated only against DataGeneration,
which ordinary writes never change, so a node kept using a mapping
after another node truncated, rewrote, hole-filled or CoW-copied the
file. Every mapping change is committed by an inode write, which bumps
MetaGeneration, so mappings now carry both generations and miss for
any later inode version. The extent invalidation hooks, which were
no-ops, now drop the cached inode so the next lookup reads the current
version.

CacheInode no longer replaces a newer cached copy, so a disk read in
File.Open that finishes after a local write cannot install the older
inode.

The kernel attribute and entry timeouts are now 0. go-fuse replaced the
AttrValid/EntryValid of 0 that GoClusterFS returns with the 1s mount
option, so a node could serve a stale size for up to 1s after another
node wrote the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	internal/cache/extent_cache.go
# Conflicts:
#	tests/docker/scripts/run-tests.sh
#	tests/vm/scripts/run-vm-tests.sh
- Keep the kernel attribute and entry timeouts at 1s. At 0, every stat
  and path step runs a directory lookup with a cluster lock; a stat loop
  on one file went from 0.01s to 13.9s. Document the 1s window and what
  an open file can read within -cache-ttl.
- Broadcast the invalidation before caching the new inode in the write
  paths, so a concurrent read cannot reinsert an older inode after the
  local invalidation.
- Count a stale extent lookup as a miss instead of a hit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default stays 1s. Document the cross-node staleness window it
allows and the cost of 0, and fix the fs.go comments that said kernel
caching was disabled: go-fuse replaces a zero AttrValid/EntryValid with
this timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit bc6fc15 into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
* docs: sync operator guides with current flags and behavior

OPERATIONS, DEPLOYMENT and TROUBLESHOOTING still used the old
goclusterfs binary name, a -format without -mount, a .snapshots
directory and snapshot API that never existed, lost+found, a
-fuse-debug flag, a mount -version flag, pprof, and read-only mounts.

They now match the code at e31fa92: format v2 and per-node journals,
-skip-recovery and damaged journal slots (#34), fsck exit codes and
-repair (#32), -attr-timeout (#24), lock renewal and takeover (#13,
#23), xattr snapshots and clones (#16), and the real log lines.

OPERATIONS.md gains a Security Model section: trust in the shared
device, single-user mounts with no permission checks, lock limits,
and unauthenticated broadcast unless -broadcast-secret is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: lock waits have no time limit

ErrLockTimeout only fires when the request context is cancelled;
lock.DefaultTimeout is never used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: make damaged-journal recovery safe and fix review findings

-skip-recovery discards the journal of whichever slot the mount takes,
which is not the damaged slot once a peer holds its recovery lease.
The steps now say when to run it and how to check the slot, and only
use it for a damaged header, not for I/O or replay errors.

Also quote the real legacy-layout and superblock errors, cover a v2
device with a wiped superblock, state that there is no HTTP API, and
correct peer-announce, replay, unicast fallback and cache lag claims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant