Repository navigation
fix(api): run snapshot and clone requests inside the mount daemon - #192
Merged
Merged
Conversation
TestAPIWritesDuringFileIO now watches each write, read and API call. One that takes over 30 s makes the test send SIGUSR1 to the mount daemon for a goroutine dump, save the kernel stacks of the daemon and the test, read the FUSE connection's waiting count, and then abort the connection, so a hang fails the test with evidence instead of leaving a zombie that holds the host lock. GCFS_STALL_DIR keeps a copy of the record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API opened the file and called fsetxattr on the daemon's own FUSE mount, so a daemon thread waited on the kernel while the kernel waited on the daemon. Under concurrent writes that wedged the daemon: its heartbeat stopped, it lost its registry slot, and the process stayed a zombie with a thread stuck in fsetxattr. The API now finds the file by walking its directories inside the filesystem, refusing a symlink anywhere on the path as openat2 did, and calls File.setxattr, the body Setxattr runs, with the same inode lock, generation check and cluster locks. Listing a file's snapshots uses the same lookup, so no API call touches the mount. After a snapshot or clone, a background goroutine expires the kernel's cached attributes of the file and any cached miss for the clone's name, and logs a warning if that takes over 5 s; no API response waits on the kernel. A file replaced between the lookup and the lock answers ESTALE, which the API maps to 409. Without the kernel's i_rwsem, the inode lock keeps each FUSE write request whole against a snapshot. A write() larger than 128 KiB reaches the daemon as several requests, and a snapshot can land between them, as one taken on another node always could. API-CONFIG.md says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot or clone captures each FUSE write request whole. goclusterfs now names the request size as MaxWrite (128 KiB, go-fuse's default, set explicitly in the mount options), and the same-node test writes MaxWrite bytes per version, so both follow a later change. OPERATIONS.md states the guarantee; API-CONFIG.md links to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…talled A node whose heartbeats failed for half the timeout stopped with "Node registry slot N is no longer this node's (owner <its own UUID>)", which reads like another node took the slot; it misled a debugging session. The slot-lost handler now gets a SlotLoss, whose message names the cause: "Heartbeats to registry slot N failed for Xs" for a stall, "(owner <uuid>)" only when another node's UUID is in the slot, and plain words when peers declared this node dead or emptied the slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 17, 2026
#192 named the request size goclusterfs.MaxWrite and kept 128 KiB until this PR. The mount now sends write and read requests of up to 1 MiB, so reads and snapshots see each write of up to 1 MiB whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 17, 2026
* feat(fs): hold a cluster file I/O lock for each whole read and write A read request took no lock, so it could return part of another node's write, or of a truncate. Each read, write, truncate and fallocate now takes a per-file I/O lock for the whole request: shared for reads, exclusive for changes, with a local stripe for the node's own requests. Snapshots and clones take it shared. A reader waits behind a writer's waiting mark for up to two turns, so readers on lower slots cannot shut a writer out. The first read after each claim reads the inode again. File I/O locks get 39 of the 256 lock records, so the record layout changes: RequiredFileIOLocks marks new volumes, mount refuses older ones, and each registry slot records its lock layout, which mount checks against live nodes. FUSE requests now carry up to 1 MB, and -direct-io bypasses the page cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(lock): keep a record nobody waits for, and skip stack walks for file I/O locks Reads now take a lock per request, which showed two costs. Finding the goroutine ID formats the caller's whole stack, about 55 us per read on FUSE stacks: file I/O locks now go through AcquireAs and ReleaseAs with a per-request owner. And a holder gave up every record at the end of each 250 ms turn and waited 50 ms even when no other node wanted it: it now reads the row when the turn ends and starts a new turn if the row shows no conflicting claim and no live mark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(lock): take the file I/O rows from extent locks, not inode locks With 96 inode rows, unrelated inodes shared records more often: in the 4-node probe, deletes ran 5 to 13% slower than main and inode lock waits rose from 125% to about 150% of node time. Whole-block writes take no extent locks now that requests carry 1 MB, so extents give up 32 rows and inodes keep 128. Deletes then ran 429 to 451 files/s at 4 nodes (main 375 to 382), and overwrites of separate files from 4 nodes stayed within noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(node): spell out the lock layout check without a negated conjunction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(operations): drop a cross-reference that pointed the wrong way Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): keep a new claim for 50 ms before a long wait ends the turn A node that has seen another node waiting for a whole turn gave up a new claim at its first release. With a reader on another node taking the file I/O lock every 10 ms, a writer's truncate and the header write right after it were split by the reader's turn: the write waited 80 to 110 ms, and readers that landed in between kept an unreadable tail cached until the next attribute refresh. Docker's staleness tests saw up to 3.7 s of lag, against a 3 s bound. A claim now lasts at least minTurn (50 ms) before that wait ends the turn, so such a write joins in about 1 ms. A hold that expires still yields at once, so a node that keeps claiming again still lets the waiter in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(operations): word the read guarantee as each write request whole, up to MaxWrite Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(fs): raise MaxWrite to 1 MiB through the shared constant #192 named the request size goclusterfs.MaxWrite and kept 128 KiB until this PR. The mount now sends write and read requests of up to 1 MiB, so reads and snapshots see each write of up to 1 MiB whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(lock): count a file I/O AcquireAs that gives up as failed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: keep the peer-truncate test and the AcquireAs give-up case working with file I/O locks #185's test paused a write before its size update and truncated the file from another node in between. Setattr now waits for the write's file I/O lock, so the truncate goes through setattrIOLocked, the part of Setattr that runs under that lock, the way a journal replay of a crashed node's truncate changes the size without it. The give-up case took a file I/O lock while holding the bitmap lock, which the lock order refuses; it now holds it through AcquireAs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(lock): check writer preference without wall-clock bounds The writer and reader starvation tests failed a 1.5 s bound on a loaded -race runner (one writer wait of 1.61 s). They now fail only when a node never gets the lock within the manager's own acquire timeout, which is how starvation shows: with writer preference off, the writer times out. A new test checks the deferral itself with lower bounds only: next to a live write mark, a read claim does not get the lock within one turn, and gets it no sooner than two turns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The API ran snapshot and clone requests by opening the file and calling
fsetxattron the daemon's own FUSE mount. A daemon thread waiting on its own mount is a structural deadlock risk.On 2026-11-13 on .66 it wedged:
fsetxattrkept the process a zombie, which held the host lock for 3 days.Fix:
internal/goclusterfs/api.golooks the path up inside the filesystem. It refuses a symlink anywhere on the path, as openat2'sRESOLVE_NO_SYMLINKSdid. It then callsFile.setxattr, the body ofSetxattr, with the same inode lock, generation check and cluster locks.Evidence, plainly: I could not reproduce the hang. With a 30 s stall watch, main passed this test about 110 times (.68 under load ~55, .66 capped). So this fix rests on removing the structural risk and on the tests below, not on a before-and-after reproduction. With the fix, the same loop ran 107 times in an hour, clean.
Write atomicity (PM decision):
i_rwsemkept a same-node snapshot from landing inside anywrite().goclusterfs.MaxWrite(128 KiB, which the mount now sets explicitly). A largerwrite()may be split, as it always could be from another node.goclusterfs.MaxWrite = 1 << 20and says "1 MiB".Also in this PR:
Checks:
-race: api, fileops, node, and goclusterfsPowerCut|Snapshot|Clone|APITestAPI|TestMount|TestDashboard|TestGfsctland the lock-reclaim tests🤖 Generated with Claude Code