Skip to content

fix(api): run snapshot and clone requests inside the mount daemon - #192

Merged
mattmattox merged 4 commits into
mainfrom
claude/api-writes-in-process
Nov 17, 2026
Merged

mattmattox merged 4 commits into
mainfrom
claude/api-writes-in-process

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

The API ran snapshot and clone requests by opening the file and calling fsetxattr on the daemon's own FUSE mount. A daemon thread waiting on its own mount is a structural deadlock risk.

On 2026-11-13 on .66 it wedged:

  • During concurrent writes, the daemon stopped answering right after an API clone.
  • Its heartbeat stalled and it exited.
  • A thread stuck in fsetxattr kept the process a zombie, which held the host lock for 3 days.

Fix:

  • No API call goes through the mount now. internal/goclusterfs/api.go looks the path up inside the filesystem. It refuses a symlink anywhere on the path, as openat2's RESOLVE_NO_SYMLINKS did. It then calls File.setxattr, the body of Setxattr, with the same inode lock, generation check and cluster locks.
  • Kernel caches: afterwards, a background goroutine expires the kernel's cached attributes for the file and any cached miss for a clone's name. It logs a WARN past 5 s, and no response waits on it.
  • Replaced file: a file replaced between the lookup and the lock answers 409.

Evidence, plainly: I could not reproduce the hang. With a 30 s stall watch, main passed this test about 110 times (.68 under load ~55, .66 capped). So this fix rests on removing the structural risk and on the tests below, not on a before-and-after reproduction. With the fix, the same loop ran 107 times in an hour, clean.

Write atomicity (PM decision):

Also in this PR:

  • Stall watch: TestAPIWritesDuringFileIO now has one. A call past 30 s triggers a goroutine dump, kernel stacks and a FUSE abort, so a hang leaves evidence and frees the lock.
  • Slot-loss message: a pure heartbeat stall now says "Heartbeats to registry slot N failed for Xs". "(owner …)" appears only for a real takeover. The 11-13 log's "owner b8918de…" was the node's own UUID: heartbeat starvation, which fix(node): judge peers on this node's own clock, and keep registry scans off the heartbeat #151 fixed. There was no second node.

Checks:

  • lint
  • unit tests with -race: api, fileops, node, and goclusterfs PowerCut|Snapshot|Clone|API
  • integration: TestAPI|TestMount|TestDashboard|TestGfsctl and the lock-reclaim tests
  • the 6 VM power-failure tests

🤖 Generated with Claude Code

mattmattox and others added 4 commits November 16, 2026 22:50
TestAPIWritesDuringFileIO now watches each write, read and API call. One
that takes over 30 s makes the test send SIGUSR1 to the mount daemon for a
goroutine dump, save the kernel stacks of the daemon and the test, read the
FUSE connection's waiting count, and then abort the connection, so a hang
fails the test with evidence instead of leaving a zombie that holds the
host lock. GCFS_STALL_DIR keeps a copy of the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API opened the file and called fsetxattr on the daemon's own FUSE
mount, so a daemon thread waited on the kernel while the kernel waited on
the daemon. Under concurrent writes that wedged the daemon: its heartbeat
stopped, it lost its registry slot, and the process stayed a zombie with a
thread stuck in fsetxattr.

The API now finds the file by walking its directories inside the
filesystem, refusing a symlink anywhere on the path as openat2 did, and
calls File.setxattr, the body Setxattr runs, with the same inode lock,
generation check and cluster locks. Listing a file's snapshots uses the
same lookup, so no API call touches the mount. After a snapshot or clone,
a background goroutine expires the kernel's cached attributes of the file
and any cached miss for the clone's name, and logs a warning if that takes
over 5 s; no API response waits on the kernel. A file replaced between the
lookup and the lock answers ESTALE, which the API maps to 409.

Without the kernel's i_rwsem, the inode lock keeps each FUSE write request
whole against a snapshot. A write() larger than 128 KiB reaches the daemon
as several requests, and a snapshot can land between them, as one taken on
another node always could. API-CONFIG.md says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot or clone captures each FUSE write request whole. goclusterfs
now names the request size as MaxWrite (128 KiB, go-fuse's default, set
explicitly in the mount options), and the same-node test writes MaxWrite
bytes per version, so both follow a later change. OPERATIONS.md states
the guarantee; API-CONFIG.md links to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…talled

A node whose heartbeats failed for half the timeout stopped with "Node
registry slot N is no longer this node's (owner <its own UUID>)", which
reads like another node took the slot; it misled a debugging session. The
slot-lost handler now gets a SlotLoss, whose message names the cause:
"Heartbeats to registry slot N failed for Xs" for a stall, "(owner
<uuid>)" only when another node's UUID is in the slot, and plain words
when peers declared this node dead or emptied the slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 184c450 into main Nov 17, 2026
18 checks passed
mattmattox added a commit that referenced this pull request Nov 17, 2026
#192 named the request size goclusterfs.MaxWrite and kept 128 KiB until
this PR. The mount now sends write and read requests of up to 1 MiB, so
reads and snapshots see each write of up to 1 MiB whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 17, 2026
* feat(fs): hold a cluster file I/O lock for each whole read and write

A read request took no lock, so it could return part of another node's
write, or of a truncate. Each read, write, truncate and fallocate now
takes a per-file I/O lock for the whole request: shared for reads,
exclusive for changes, with a local stripe for the node's own requests.
Snapshots and clones take it shared. A reader waits behind a writer's
waiting mark for up to two turns, so readers on lower slots cannot shut
a writer out. The first read after each claim reads the inode again.

File I/O locks get 39 of the 256 lock records, so the record layout
changes: RequiredFileIOLocks marks new volumes, mount refuses older
ones, and each registry slot records its lock layout, which mount
checks against live nodes. FUSE requests now carry up to 1 MB, and
-direct-io bypasses the page cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(lock): keep a record nobody waits for, and skip stack walks for file I/O locks

Reads now take a lock per request, which showed two costs. Finding the
goroutine ID formats the caller's whole stack, about 55 us per read on
FUSE stacks: file I/O locks now go through AcquireAs and ReleaseAs with a
per-request owner. And a holder gave up every record at the end of each
250 ms turn and waited 50 ms even when no other node wanted it: it now
reads the row when the turn ends and starts a new turn if the row shows
no conflicting claim and no live mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(lock): take the file I/O rows from extent locks, not inode locks

With 96 inode rows, unrelated inodes shared records more often: in the
4-node probe, deletes ran 5 to 13% slower than main and inode lock waits
rose from 125% to about 150% of node time. Whole-block writes take no
extent locks now that requests carry 1 MB, so extents give up 32 rows and
inodes keep 128. Deletes then ran 429 to 451 files/s at 4 nodes (main
375 to 382), and overwrites of separate files from 4 nodes stayed within
noise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(node): spell out the lock layout check without a negated conjunction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): drop a cross-reference that pointed the wrong way

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): keep a new claim for 50 ms before a long wait ends the turn

A node that has seen another node waiting for a whole turn gave up a new
claim at its first release. With a reader on another node taking the file
I/O lock every 10 ms, a writer's truncate and the header write right
after it were split by the reader's turn: the write waited 80 to 110 ms,
and readers that landed in between kept an unreadable tail cached until
the next attribute refresh. Docker's staleness tests saw up to 3.7 s of
lag, against a 3 s bound.

A claim now lasts at least minTurn (50 ms) before that wait ends the turn,
so such a write joins in about 1 ms. A hold that expires still yields at
once, so a node that keeps claiming again still lets the waiter in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): word the read guarantee as each write request whole, up to MaxWrite

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(fs): raise MaxWrite to 1 MiB through the shared constant

#192 named the request size goclusterfs.MaxWrite and kept 128 KiB until
this PR. The mount now sends write and read requests of up to 1 MiB, so
reads and snapshots see each write of up to 1 MiB whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): count a file I/O AcquireAs that gives up as failed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: keep the peer-truncate test and the AcquireAs give-up case working with file I/O locks

#185's test paused a write before its size update and truncated the file
from another node in between. Setattr now waits for the write's file I/O
lock, so the truncate goes through setattrIOLocked, the part of Setattr
that runs under that lock, the way a journal replay of a crashed node's
truncate changes the size without it. The give-up case took a file I/O
lock while holding the bitmap lock, which the lock order refuses; it now
holds it through AcquireAs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): check writer preference without wall-clock bounds

The writer and reader starvation tests failed a 1.5 s bound on a loaded
-race runner (one writer wait of 1.61 s). They now fail only when a node
never gets the lock within the manager's own acquire timeout, which is
how starvation shows: with writer preference off, the writer times out.
A new test checks the deferral itself with lower bounds only: next to a
live write mark, a read claim does not get the lock within one turn, and
gets it no sooner than two turns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant