Repository navigation
fix(snapshot): order in-place writes with snapshots on other nodes - #42
Merged
Merged
Conversation
A write decides to write a block in place from an inode it read without the inode lock. A snapshot or clone on another node could mark the file shared and count the new owner between that decision and the data write, so the snapshot then showed data written after it was made. Snapshots and clones now mark the file shared and count the owners under its exclusive inode lock, holding the refcount lock first as lock order requires. In-place writes take the shared inode lock and check again that the block maps to the same place and has no other owner, else they retry and copy it. Concurrent writers to one file still run in parallel, since the check looks at the block, not MetaGeneration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Holding the refcount lock while waiting for the inode lock could take longer than the lock manager lets a hold be reused, so Share hit the lock-order check and the snapshot failed with EIO under write load. Snapshots now count the owners first and then mark the file shared under the exclusive inode lock, undoing the counts and retrying if a write copied a block in between. The stress test skips in -short mode and in Docker and VM until #29: two nodes can both hold the refcount lock, and a lost count lets a later write go in place into a block a snapshot shares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Move the doc comments the new functions separated from their owners. - Skip the two-node stress test in full mode too until #29. - Test the release-and-retry path when a write, copy, truncate or append lands between counting the blocks and marking the file shared. - A snapshot or clone that keeps losing that race fails with EAGAIN. - The in-place re-check retries only when the block moved or is gone, and returns a corrupt tree as EIO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep main's refcount coverage check: shareSource refuses blocks past the refcount region before it counts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
…irmed #42 skipped it in Go, and its Docker and VM versions, because two nodes could both hold the refcount lock. The merge resolution re-enables the shell ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
#29) * feat(lock): confirm claims after a settle wait and cache released holds Taking a lock reads the record and then writes a claim, so two nodes that read a free record at the same moment could both win. A node now waits ClaimSettleDelay (20ms) after writing a claim or a renewal and re-reads the record; it keeps the lock only if its claim is still there. A shared add on a record that is already shared skips the wait: losing that race can only over-count readers. To keep the wait off hot paths, a node keeps a record for CacheGrace (250ms) after its last local holder releases it, so the next local acquire needs no disk I/O. A cached hold that is too weak for the request, lost, or older than renewAfter is released and acquired afresh. FS.Close releases cached holds. RetryMaxDelay drops to 100ms so waiters notice a released hold soon. Writes of whole 4KB sectors to an allocated, written, unshared block no longer take the per-block extent lock or read the sector first. FUSE cancels a request's context whenever the caller gets a signal; Go callers get SIGURG constantly, so lock waits now end only on a deadline or after DefaultTimeout. Torn reads of a record another node is writing are read again before the record is reported invalid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(operations): document the lock settle wait, cache grace and I/O limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(lock): benchmark an acquire that cannot reuse a cached hold Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): let a node take the locks its registry slot's previous owner held With cached holds, a node that crashes leaves every record it used in the last 250ms held, so its restarted mount waited 15s for its own earlier incarnation's locks. A record held exclusively under this node's registry slot by another UUID belongs to an earlier occupant of the slot, which the registry hands out only after that occupant failed, so it is taken at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): yield records a busy node has held past the cache grace A node that kept reusing a record rejoined its cached hold before the grace ran out, so it never released it, and another node waited as long as the first stayed busy: a restarted mount waited while a peer kept creating files. A hold taken from disk more than CacheGrace ago is now released when its last local key drops, and this node waits YieldGap before claiming it again. RetryMaxDelay drops to 25ms so a waiter polls within that gap. A block allocated by the current write stays invisible to other nodes until the inode, whose lock the write holds, is committed, so writing it no longer takes the per-block extent lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: un-skip the first-write test and check readers during allocating writes Claim confirmation stops two nodes from both taking a free inode lock, so the first-write test runs in short mode again. A new test has one node read a file while another fills it with fresh blocks, which it writes without the extent lock; the reader must see only holes, the old size or each block's own data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fs): check a reader on another node never sees uncommitted blocks The mounted-FS version could not see the file grow on the second node: its inode stays cached while the file is open when the cache broadcast is not running. Two FS instances on one image, with their own lock managers on a shared lock region and no inode cache, read the on-disk state every time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(lock): drop the slot-owner takeover now that #23 adopts a dead node's locks #23 frees a crashed node's exclusive locks once it is proven dead, during the journal replay a restarted node runs before it serves, so the restarted mount no longer waits for its earlier incarnation's cached holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): keep shared claims that other readers join, drop slow claim writes A node's first shared claim failed its settle check when another node joined it during the wait, then re-joined on top, leaking one reader count until the lock went stale. A shared claim now also holds when the record is still shared at a newer sequence. A claim whose write would land more than half the settle delay after its read is dropped and retried, so a slow or queued write cannot overwrite a claim another node has already confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(goclusterfs): name the extent-lock skip condition Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): check claim write deadlines after the shard wait, bound record retries The 10ms check on a claim write ran before WriteAt took the device's I/O shard mutex, so a wait there went unmeasured. storage.Device.WriteAtBefore now checks the deadline once it holds the shard, right before the write. A write that still finishes more than 10ms after its read is not held, and the node leaves that record alone until it is stale, because the write may have replaced a claim another node already confirmed. Releases, renewals and dead-node takeovers no longer retry forever on storage that keeps missing the deadline: they back off with jitter and give up after 10 tries. Adds the PM's repro tests: two nodes reading one directory, or the refcounts, at once must not delay the next create, unlink, snapshot create, copy-on-write or snapshot delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(ops): note late read claims beside a writer and the cost of a late lock write Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): close late read claims to other readers, free kept locks in parallel A read claim whose write lands late can sit on top of a write claim another node has just confirmed. Left as a read claim, it let further readers in beside that writer. The node now rewrites it as its own write claim before leaving the record to go stale. ReleaseCached frees kept records eight at a time, so an unmount on slow storage no longer pays each record's retries one after another. The two CAS test hooks become one injectable record write, so tests can model a write that passes its deadline check and then lands late. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): write the late read claim fence without a deadline The fence that turns a late read claim into this node's write claim used the normal 10ms write deadline, so on still-slow storage it was skipped and the record stayed open to readers beside a writer. The fence can only make the record exclusive to this node, so it now writes without a deadline. It fences the record as it is now, so readers that joined before it are shut out from then on too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(storage): check deferred close and remove errors like the rest of the file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): bound the late read claim fence to half the stale threshold An unbounded fence write that stalled past StaleThreshold could land on a claim another node made after taking the record over as stale. The fence now skips a record that is already stale, and its write must start within fenceWindow (StaleThreshold/2) of its read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(lock): keep unit tests off the 10ms record write limit unless they test it On the CI runner's disk a 4KB O_DIRECT lock record write took 25-27ms, so every claim counted as late, froze its record for 15s, and 12 tests timed out. Each Manager now carries its write window. The lock tests' TestMain relaxes the default, and the tests of the window itself restore the real 10ms on the nodes whose writes they slow down. CI also runs unit test images on tmpfs, checking first that tmpfs takes O_DIRECT writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run concurrent snapshot creates again now that lock claims are confirmed #33 skipped them in Docker, VM and Go -short because two nodes could both claim one free inode's lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): run concurrent file creation in -short mode Its skip blamed a race in the distributed lock. Two nodes allocating inodes and blocks at once go through the lock claims #29 now confirms, and the test passed 23 of 23 runs on this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lock): log who holds the record when an acquire gives up Operators need to see which node holds a lock that made an operation fail. The timeout now logs at WARN with the lock key and the record's holder, mode and age. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run snapshots during writes again now that lock claims are confirmed #42 skipped it in Go, and its Docker and VM versions, because two nodes could both hold the refcount lock. The merge resolution re-enables the shell ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(goclusterfs): retry a write up to 10 times when other nodes change the file A node writing a file while another node snapshots it back to back loses the copy-on-write check each time a snapshot lands. With the 20ms lock claim wait those windows are wider, and 3 tries made TestSnapshotsDuringWritesAcrossNodes fail 4 of 10 runs with EAGAIN. With 10 tries it passed 20 of 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run integration test images on tmpfs and allow 8 minutes Lock record writes on the runner's disk can miss their 10ms limit and stall a lock for 15s, which pushed the -short suite past its 5 minute timeout. Main's suite already takes about 4 minutes, and lock claims now wait 20ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lock): measure where the time of late lock record writes goes Each skipped or late record write now logs its time split into waits inside this process (the device's shard lock, the gap between the read and the write call) and the read and write system calls. Each node counts them and logs the totals at unmount. CI integration tests go back to the runner's own disk, so these numbers show what users on ordinary disks would see. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(lock): log acquire counts per lock type and a write gap histogram At unmount each node logs, per lock type, how its acquires were served (held key, joined record, cached record, fresh claim with or without the settle wait, blocked tries), and a histogram of the time from a record CAS's read to its write finishing. These are the numbers ClaimSettleDelay and CacheGrace are chosen from. Docker CI keeps each node's lock log lines with its test results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): read claims skip the settle wait and are not kept after release A node killed within 250ms of a read left its cached read hold on disk, and other nodes' writers waited 15s for it to go stale (TestNetworkPartition: 3s on main, 17s here). Read claims now release as soon as their last local reader does, and they no longer wait out ClaimSettleDelay. Only write claims need that wait: every record write still finishes within half the settle delay of its read, so a read claim that replaced a writer's claim lands before the writer's confirm re-read, and the writer backs off. A read can still run beside a writer that won the race, as on main; two writers cannot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): give up a claim write only if it finished after the settle delay A claim write can replace a claim another node already confirmed only if it lands more than ClaimSettleDelay after its own read: until then, the other node's confirm re-read is still ahead. The 15s give-up now applies past the full delay, not half of it, and the margin moves to the start: a write that cannot start within a quarter of the delay is skipped and retried, which is cheap. On CI, both late writes in run 42863343192/42863342671 (10ms, 16.5ms) would have been held. Lock record reads and writes also stop taking the device's shard lock, so a node's own data and journal I/O cannot delay them: 2 of the 6 missed writes on CI waited about 10ms for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): name readers with per-node bits and confirm read claims A read lock's holder count was updated by read-modify-write from several nodes, so racing claims and releases lost updates. A lost release left the count too high, and writers waited 15s for the record to go stale (TestMultiNodeConcurrentDirectoryOps, ...CreatesDoNotCollide on 9af90ae). Each read-locked record now has a bit per registry slot in its reserved bytes. A read claim sets this node's bit and waits out ClaimSettleDelay like a write claim, then keeps the lock only if its bit is still set. A release clears the bit and checks again in the background that no racing update put it back. Read holds are cached again. AdoptOwner, given the dead node's registry slot, clears its reader bits, so a reader killed while it kept a hold blocks writers only until the node is proven dead. Every claim writes the current time, since readers' bits now clear on their own. Late claims (a claim write that finished after the settle delay) are logged with their lock key and latency and counted at unmount. Lock records change meaning in the reserved bytes (bits 64:72), which goes with on-disk format v3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: say 20ms in the unit job's tmpfs comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): stop waiting on lock hint counters and retry reads past attr timeout The lock hint cache is gone, so its counters stay 0. A restarted node can cache the size of a peer's file it looked up mid-write for -attr-timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): tie the verifyFiles retry to the node's -attr-timeout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): keep idle holds 10ms and cap busy holds with a separate 250ms turn An idle cached hold kept other nodes waiting up to 250ms. With the root directory and the block bitmap both hot, two nodes waited on each other's idle holds and concurrent creates ran 2.7x slower than main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: add the idle cached hold test to the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): address the #29 review - Whole-sector writes to existing blocks take the extent lock again. A peer's read-modify-write of the same sector could otherwise write back a stale copy over an acknowledged write. - Once a hold's turn is over, new local requests wait for the node's holders to drain, so a record whose holds keep overlapping is still yielded. Only a goroutine that holds other locks stops waiting after another turn. - A record adopted in this node's own slot keeps its adopted write claim through the replay's own acquires and releases until ReleaseAdopted. - Adopted claims are flagged, and every claimant defers to them, so a waiter in a lower slot cannot keep the replaying node out. - ReleaseAdopted leaves a slot that a new node has written since. - A slot that stays torn for StaleThreshold stops blocking its record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(goclusterfs): run the whole-sector lost-update test with a full lock region With the 1MB test region the two writes' other locks share one of 4 records, which serializes them and hides the lost update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: make the lost-update and overlapping-hold tests catch their bugs The whole-sector test raced two writers through a gate, and on fast storage the partial write finished before the other woke, so it passed with the bug. A hook after the partial write's sector read now runs the whole-sector write while the old sector is held. The overlapping-hold test gets a case with holds longer than a turn, which only an unbounded drain wait passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): expire a torn slot only while it stays torn with the same bytes A node kept the time it first read a slot torn until it read that slot whole again. Long after it stopped reading the record, a torn read of a live node's renewal in flight looked expired, and both nodes held the record. The wait now restarts whenever the torn bytes change. They are compared by FNV-1a, since CRC32C gives every slot torn the same way one value. OPERATIONS.md no longer says whole-sector writes skip the block lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(goclusterfs): take the inode lock exclusive when the block's lock shares its record A write into an existing block holds the inode lock shared and then takes the block's extent lock exclusive. When both keys hash to one lock record, the lock manager sees a shared-to-exclusive upgrade and the write fails with EIO. Main hits this only on partial-sector writes; #29 makes every write to an existing block take the extent lock, so the Docker "Snapshot During Writes" test failed when inode 130 and block 163 both mapped to record 36. The Docker test now prints dd's error and the node's write errors when the writer fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The race is real. On main, the new deterministic test fails: node A pauses after deciding to write a block in place, node B snapshots the file, and the snapshot then shows A's data. That holds both for a never-shared block and for one shared earlier. The real-binary stress test also caught it on main (3 failures in 25 runs).
The remaining stress failures (1 in 20 on this branch) follow a refcount-lock "not the lock owner" event. That is #29's double-holder race losing a snapshot's count, so the stress test skips in -short mode and in Docker/VM until #29. Nothing in internal/lock changed.
Cost: one shared inode lock round-trip per in-place block write.
Not changed here: writes give up with EAGAIN after 3 retries under frequent snapshots, as on main.
Tested on .68: lint, unit -race, integration -short (88 passed / 17 skipped).
🤖 Generated with Claude Code