Repository navigation
fix(lock): claim locks through per-node slots and cache released holds - #29
Merged
Merged
Conversation
Taking a lock reads the record and then writes a claim, so two nodes that read a free record at the same moment could both win. A node now waits ClaimSettleDelay (20ms) after writing a claim or a renewal and re-reads the record; it keeps the lock only if its claim is still there. A shared add on a record that is already shared skips the wait: losing that race can only over-count readers. To keep the wait off hot paths, a node keeps a record for CacheGrace (250ms) after its last local holder releases it, so the next local acquire needs no disk I/O. A cached hold that is too weak for the request, lost, or older than renewAfter is released and acquired afresh. FS.Close releases cached holds. RetryMaxDelay drops to 100ms so waiters notice a released hold soon. Writes of whole 4KB sectors to an allocated, written, unshared block no longer take the per-block extent lock or read the sector first. FUSE cancels a request's context whenever the caller gets a signal; Go callers get SIGURG constantly, so lock waits now end only on a deadline or after DefaultTimeout. Torn reads of a record another node is writing are read again before the record is reported invalid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er held With cached holds, a node that crashes leaves every record it used in the last 250ms held, so its restarted mount waited 15s for its own earlier incarnation's locks. A record held exclusively under this node's registry slot by another UUID belongs to an earlier occupant of the slot, which the registry hands out only after that occupant failed, so it is taken at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node that kept reusing a record rejoined its cached hold before the grace ran out, so it never released it, and another node waited as long as the first stayed busy: a restarted mount waited while a peer kept creating files. A hold taken from disk more than CacheGrace ago is now released when its last local key drops, and this node waits YieldGap before claiming it again. RetryMaxDelay drops to 25ms so a waiter polls within that gap. A block allocated by the current write stays invisible to other nodes until the inode, whose lock the write holds, is committed, so writing it no longer takes the per-block extent lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g writes Claim confirmation stops two nodes from both taking a free inode lock, so the first-write test runs in short mode again. A new test has one node read a file while another fills it with fresh blocks, which it writes without the extent lock; the reader must see only holes, the old size or each block's own data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mounted-FS version could not see the file grow on the second node: its inode stays cached while the file is open when the cache broadcast is not running. Two FS instances on one image, with their own lock managers on a shared lock region and no inode cache, read the on-disk state every time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # tests/integration/multinode_test.go
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad node's locks #23 frees a crashed node's exclusive locks once it is proven dead, during the journal replay a restarted node runs before it serves, so the restarted mount no longer waits for its earlier incarnation's cached holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m writes A node's first shared claim failed its settle check when another node joined it during the wait, then re-joined on top, leaking one reader count until the lock went stale. A shared claim now also holds when the record is still shared at a newer sequence. A claim whose write would land more than half the settle delay after its read is dropped and retried, so a slow or queued write cannot overwrite a claim another node has already confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cord retries The 10ms check on a claim write ran before WriteAt took the device's I/O shard mutex, so a wait there went unmeasured. storage.Device.WriteAtBefore now checks the deadline once it holds the shard, right before the write. A write that still finishes more than 10ms after its read is not held, and the node leaves that record alone until it is stale, because the write may have replaced a claim another node already confirmed. Releases, renewals and dead-node takeovers no longer retry forever on storage that keeps missing the deadline: they back off with jitter and give up after 10 tries. Adds the PM's repro tests: two nodes reading one directory, or the refcounts, at once must not delay the next create, unlink, snapshot create, copy-on-write or snapshot delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te lock write Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n parallel A read claim whose write lands late can sit on top of a write claim another node has just confirmed. Left as a read claim, it let further readers in beside that writer. The node now rewrites it as its own write claim before leaving the record to go stale. ReleaseCached frees kept records eight at a time, so an unmount on slow storage no longer pays each record's retries one after another. The two CAS test hooks become one injectable record write, so tests can model a write that passes its deadline check and then lands late. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fence that turns a late read claim into this node's write claim used the normal 10ms write deadline, so on still-slow storage it was skipped and the record stayed open to readers beside a writer. The fence can only make the record exclusive to this node, so it now writes without a deadline. It fences the record as it is now, so readers that joined before it are shut out from then on too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…f the file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unbounded fence write that stalled past StaleThreshold could land on a claim another node made after taking the record over as stale. The fence now skips a record that is already stale, and its write must start within fenceWindow (StaleThreshold/2) of its read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ey test it On the CI runner's disk a 4KB O_DIRECT lock record write took 25-27ms, so every claim counted as late, froze its record for 15s, and 12 tests timed out. Each Manager now carries its write window. The lock tests' TestMain relaxes the default, and the tests of the window itself restore the real 10ms on the nodes whose writes they slow down. CI also runs unit test images on tmpfs, checking first that tmpfs takes O_DIRECT writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
#33) Main's Docker group 1 fails this test with EIO. On .68 the Go version of the test failed 1 run in 10 the same way: both nodes found inode 5's lock record free, both took it, and both claimed inode 5 for a snapshot's data, so the second write saw the inode change under it. #29 confirms lock claims and re-enables all three. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…confirmed #33 skipped them in Docker, VM and Go -short because two nodes could both claim one free inode's lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its skip blamed a race in the distributed lock. Two nodes allocating inodes and blocks at once go through the lock claims #29 now confirms, and the test passed 23 of 23 runs on this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
- Move the doc comments the new functions separated from their owners. - Skip the two-node stress test in full mode too until #29. - Test the release-and-retry path when a write, copy, truncate or append lands between counting the blocks and marking the file shared. - A snapshot or clone that keeps losing that race fails with EAGAIN. - The in-place re-check retries only when the block moved or is gone, and returns a corrupt tree as EIO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Oct 24, 2026
File operations ignore interrupts like an NFS hard mount; a lock wait that cannot finish fails with EIO after DefaultTimeout (from #29). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
marked this pull request as ready for review
November 13, 2026 11:30
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
Two nodes can both hold the root directory lock and lose an entry, which fails this test in about one of every two or three main CI runs. Skip it in the Go, Docker and VM suites until #29 confirms lock claims. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- Whole-sector writes to existing blocks take the extent lock again. A peer's read-modify-write of the same sector could otherwise write back a stale copy over an acknowledged write. - Once a hold's turn is over, new local requests wait for the node's holders to drain, so a record whose holds keep overlapping is still yielded. Only a goroutine that holds other locks stops waiting after another turn. - A record adopted in this node's own slot keeps its adopted write claim through the replay's own acquires and releases until ReleaseAdopted. - Adopted claims are flagged, and every claimant defers to them, so a waiter in a lower slot cannot keep the replaying node out. - ReleaseAdopted leaves a slot that a new node has written since. - A slot that stays torn for StaleThreshold stops blocking its record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#79 skipped TestMultiNodeConcurrentDirectoryOps in the Go, Docker and VM suites until lock claims are confirmed. This branch confirms them, so the test runs in all three again, in Docker without the SKIP_CONCURRENT_TESTS gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lock region With the 1MB test region the two writes' other locks share one of 4 records, which serializes them and hides the lost update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
Under #29 a record is a 256 KB row, so the test needs 256 records rather than 1 MB to have a reserved slice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The whole-sector test raced two writers through a gate, and on fast storage the partial write finished before the other woke, so it passed with the bug. A hook after the partial write's sector read now runs the whole-sector write while the old sector is held. The overlapping-hold test gets a case with holds longer than a turn, which only an unbounded drain wait passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bytes A node kept the time it first read a slot torn until it read that slot whole again. Long after it stopped reading the record, a torn read of a live node's renewal in flight looked expired, and both nodes held the record. The wait now restarts whenever the torn bytes change. They are compared by FNV-1a, since CRC32C gives every slot torn the same way one value. OPERATIONS.md no longer says whole-sector writes skip the block lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e/lock-claim-confirm-9658e1
… shares its record A write into an existing block holds the inode lock shared and then takes the block's extent lock exclusive. When both keys hash to one lock record, the lock manager sees a shared-to-exclusive upgrade and the write fails with EIO. Main hits this only on partial-sector writes; #29 makes every write to an existing block take the extent lock, so the Docker "Snapshot During Writes" test failed when inode 130 and block 163 both mapped to record 36. The Docker test now prints dd's error and the node's write errors when the writer fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
With claims through per-node slots, 3 short and 8 full-size runs on .68 passed with no lost entries, no lock give-ups and no double holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
With #29's 1MB region of 4 records, the two nodes' inode locks share a record, which serializes their writes and hides the lost update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
… as ENOSPC (#66) * fix(fs): report a failed inode claim as EIO, not ENOSPC Create, mkdir and clone turned every ClaimSlot error other than a slot collision into ENOSPC. A lock wait that gives up after 30 seconds is the usual cause, and the inode table is not full then. FindFreeSlot alone reports a full table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dir): use every 4KB chunk of a directory block New entries only went into a block's first 4KB chunk, so a 1MB block held about 250 short or 15 long names, and a busy directory grew a block every few creates. Each lookup then read every block 4KB at a time while holding the directory lock. Entries now fill the chunks in order, a block is read in one I/O, and an add or remove writes only the chunk it changed. Existing blocks keep working: their unused chunks start with RecLen 0. fsck checks each chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: name lost directory entries and check fsck on multi-chunk directories The one-directory create test lists the created files it cannot find. It skips, not fails, when a lock release found another node holding the lock, since two holders of the directory lock can overwrite each other's chunk until #29 lands. The runtime fsck test now fills and unlinks across several chunks of the root directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: skip one-directory create failures only when #29 lock races explain them A create also failed with EIO once in nine runs: a lock record read during another node's write failed its checksum. #29's per-node slots treat such a read as a held lock. The test now collects create errors and missing entries, and skips only when the logs show a double hold or a torn lock record; lock give-ups and anything else still fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: excuse each one-directory create failure only by its own #29 race Missing entries need a double hold in the logs and failed creates need a torn lock read; any other combination fails the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: skip the one-directory create test until #29 CI hit the other #29 failure: a reader count lost by a racing claim kept the directory lock read-locked, so two creates gave up after 30s. With creates now fast, every #29 race shows up in this test, so it is skipped like TestSnapshotsDuringWritesAcrossNodes and fails on any lost entry or create error once enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(snapshot): report a failed inode claim as EIO, not ENOSPC The snapshot path still turned every claimInode error into ENOSPC. claimErrno now also maps ErrInodeFull to ENOSPC, so all callers share it. CHANGELOG notes the errno change, the directory capacity, and that an old binary's remove can corrupt a multi-chunk directory block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: re-enable the one-directory create test now that #29 is in With claims through per-node slots, 3 short and 8 full-size runs on .68 passed with no lost entries, no lock give-ups and no double holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
Since #29 the two-node integration test passes on main too, 6 of 6 runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
…#81) * test: two nodes update neighboring inodes in one table sector Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(inode): lock each inode table sector around its read-modify-write writeInodeSector reads the 4 KB sector that holds 16 inodes, patches one and writes all 16 back, while holding only that inode's lock. Two nodes writing neighboring inodes wrote back each other's older copies: a file lost its size and the block its inode mapped. A new leaf lock, LockTypeInodeSector, is held from the sector read to the sector write. Every inode table write goes through writeInodeSector: create, setattr, size and extent updates, free, the allocator, and journal replay on this node or for a dead peer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: two nodes write files whose inodes share table sectors Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: track the inode sector tests in the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): give inode table sector locks their own records All lock types hash onto the same 256 records, so node A could hold an inode lock's record while waiting for a sector lock whose record node B holds for another inode, and B the reverse, until the 30s timeout. A sixteenth of the records is now reserved for sector locks. A sector lock is a leaf, so its holder never waits, and no cycle of waits can pass through those records. This remaps every lock type to a record. No release has shipped, but old and new builds mounting one volume would disagree on records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: size the sector lock test's region in records Under #29 a record is a 256 KB row, so the test needs 256 records rather than 1 MB to have a reserved slice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(inode): take the sector lock before the table mutex Waiting for another node's sector lock while holding t.mu blocked every inode read and write on this node. Writers now take the sector lock after the inode lock and before t.mu; t.mu still serializes this node's writers, which share the node's hold on the sector lock. Held across FlushTree it stays a leaf: tree node writes take no locks, and freeing released blocks runs after it is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: note in the changelog that lock placement changed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(inode): run the neighbor inode test with a full lock region With #29's 1MB region of 4 records, the two nodes' inode locks share a record, which serializes their writes and hides the lost update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: say the neighbor inode unit test is the regression check Since #29 the two-node integration test passes on main too, 6 of 6 runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
…k region (#88) * test(goclusterfs): give the snapshot and lock test helpers a full lock region Since #29 a lock record is a 256 KB row, so the snapshot helper's 512 KB region held 2 records and the lock-commit helper's 1 MB held 4. Locks of different types shared records, a claim went stale after 15s, and TestCopyOnWriteKeepsConcurrentInPlaceWrite could read old data. Both helpers now use superblock.LockRegionSize. The test that needs an inode and a block lock in one record keeps a 4-record region. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(goclusterfs): release cached lock holds before a test device closes A hold still cached when the device closes stays claimed until it goes stale, so the next node waited 15s. Taken from #87. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
…refused (#111) The On-Disk Layout section already describes format v3. It listed the v3 images mount refuses for the per-node lock slot change (#29) but not for the typed lock records flag (#85), nor the error older builds give on newer images. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Taking a lock was a read then a write, so two nodes claiming a free record at once could both win. Now each lock record is a row of 64 claim slots of 4KB, one per registry slot, and a node writes only its own slot. A claim writes the node's slot, then reads the whole row in one I/O, and wins if no other live slot conflicts.
Why two conflicting claims cannot both win. Of two claimants, the second to finish its slot write starts its row read after the first's write finished, so it sees that claim and loses. No timing bound is needed, so slow storage can delay a lock but cannot freeze it (#29's settle-wait design froze records for 15s on slow CI runners). It assumes:
Tie-break. When claims collide, the lower slot keeps its claim and waits. A node that sees a conflict in a lower slot clears its own and reads the row before writing again. Waits only point to higher slots, so there is no cycle. A dead claimant's slot blocks only until it is 15s old, or until
AdoptOwnerandReleaseAdoptedhandle it.Dead nodes.
AdoptOwnerrewrites a dead node's write and torn slots with a fresh time and marks them adopted, so peers stay off during the replay. Other nodes treat an adopted slot as a lower slot, so the replaying node gets those records ahead of waiting peers. It clears the dead node's read slots at once.ReleaseAdoptedclears the rest after the replay, and only slots still adopted for that dead node.Format. The superblock now stores the lock slot size at offset 264. Mount refuses v3 images formatted by current main and says to reformat. The lock region grows from 1MB to 64MB, so the smallest device is now 327MB, up from 264MB. No release has shipped with the old layout.
Kept from the old #29:
Also fixed: every write into an existing block now takes the block's extent lock. Main skipped it for whole-sector writes, and a peer's read-modify-write of the same sector could erase them. When the inode lock and the block's lock share a record, the write takes the inode lock exclusive, because a shared hold cannot then take the block's lock.
Dropped: the settle wait, the late-claim freeze, reader bits, the background verifier and the write-deadline storage calls. The API's lock hint-cache fields are now always 0.
Evidence:
-raceunit suite, lint and parity pass on .68.Cost vs main, on .68 with 3 alternating rounds:
-shortsuite, 8f2c405 vs main 292a5ec: 1.00× over 111 tests (pairs 0.96, 0.99, 1.08).Not covered: a holder paused for more than 15s keeps writing after another node takes its lock. This is the next PR, a pre-write ownership check.
🤖 Generated with Claude Code