Repository navigation
fix(inode): lock each inode table sector around its read-modify-write - #81
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
writeInodeSector reads the 4 KB sector that holds 16 inodes, patches one and writes all 16 back, while holding only that inode's lock. Two nodes writing neighboring inodes wrote back each other's older copies: a file lost its size and the block its inode mapped. A new leaf lock, LockTypeInodeSector, is held from the sector read to the sector write. Every inode table write goes through writeInodeSector: create, setattr, size and extent updates, free, the allocator, and journal replay on this node or for a dead peer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All lock types hash onto the same 256 records, so node A could hold an inode lock's record while waiting for a sector lock whose record node B holds for another inode, and B the reverse, until the 30s timeout. A sixteenth of the records is now reserved for sector locks. A sector lock is a leaf, so its holder never waits, and no cycle of waits can pass through those records. This remaps every lock type to a record. No release has shipped, but old and new builds mounting one volume would disagree on records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under #29 a record is a 256 KB row, so the test needs 256 records rather than 1 MB to have a reserved slice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Waiting for another node's sector lock while holding t.mu blocked every inode read and write on this node. Writers now take the sector lock after the inode lock and before t.mu; t.mu still serializes this node's writers, which share the node's hold on the sector lock. Held across FlushTree it stays a leaf: tree node writes take no locks, and freeing released blocks runs after it is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With #29's 1MB region of 4 records, the two nodes' inode locks share a record, which serializes their writes and hides the lost update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since #29 the two-node integration test passes on main too, 6 of 6 runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
marked this pull request as ready for review
November 13, 2026 16:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
* test(lock): reproduce a wait cycle between inode and extent locks Two nodes each hold an inode lock and then want an extent lock that hashes to the record of the other node's inode. Both follow type order, and both wait until the timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(lock): give each lock type its own records Two nodes could each hold an inode lock and wait for an extent lock in the record the other's inode lock held, then fail with EIO after the timeout. Type order did not prevent it, because locks of different types shared records. Each type now has its own records, laid out in type order and sized by how many keys of the type are held at once. checkLockOrder also refuses a lock of a type the goroutine holds in a lower record, so a goroutine's records only go up and waits between nodes cannot form a cycle. No production path holds two locks of one type at once today. Lock regions smaller than a volume's, which only tests use, still hash over all records. The sector lock test from #81 becomes a test that no type waits on records another type uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(superblock): refuse volumes whose lock records hash over all types Nodes that map locks to records differently do not exclude each other. New volumes set RequiredTypedLockRecords: older builds refuse them as an unknown required flag, and mount refuses volumes without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(lock): note that adopted records can still close a wait cycle A replaying node holds a failed node's adopted records until the replay ends and takes its own locks in journal order, so the record order rule does not cover waits on those records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fsck): refuse volumes without per-type lock records, as mount does fsck phase 1 only rejected unknown required flags, so it checked a volume mount refuses and then told fsck slot release to "run fsck first". Both now use superblock.CheckRequiredFlags. Acquire also refuses a lock type outside the record layout instead of hashing it over all records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
writeInodeSectorreads the 4 KB sector that holds 16 inodes, patches one, and writes all 16 back, under only that inode's lock. When two nodes write neighboring inodes, each writes back the other's older copy. That's main CI's "dataA.txt: got 0 bytes". A new leaf lock,LockTypeInodeSector(type 10, the highest), is now held from the sector read to the sector write. Every inode table write goes throughwriteInodeSector: create, setattr, size and extent updates, free (including #80'sFreeIf), the allocator, local journal replay, and a peer replaying a dead node's journal.Lock order is inode lock, then sector lock, then the table's local mutex. Taking the sector lock before the mutex keeps local
Reads from waiting on a cluster lock.Sector locks get their own 16 of #29's 256 lock rows, as agreed with the lock session. A sector lock is a leaf, so a cycle of waits between nodes can't pass through those rows. Without them, an inode lock and a sector lock could share a row across nodes and stall until the 30s timeout. This remaps every lock type, so old and new builds must not mount one volume at the same time (CHANGELOG upgrade note).
Results on .68, main de89d48 with and without this PR:
TestNeighborInodeWritesFromTwoNodes(unit)TestMultiNodeNeighborInodeWrites(2 mounts, 256 files)TestSectorLocksDoNotWaitOnOtherLocks-shortThe unit test needs a full-size lock region: with #29's 4-row 1 MB region the two nodes' inode locks share a row, which serializes the writes and hides the bug.
Before #29, lock claims weren't atomic, and this PR alone still failed about 1 in 10 integration runs. In one of those, both nodes allocated the same data block, so one file held another's bytes. #29 fixed that.
The unit test is the regression check. Since #29 the two-node integration test no longer hits the race on main either; it stays as end-to-end coverage of neighbor inode writes. Docker and VM variants are TODO, since shell loops over docker exec or ssh would not hit it.
Cost. Shell loops on .68 against a 2 GB file image, main de89d48 with and without this PR, 3 interleaved rounds, means. Node B mounts after the single-node phase;
statruns on node A over a 10-file directory.stat, idle: p50 / p99statduring 2-node writes: p50 / p99The host's load average stayed near 400 from other sessions' runs, and rounds vary up to 3x, so the wall-clock cost is below what this host can measure. An earlier set against #29 at 793068e showed +15% for single-node creates and +21% for single-node overwrites. The lock counters show the added work: per round, node A makes about 3,800 sector lock acquires, and about 2,200 of them claim the row from disk (a slot write, a row read and a release write). Lock slot writes go up 27%, and 14 to 21 sector lock acquires per round had to wait.
🤖 Generated with Claude Code