Skip to content

fix(inode): lock each inode table sector around its read-modify-write - #81

Merged
mattmattox merged 12 commits into
mainfrom
fix/inode-sector-lock
Nov 13, 2026
Merged

mattmattox merged 12 commits into
mainfrom
fix/inode-sector-lock

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

writeInodeSector reads the 4 KB sector that holds 16 inodes, patches one, and writes all 16 back, under only that inode's lock. When two nodes write neighboring inodes, each writes back the other's older copy. That's main CI's "dataA.txt: got 0 bytes". A new leaf lock, LockTypeInodeSector (type 10, the highest), is now held from the sector read to the sector write. Every inode table write goes through writeInodeSector: create, setattr, size and extent updates, free (including #80's FreeIf), the allocator, local journal replay, and a peer replaying a dead node's journal.

Lock order is inode lock, then sector lock, then the table's local mutex. Taking the sector lock before the mutex keeps local Reads from waiting on a cluster lock.

Sector locks get their own 16 of #29's 256 lock rows, as agreed with the lock session. A sector lock is a leaf, so a cycle of waits between nodes can't pass through those rows. Without them, an inode lock and a sector lock could share a row across nodes and stall until the 30s timeout. This remaps every lock type, so old and new builds must not mount one volume at the same time (CHANGELOG upgrade note).

Results on .68, main de89d48 with and without this PR:

Test main this PR
TestNeighborInodeWritesFromTwoNodes (unit) fails 5/5 passes 50/50
TestMultiNodeNeighborInodeWrites (2 mounts, 256 files) passes 6/6 (failed 2/6 before #29) passes 10/10
TestSectorLocksDoNotWaitOnOtherLocks passes 50/50; fails 3/3 without the reserved rows
VM "Concurrent First Write Same Block", 10 runs passes 10/10
Integration -short 118 pass, 0 fail

The unit test needs a full-size lock region: with #29's 4-row 1 MB region the two nodes' inode locks share a row, which serializes the writes and hides the bug.

Before #29, lock claims weren't atomic, and this PR alone still failed about 1 in 10 integration runs. In one of those, both nodes allocated the same data block, so one file held another's bytes. #29 fixed that.

The unit test is the regression check. Since #29 the two-node integration test no longer hits the race on main either; it stays as end-to-end coverage of neighbor inode writes. Docker and VM variants are TODO, since shell loops over docker exec or ssh would not hit it.

Cost. Shell loops on .68 against a 2 GB file image, main de89d48 with and without this PR, 3 interleaved rounds, means. Node B mounts after the single-node phase; stat runs on node A over a 10-file directory.

main this PR change
1 node, 500 creates 32.5 s 35.1 s +8%
1 node, 500 4 KB overwrites 58.9 s 56.8 s −4%
2 nodes, 250 + 250 creates 11.1 s 9.3 s −16%
2 nodes, 250 + 250 overwrites 25.6 s 15.4 s −40%
stat, idle: p50 / p99 38 / 90 ms 38 / 91 ms
stat during 2-node writes: p50 / p99 51 / 192 ms 45 / 145 ms

The host's load average stayed near 400 from other sessions' runs, and rounds vary up to 3x, so the wall-clock cost is below what this host can measure. An earlier set against #29 at 793068e showed +15% for single-node creates and +21% for single-node overwrites. The lock counters show the added work: per round, node A makes about 3,800 sector lock acquires, and about 2,200 of them claim the row from disk (a slot write, a row read and a release write). Lock slot writes go up 27%, and 14 to 21 sector lock acquires per round had to wait.

🤖 Generated with Claude Code

mattmattox and others added 11 commits November 13, 2026 05:33
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
writeInodeSector reads the 4 KB sector that holds 16 inodes, patches one
and writes all 16 back, while holding only that inode's lock. Two nodes
writing neighboring inodes wrote back each other's older copies: a file
lost its size and the block its inode mapped.

A new leaf lock, LockTypeInodeSector, is held from the sector read to
the sector write. Every inode table write goes through writeInodeSector:
create, setattr, size and extent updates, free, the allocator, and
journal replay on this node or for a dead peer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All lock types hash onto the same 256 records, so node A could hold an
inode lock's record while waiting for a sector lock whose record node B
holds for another inode, and B the reverse, until the 30s timeout. A
sixteenth of the records is now reserved for sector locks. A sector
lock is a leaf, so its holder never waits, and no cycle of waits can
pass through those records.

This remaps every lock type to a record. No release has shipped, but
old and new builds mounting one volume would disagree on records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under #29 a record is a 256 KB row, so the test needs 256 records rather than 1 MB to have a reserved slice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Waiting for another node's sector lock while holding t.mu blocked every
inode read and write on this node. Writers now take the sector lock
after the inode lock and before t.mu; t.mu still serializes this node's
writers, which share the node's hold on the sector lock. Held across
FlushTree it stays a leaf: tree node writes take no locks, and freeing
released blocks runs after it is released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hashToRecord keeps the sector lock records on top of #29's 256 rows, and
FreeIf from #80 takes the sector lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With #29's 1MB region of 4 records, the two nodes' inode locks share a
record, which serializes their writes and hides the lost update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since #29 the two-node integration test passes on main too, 6 of 6 runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox marked this pull request as ready for review November 13, 2026 16:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit cddbaf5 into main Nov 13, 2026
13 checks passed
mattmattox added a commit that referenced this pull request Nov 13, 2026
* test(lock): reproduce a wait cycle between inode and extent locks

Two nodes each hold an inode lock and then want an extent lock that hashes
to the record of the other node's inode. Both follow type order, and both
wait until the timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): give each lock type its own records

Two nodes could each hold an inode lock and wait for an extent lock in the
record the other's inode lock held, then fail with EIO after the timeout.
Type order did not prevent it, because locks of different types shared
records.

Each type now has its own records, laid out in type order and sized by how
many keys of the type are held at once. checkLockOrder also refuses a lock
of a type the goroutine holds in a lower record, so a goroutine's records
only go up and waits between nodes cannot form a cycle. No production path
holds two locks of one type at once today. Lock regions smaller than a
volume's, which only tests use, still hash over all records.

The sector lock test from #81 becomes a test that no type waits on records
another type uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(superblock): refuse volumes whose lock records hash over all types

Nodes that map locks to records differently do not exclude each other. New
volumes set RequiredTypedLockRecords: older builds refuse them as an unknown
required flag, and mount refuses volumes without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(lock): note that adopted records can still close a wait cycle

A replaying node holds a failed node's adopted records until the replay
ends and takes its own locks in journal order, so the record order rule
does not cover waits on those records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fsck): refuse volumes without per-type lock records, as mount does

fsck phase 1 only rejected unknown required flags, so it checked a volume
mount refuses and then told fsck slot release to "run fsck first". Both now
use superblock.CheckRequiredFlags.

Acquire also refuses a lock type outside the record layout instead of
hashing it over all records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant