Repository navigation
fix(lock): give each lock type its own records - #85
Merged
Merged
Conversation
Two nodes each hold an inode lock and then want an extent lock that hashes to the record of the other node's inode. Both follow type order, and both wait until the timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two nodes could each hold an inode lock and wait for an extent lock in the record the other's inode lock held, then fail with EIO after the timeout. Type order did not prevent it, because locks of different types shared records. Each type now has its own records, laid out in type order and sized by how many keys of the type are held at once. checkLockOrder also refuses a lock of a type the goroutine holds in a lower record, so a goroutine's records only go up and waits between nodes cannot form a cycle. No production path holds two locks of one type at once today. Lock regions smaller than a volume's, which only tests use, still hash over all records. The sector lock test from #81 becomes a test that no type waits on records another type uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodes that map locks to records differently do not exclude each other. New volumes set RequiredTypedLockRecords: older builds refuse them as an unknown required flag, and mount refuses volumes without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A replaying node holds a failed node's adopted records until the replay ends and takes its own locks in journal order, so the record order rule does not cover waits on those records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fsck phase 1 only rejected unknown required flags, so it checked a volume mount refuses and then told fsck slot release to "run fsck first". Both now use superblock.CheckRequiredFlags. Acquire also refuses a lock type outside the record layout instead of hashing it over all records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
…hanges (#99) * docs: sync operator guides with main after the API, lock and layout changes README, OPERATIONS, DEPLOYMENT, TROUBLESHOOTING and API-CONFIG still said there was no metrics endpoint, that cache hit rates were wrong, that a broadcast failure only logged a warning, that lock waits always ended in EIO after 30 seconds, that the inode count was fixed at 4096, and listed env vars that do not exist. They now cover the HTTP API, gfsctl and /metrics, the unicast port and the broadcast startup check, lock wait limits for live and failed holders, the v2 and pre-#85 format errors, when -skip-recovery is safe, the log lines that show whether recovery ran, what SIGINT and SIGTERM do at shutdown, and how to run the tests -short skips. Commands and log lines were checked on a test host at 30500b7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: note that overwrites retry instead of failing on a failed holder Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
…refused (#111) The On-Disk Layout section already describes format v3. It listed the v3 images mount refuses for the per-node lock slot change (#29) but not for the typed lock records flag (#85), nor the error older builds give on newer images. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two nodes could deadlock until the 30s timeout: each holds an inode lock and wants an extent lock that hashes to the other's inode record. Type order didn't help because all types shared records (found in #29's review).
Each type now has its own records in type order, and
checkLockOrderalso refuses a lower record of a type the goroutine holds. A goroutine's records only go up, so no wait cycle can form. Logging every nested acquire in all unit and-shortintegration tests found no production path holding two locks of one type, so the new check refuses nothing today. A future rename that can't pick its order would need the rule dropped for inodes.New volumes set required flag
RequiredTypedLockRecords. Older builds refuse them, and mount refuses volumes without it. The format version stays 3.TestInodeAndExtentLocksDoNotWaitInACycle-shortTestCopyOnWriteKeepsConcurrentInPlaceWritealso fails on main on .68 (2/5 to 5/5) and is unaffected by this PR; reported to the shepherd.🤖 Generated with Claude Code