Repository navigation
fix(dir): fill every directory chunk and stop reporting lock timeouts as ENOSPC - #66
Merged
Merged
Conversation
Create, mkdir and clone turned every ClaimSlot error other than a slot collision into ENOSPC. A lock wait that gives up after 30 seconds is the usual cause, and the inode table is not full then. FindFreeSlot alone reports a full table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New entries only went into a block's first 4KB chunk, so a 1MB block held about 250 short or 15 long names, and a busy directory grew a block every few creates. Each lookup then read every block 4KB at a time while holding the directory lock. Entries now fill the chunks in order, a block is read in one I/O, and an add or remove writes only the chunk it changed. Existing blocks keep working: their unused chunks start with RecLen 0. fsck checks each chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tories The one-directory create test lists the created files it cannot find. It skips, not fails, when a lock release found another node holding the lock, since two holders of the directory lock can overwrite each other's chunk until #29 lands. The runtime fsck test now fills and unlinks across several chunks of the root directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lain them A create also failed with EIO once in nine runs: a lock record read during another node's write failed its checksum. #29's per-node slots treat such a read as a held lock. The test now collects create errors and missing entries, and skips only when the logs show a double hold or a torn lock record; lock give-ups and anything else still fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Missing entries need a double hold in the logs and failed creates need a torn lock read; any other combination fails the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI hit the other #29 failure: a reader count lost by a racing claim kept the directory lock read-locked, so two creates gave up after 30s. With creates now fast, every #29 race shows up in this test, so it is skipped like TestSnapshotsDuringWritesAcrossNodes and fails on any lost entry or create error once enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The snapshot path still turned every claimInode error into ENOSPC. claimErrno now also maps ErrInodeFull to ENOSPC, so all callers share it. CHANGELOG notes the errno change, the directory capacity, and that an old binary's remove can corrupt a multi-chunk directory block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # internal/goclusterfs/fs.go
With claims through per-node slots, 3 short and 8 full-size runs on .68 passed with no lost entries, no lock give-ups and no double holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "No space left on device" in Docker group 3 (run 42914371885) was not a space problem. Create, mkdir, clone and snapshot turned every inode-claim error into ENOSPC, and that error usually meant a lock wait gave up after 30s. CI keeps no node logs, so I found this by tracing every ENOSPC exit on the create path.
TestCreateReportsStuckInodeLockAsEIOreproduces it. Those errors now return EIO, as docs/OPERATIONS.md says.Lock waits gave up because directory operations held the lock a long time. Only each block's first 4KB chunk held entries, lookups read 4KB at a time, and every add rewrote the whole 1MB block.
4 nodes × 1050 creates in one directory on .68:
Before #29 landed, the faster creates hit #29's claim race often and lost directory entries, so this PR waited for it.
TestMultiNodeCreatesInOneDirectorynow runs again. Docker group 3 ("More Than 4096 Files", 42 directories) passed in 2m58s; on main it takes 13-16 minutes.Existing directories work as they are, and older binaries can read the new blocks. The CHANGELOG upgrade note covers the one case that breaks: an older binary removing an entry. Still open: directory extent trees under concurrent creates.
🤖 Generated with Claude Code