Skip to content

fix(dir): fill every directory chunk and stop reporting lock timeouts as ENOSPC - #66

Merged
mattmattox merged 9 commits into
mainfrom
fix/dir-block-chunks
Nov 13, 2026
Merged

mattmattox merged 9 commits into
mainfrom
fix/dir-block-chunks

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The "No space left on device" in Docker group 3 (run 42914371885) was not a space problem. Create, mkdir, clone and snapshot turned every inode-claim error into ENOSPC, and that error usually meant a lock wait gave up after 30s. CI keeps no node logs, so I found this by tracing every ENOSPC exit on the create path. TestCreateReportsStuckInodeLockAsEIO reproduces it. Those errors now return EIO, as docs/OPERATIONS.md says.

Lock waits gave up because directory operations held the lock a long time. Only each block's first 4KB chunk held entries, lookups read 4KB at a time, and every add rewrote the whole 1MB block.

4 nodes × 1050 creates in one directory on .68:

Before #29 landed, the faster creates hit #29's claim race often and lost directory entries, so this PR waited for it. TestMultiNodeCreatesInOneDirectory now runs again. Docker group 3 ("More Than 4096 Files", 42 directories) passed in 2m58s; on main it takes 13-16 minutes.

Existing directories work as they are, and older binaries can read the new blocks. The CHANGELOG upgrade note covers the one case that breaks: an older binary removing an entry. Still open: directory extent trees under concurrent creates.

🤖 Generated with Claude Code

mattmattox and others added 9 commits November 12, 2026 00:44
Create, mkdir and clone turned every ClaimSlot error other than a slot
collision into ENOSPC. A lock wait that gives up after 30 seconds is the
usual cause, and the inode table is not full then. FindFreeSlot alone
reports a full table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New entries only went into a block's first 4KB chunk, so a 1MB block held
about 250 short or 15 long names, and a busy directory grew a block every
few creates. Each lookup then read every block 4KB at a time while holding
the directory lock.

Entries now fill the chunks in order, a block is read in one I/O, and an
add or remove writes only the chunk it changed. Existing blocks keep
working: their unused chunks start with RecLen 0. fsck checks each chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tories

The one-directory create test lists the created files it cannot find.
It skips, not fails, when a lock release found another node holding the
lock, since two holders of the directory lock can overwrite each other's
chunk until #29 lands. The runtime fsck test now fills and unlinks across
several chunks of the root directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lain them

A create also failed with EIO once in nine runs: a lock record read during
another node's write failed its checksum. #29's per-node slots treat such a
read as a held lock. The test now collects create errors and missing
entries, and skips only when the logs show a double hold or a torn lock
record; lock give-ups and anything else still fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Missing entries need a double hold in the logs and failed creates need a
torn lock read; any other combination fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI hit the other #29 failure: a reader count lost by a racing claim kept
the directory lock read-locked, so two creates gave up after 30s. With
creates now fast, every #29 race shows up in this test, so it is skipped
like TestSnapshotsDuringWritesAcrossNodes and fails on any lost entry or
create error once enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The snapshot path still turned every claimInode error into ENOSPC.
claimErrno now also maps ErrInodeFull to ENOSPC, so all callers share it.
CHANGELOG notes the errno change, the directory capacity, and that an old
binary's remove can corrupt a multi-chunk directory block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With claims through per-node slots, 3 short and 8 full-size runs on .68
passed with no lost entries, no lock give-ups and no double holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit ea5ba6d into main Nov 13, 2026
13 checks passed
mattmattox added a commit that referenced this pull request Nov 13, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant