Skip to content

perf(dir): answer lookups from an in-memory directory index - #127

Merged
mattmattox merged 6 commits into
mainfrom
feat/dir-index
Nov 13, 2026
Merged

mattmattox merged 6 commits into
mainfrom
feat/dir-index

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Option A from the PM's plan. Each node keeps an index per directory. A lookup reads the directory inode without a lock and trusts the index only while the inode's generation matches. A torn read fails the inode CRC and takes the locked path.

PM conditions:

  • Order: entry changes write the directory inode before the chunk, inside one lock hold. With chunk-first, a failed or crashed chunk write leaves new entries under the old generation, and other nodes' indexes miss them. RemoveIf finishes before the child is freed. Create, mkdir, unlink, rmdir, clone and replay all go through Add/RemoveIf; there is no rename.
  • Add/RemoveIf re-check the index under the directory lock and rebuild it on a mismatch.
  • -dir-index-entries (default 1,048,576 names, about 100 MB) with an LRU cap; goclusterfs_dir_index_{hits,misses,rebuilds}_total.

On .68, with two nodes mounted outside $HOME (load 53–66), main vs this PR:

entries ls -l one stat create ls -l while the other node creates 200
1,000 1.6 s → 0.14 s 2.07 → 0.11 ms 11.5 → 4.6 ms 1.9 s → 0.4 s
4,000 65 s → 0.96 s 3.79 → 0.25 ms 14.1 → 5.8 ms 73 s → 3.8 s
10,000 266 s → 14 s 6.18 → 0.26 ms 21.1 → 7.0 ms 258 s → 17 s

At 10k entries, ls -l still takes 14 s. Once a pass outlasts the 1 s kernel cache it does 4 lookups per file, and each lookup reads two inodes (about 0.3 ms).

On .68 at 2a950c6: make lint, all unit tests and -race on dir, goclusterfs, stats and api pass. TestMultiNodeDirIndexWithoutBroadcast, TestMultiNodeCreatesInOneDirectory and TestMultiNodeAttrTimeoutZero pass 3/3. In the -short integration run only TestFsckRepairsOlderSharing failed, and it fails the same way on main (its remount after the offline image edit times out). The new Docker and VM shell tests weren't run locally.

🤖 Generated with Claude Code

mattmattox and others added 6 commits November 13, 2026 16:25
Each lookup took the directory lock, read the whole 1 MB block and
decoded every entry, and ls -l repeats lookups once it outlasts the
kernel's 1s cache, so it went quadratic: 65s for 4,000 files.

Each node now keeps an index per directory (name to inode, and free
room per chunk). A lookup reads the directory inode without a lock and
uses the index only while the generation on disk is the one the index
was built at; a torn read fails the inode checksum and takes the locked
path. Every entry change writes the directory inode before the chunk,
inside one lock hold, so no node can read new entries under an old
generation, and RemoveIf now does so before the child is freed. Add and
RemoveIf re-check the index against the disk under the directory lock
and rebuild it on mismatch. Writes that change no entries, such as
mtime and link count updates, carry the index to the new generation.

-dir-index-entries caps the names held, least recently used directory
first, and /metrics exports hits, misses and rebuilds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…egration test within the image's blocks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ck requires

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cluster lock admits every goroutine on a node at once: a second
exclusive or shared acquire of a held key joins the hold. Replay and API
clones bypass the kernel's per-directory lock, so two goroutines could
edit the same chunk and lose an entry, and the index then named the lost
entry at the current generation.

dir.Manager now takes a striped per-directory lock before the cluster
lock: exclusive in Add, RemoveIf and InitDirectory, shared when building
an index. Directory inode writes in goclusterfs (mtime, link count,
setattr) take it too, through updateDirLocked, so KeepIndex runs only
while no entry change is in flight.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 019f9e6 into main Nov 13, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant