diff --git a/CHANGELOG.md b/CHANGELOG.md index cb21e0fb..7c8432f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,7 +64,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - `-cache-inodes`, `-cache-extents`, `-cache-negative` and `-broadcast-max-peers` set the cache and broadcast table sizes, and `/metrics` reports each limit next to its size. - A node that hears no broadcasts from a node the registry shows mounted, for 3 `-broadcast-heartbeat` intervals (90 seconds by default), logs a warning naming it, reports `broadcast-peers` as degraded on `/health`, and counts it in `goclusterfs_broadcast_silent_peers`. A network that silently drops multicast was invisible before. - Rename of files and directories, in one directory or between two, with `RENAME_NOREPLACE`. A rename over a file or an empty directory replaces it; a file another node has open stays readable there until its last close. `RENAME_EXCHANGE` and `RENAME_WHITEOUT` fail with `EINVAL`. -- `goclusterfs-fsck` checks the directory tree: a directory with more than one name, a `..` that does not name the directory that names it, a directory link count other than 2 plus its subdirectories, and a `..` chain that never reaches the root. Repair reports these but does not change directories. +- `goclusterfs-fsck` checks the directory tree: a directory with more than one name, a `..` that does not name the directory that names it, a directory link count other than 2 plus its subdirectories, and a `..` chain that never reaches the root. Repair points a wrong `..` at the directory that names it and reports the rest. - `-preflight` warns when a broadcast port or the API port is in the NodePort range 30000-32767 on a host that looks like a Kubernetes node. ### Changed @@ -88,6 +88,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed +- `goclusterfs-fsck` never checked a directory's `.` entry. It now reports a missing `.` or one that names another inode, with both inode numbers. `-repair` points a wrong `.` back at the directory and a wrong `..` at the directory that names it, found by walking the names down from the root, for example after a crash in the middle of a rename that moves a directory. - A power failure or an I/O error during mkdir or rmdir could leave the parent directory's link count off by one: the entry and the count were written under separate locks, and replay did not fix the count. Both now change in one write of the parent's inode. Replay, or the failed mkdir or rmdir itself, then sets the parent's count from its subdirectories. - `goclusterfs-fsck` checked a directory entry's type only when the entry's inode number was below the count of inodes in use, and then only that the type was a valid value. It now reports every entry whose type differs from its inode's, which lookup, unlink and rmdir trust: an unlink could remove a directory that its entry calls a file. It also reports an entry whose inode number is past the inode table; before, a number past 2^32 wrapped and could pass as an inode in use. - `goclusterfs-fsck` never checked a snapshot's parent: the check was an empty block. It now warns about a snapshot whose parent snapshot does not exist, is the snapshot itself, or was taken of another file. Deleting a snapshot hands its children to its parent. diff --git a/CLAUDE.md b/CLAUDE.md index 033a2d64..1d113dfc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -96,7 +96,7 @@ Clone paths are absolute from the filesystem root, and clones can only go in the - Linux only. Needs FUSE and a backing store that supports O_DIRECT. - At most 64 nodes mount a volume at once, one per registry slot. A node judges another by whether its heartbeat changes, timed on its own clock, so failover does not need clocks to agree. -- `goclusterfs-fsck -repair` fixes only block refcounts, sharing that older versions left uncounted, leaked blocks, inodes no directory names, snapshot registry sectors that fail their checksum, and the superblock checksum. It reports every other problem without fixing it. +- `goclusterfs-fsck -repair` fixes only block refcounts, sharing that older versions left uncounted, leaked blocks, inodes no directory names, snapshot registry sectors that fail their checksum, a directory's wrong `.` or `..`, and the superblock checksum. It reports every other problem without fixing it. - The inode count is set at format time and cannot change. Every `statfs` (`df`) reads the whole inode table to count free inodes: 50 MB on a 200 GB device at the default ratio. - SFB is disabled: writes never allocate SFB slots, so every file uses full 1 MB blocks. `internal/sfb` container I/O is broken (unaligned header buffers, no data offset) and stays until it is fixed or removed. - No hard links. Rename supports `RENAME_NOREPLACE`; `RENAME_EXCHANGE` and `RENAME_WHITEOUT` fail with `EINVAL`. Symlinks need a volume formatted by a build with symlink support. diff --git a/README.md b/README.md index be82a20e..d08a36b9 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod - **The inode count is set at format time.** `-format` creates one inode per `-bytes-per-inode` bytes of device (default 1 MB, so one per data block), between 4096 and 16,777,216. It cannot change later. - **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md). - **Snapshots and clones are experimental.** Before you mount an image that has snapshots or clones with a newer build, read the upgrade notes in [CHANGELOG.md](CHANGELOG.md). They say whether to run `goclusterfs-fsck` first. -- **fsck repairs little.** `-repair` rebuilds refcounts, frees leaked blocks and inodes no directory names, fixes the superblock checksum, and rewrites snapshot registry sectors that fail their checksum. It only reports other problems. See [Filesystem Check](#filesystem-check). +- **fsck repairs little.** `-repair` rebuilds refcounts, frees leaked blocks and inodes no directory names, fixes the superblock checksum, rewrites snapshot registry sectors that fail their checksum, and points a wrong `.` or `..` at the right directory. It only reports other problems. See [Filesystem Check](#filesystem-check). - **Cluster safety limits.** Node stalls and storage timeouts have hard limits. Read [Node Requirements](docs/OPERATIONS.md#node-requirements) and [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) before you run more than one node. ## Quick Start diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 504e021f..43651b6d 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -447,7 +447,7 @@ goclusterfs-fsck -repair /dev/device goclusterfs-mount -device /dev/device -mount /mnt/cluster ``` -`-repair` fixes refcounts and the superblock checksum, frees leaked blocks and inodes no directory names, and rewrites a snapshot registry sector that fails its checksum, keeping the snapshots whose descriptors pass their own checksum. It never edits directories, and there is no `lost+found`. Restore anything else from backup. See the [fsck Guide](TROUBLESHOOTING.md#fsck-guide). +`-repair` fixes refcounts and the superblock checksum, frees leaked blocks and inodes no directory names, and rewrites a snapshot registry sector that fails its checksum, keeping the snapshots whose descriptors pass their own checksum. It edits directories only to point a wrong `.` or `..` back at the right directory, and there is no `lost+found`. Restore anything else from backup. See the [fsck Guide](TROUBLESHOOTING.md#fsck-guide). --- diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index be265175..119c0ff4 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -541,8 +541,9 @@ If a node mounts the device while `-repair` runs, repair stops before its next w - It fixes the superblock checksum, but only when every other superblock field passed its checks. - It frees leaked blocks: blocks the bitmap marks allocated that no inode, extent tree or older small-file container maps. Writes and mkdirs that fail, interrupted truncates and unlinks, frees that gave up waiting for a lock, and small-file migrations leave them behind. It sets their refcounts to 0 first. It keeps all of them, refcounts included, when it could not read an inode or extent tree, when `-skip-phase` skipped the bitmap, inode or refcount phase, or when a node's journal still needs replay (even with `-force`). It also keeps those in bitmap sectors with a bad checksum. - It frees inodes that no directory names, with their blocks, the way unlink does. Replay leaves them behind when it skips an unlink's records, for example after another node wrote a file that was unlinked while open there. A block that a clone or snapshot also maps loses one owner and stays. It keeps inodes a snapshot refers to, reserved inodes, directories that are not empty, and files that map older small-file slots. It keeps all of them when it could not read every inode, directory or snapshot descriptor, when `-skip-phase` skipped a phase it needs, or when a node's journal still needs replay. There is no `lost+found`, so the data of a freed file is gone: read the `-dry-run` list first. +- It points a directory's `.` back at the directory itself, and its `..` at the directory that names it, found by walking the names down from the root. It leaves a `..` alone when the directory has more than one name or the walk does not reach it, and it does not add a missing `.` or `..`. -It never edits directories. Problems not marked `[FIXABLE]` stay as they are. +It edits directories only for that. Problems not marked `[FIXABLE]` stay as they are. ### Exit Codes diff --git a/internal/fsck/phase6_directories.go b/internal/fsck/phase6_directories.go index e632f88c..9d8458be 100644 --- a/internal/fsck/phase6_directories.go +++ b/internal/fsck/phase6_directories.go @@ -1,11 +1,17 @@ package fsck import ( + "context" "encoding/binary" "fmt" "maps" "slices" + "github.com/supporttools/GoClusterFS/internal/bitmap" + "github.com/supporttools/GoClusterFS/internal/dir" + "github.com/supporttools/GoClusterFS/internal/extent" + "github.com/supporttools/GoClusterFS/internal/inode" + "github.com/supporttools/GoClusterFS/internal/refcount" "github.com/supporttools/GoClusterFS/internal/storage" ) @@ -71,7 +77,7 @@ func (c *Checker) checkDirectories() error { dirCount := 0 entryCount := 0 invalidEntries := 0 - tree := dirTree{dotdot: map[uint32]uint32{}, parents: map[uint32][]uint32{}, subdirs: map[uint32]int{}, links: map[uint32]uint32{}} + tree := dirTree{dot: map[uint32]uint64{}, dotdot: map[uint32]uint32{}, parents: map[uint32][]uint32{}, subdirs: map[uint32]int{}, links: map[uint32]uint32{}} totalInodes := sb.InodeTableSize / InodeSize // Find all directory inodes and validate their entries @@ -128,6 +134,9 @@ func (c *Checker) checkDirectories() error { if entry.InodeNum == 0 { continue // Deleted entry } + if entry.Name == "." { + tree.dot[inodeNum] = entry.InodeNum + } if entry.InodeNum >= totalInodes { c.addProblem(PhaseDirectories, Error, fmt.Sprintf("inode %d", inodeNum), @@ -177,7 +186,9 @@ func (c *Checker) checkDirectories() error { } } if len(c.namesIncomplete) == 0 { - c.checkDirectoryTree(tree) + if err := c.checkDirectoryTree(tree); err != nil { + return err + } } else { c.logf(" Directory tree not checked: %s\n", c.namesIncomplete[0]) } @@ -297,36 +308,53 @@ func parseDirChunk(chunk []byte, base int) ([]*DirEntryData, []string) { return entries, errors } -// dirTree is what phase 6 read of every directory: its "..", the directories that name it, its subdirectories and -// its link count. +// dirTree is what phase 6 read of every directory: its "." and "..", the directories that name it, its +// subdirectories and its link count. type dirTree struct { + dot map[uint32]uint64 dotdot map[uint32]uint32 parents map[uint32][]uint32 subdirs map[uint32]int links map[uint32]uint32 } -// checkDirectoryTree reports a directory that has other than one name, whose ".." does not name the directory that -// names it, whose link count is not 2 plus its subdirectories, or from which ".." never reaches the root. -func (c *Checker) checkDirectoryTree(t dirTree) { +// checkDirectoryTree reports a directory that has other than one name, whose "." does not name itself, whose ".." +// does not name the directory that names it, whose link count is not 2 plus its subdirectories, or from which ".." +// never reaches the root. Repair points a wrong "." at the directory itself and a wrong ".." at the parent found by +// walking the names down from the root. +func (c *Checker) checkDirectoryTree(t dirTree) error { dirs := slices.Sorted(maps.Keys(t.links)) + parent := t.parentsFromRoot() + var fixes []dotFix + wrong := func(d uint32, name string, cur uint64, want uint32, found bool, msg string) { + c.addProblem(PhaseDirectories, Error, fmt.Sprintf("inode %d", d), msg, found) + if found { + fixes = append(fixes, dotFix{dir: d, name: name, cur: cur, want: want, problem: c.lastProblem()}) + } + } for _, d := range dirs { loc := fmt.Sprintf("inode %d", d) if want := 2 + uint32(t.subdirs[d]); t.links[d] != want { c.addProblem(PhaseDirectories, Warning, loc, fmt.Sprintf("directory has %d links, want %d: 2 plus one per subdirectory", t.links[d], want), false) } + if dot, ok := t.dot[d]; !ok { + c.addProblem(PhaseDirectories, Error, loc, "directory has no \".\"", false) + } else if dot != uint64(d) { + wrong(d, ".", dot, d, true, fmt.Sprintf("\".\" names inode %d, not directory %d itself", dot, d)) + } dots, hasDots := t.dotdot[d] + p, reached := parent[d] switch ps := t.parents[d]; { case d == InodeRoot: if dots != InodeRoot { - c.addProblem(PhaseDirectories, Error, loc, fmt.Sprintf("\"..\" of the root names inode %d, not the root", dots), false) + wrong(d, "..", uint64(dots), InodeRoot, hasDots, fmt.Sprintf("\"..\" of the root names inode %d, not the root", dots)) } case len(ps) > 1: c.addProblem(PhaseDirectories, Error, loc, fmt.Sprintf("directory has %d names, in directories %v; a directory has exactly one", len(ps), ps), false) case len(ps) == 1 && !hasDots: c.addProblem(PhaseDirectories, Error, loc, "directory has no \"..\"", false) case len(ps) == 1 && dots != ps[0]: - c.addProblem(PhaseDirectories, Error, loc, fmt.Sprintf("\"..\" names inode %d, but directory %d names it", dots, ps[0]), false) + wrong(d, "..", uint64(dots), p, reached, fmt.Sprintf("\"..\" names inode %d, but directory %d names it", dots, ps[0])) } } @@ -352,4 +380,71 @@ func (c *Checker) checkDirectoryTree(t dirTree) { reaches[p], cut[p] = reaches[cur], !reaches[cur] } } + if c.opts.Mode != Repair || len(fixes) == 0 { + return nil + } + return c.repairDots(fixes) +} + +// parentsFromRoot walks the names down from the root and returns, for each directory it reaches, the directory that +// names it. It does not go through a directory with more than one name. +func (t dirTree) parentsFromRoot() map[uint32]uint32 { + children := map[uint32][]uint32{} + for d, ps := range t.parents { + if len(ps) == 1 { + children[ps[0]] = append(children[ps[0]], d) + } + } + parent := map[uint32]uint32{} + for queue := []uint32{InodeRoot}; len(queue) > 0; queue = queue[1:] { + for _, d := range children[queue[0]] { + if _, seen := parent[d]; !seen && d != InodeRoot { + parent[d] = queue[0] + queue = append(queue, d) + } + } + } + return parent +} + +type dotFix struct { + dir uint32 + name string + cur uint64 + want uint32 + problem int +} + +// repairDots points each wrong "." or ".." at want, if it still names the inode phase 6 read. +func (c *Checker) repairDots(fixes []dotFix) error { + if err := c.beforeWrites(); err != nil { + return err + } + if c.opts.DryRun { + for _, f := range fixes { + c.markRepaired(f.problem) + } + return nil + } + sb := c.sbData + var uuid [16]byte + tbl := inode.NewTable(c.dev, sb.InodeTableOffset, sb.InodeTableSize, nil, uuid) + mgr := extent.NewManager(c.dev, bitmap.NewRegion(c.dev, sb.BlockBitmapOffset, sb.TotalBlocks, nil, uuid), nil, sb.DataRegionOffset) + mgr.SetRefcounts(refcount.NewRegion(c.dev, sb.RefCountTableOffset, sb.RefCountTableSize, sb.TotalBlocks, nil)) + tbl.SetTreeFlusher(mgr) + dirs := dir.NewManager(c.dev, tbl, mgr, nil, sb.DataRegionOffset) + for _, f := range fixes { + ino, err := tbl.Read(f.dir) + if err != nil { + return fmt.Errorf("directory %d: %w", f.dir, err) + } + pointed, err := dirs.PointIf(context.Background(), ino, f.name, uint64(f.want), dir.DT_DIR, func(cur uint64) bool { return cur == f.cur }) + if err != nil { + return fmt.Errorf("directory %d %q: %w", f.dir, f.name, err) + } + if pointed { + c.markRepaired(f.problem) + } + } + return c.dev.Sync() } diff --git a/internal/goclusterfs/fsck_tree_test.go b/internal/goclusterfs/fsck_tree_test.go index 17744c80..c51f66a1 100644 --- a/internal/goclusterfs/fsck_tree_test.go +++ b/internal/goclusterfs/fsck_tree_test.go @@ -2,6 +2,7 @@ package goclusterfs import ( "context" + "fmt" "strings" "testing" @@ -65,3 +66,48 @@ func pointDots(t *testing.T, fsys *FS, num, parent uint32) { t.Fatal(err) } } + +// A directory moved by rename keeps its old parent in "..", and its new parent's "." names another directory. fsck +// reports both with the inode numbers, a dry run changes nothing, and repair points them at the right directories. +func TestFsckRepairsDotEntriesAfterRename(t *testing.T) { + path := newFormattedImage(t) + fsys := openFormattedFS(t, path) + ctx := context.Background() + if err := fsys.InitRoot(ctx); err != nil { + t.Fatal(err) + } + p1, p2 := newSubdir(t, fsys, "p1"), newSubdir(t, fsys, "p2") + d := newDirIn(t, fsys, p1.inodeNum, "d") + if errno := p1.Rename(ctx, "d", p2, "e", 0); errno != 0 { + t.Fatalf("rename: %v", errno) + } + expectNoProblems(t, fsys, "after the rename") + + pointDots(t, fsys, d.inodeNum, p1.inodeNum) + if _, err := fsys.dirMgr.PointIf(ctx, readInode(t, fsys, p2.inodeNum), ".", uint64(p1.inodeNum), dir.DT_DIR, func(uint64) bool { return true }); err != nil { + t.Fatal(err) + } + dotdot := fmt.Sprintf("inode %d: \"..\" names inode %d, but directory %d names it", d.inodeNum, p1.inodeNum, p2.inodeNum) + dot := fmt.Sprintf("inode %d: \".\" names inode %d, not directory %d itself", p2.inodeNum, p1.inodeNum, p2.inodeNum) + + dry := repairOpts() + dry.DryRun = true + r := runFsck(t, fsys, dry) + expectProblem(t, r, dotdot+" [WOULD FIX]") + expectProblem(t, r, dot+" [WOULD FIX]") + if got := named(t, openFormattedFS(t, path), d.inodeNum, ".."); got != p1.inodeNum { + t.Fatalf("dry run changed \"..\" to %d", got) + } + + r = runFsck(t, fsys, repairOpts()) + expectProblem(t, r, dotdot+" [FIXED]") + expectProblem(t, r, dot+" [FIXED]") + reopened := openFormattedFS(t, path) + if got := named(t, reopened, d.inodeNum, ".."); got != p2.inodeNum { + t.Errorf("\"..\" of the moved directory names %d after repair, want %d", got, p2.inodeNum) + } + if got := named(t, reopened, p2.inodeNum, "."); got != p2.inodeNum { + t.Errorf("\".\" of the new parent names %d after repair, want %d", got, p2.inodeNum) + } + expectNoProblems(t, reopened, "after repair") +} diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index 4a8bd4e4..b78889fb 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -4361,6 +4361,13 @@ scenarios: environments: go: internal/goclusterfs/fsck_dir_test.go::TestFsckChecksEveryDirectoryEntry + - id: fsck-repair-dot-entries + name: fsck repairs a wrong "." or ".." after a rename + description: After a rename moves a directory, a ".." left at the old parent and a "." naming another directory are reported with both inode numbers; a dry run changes nothing and repair points them at the right directories + priority: go_only + environments: + go: internal/goclusterfs/fsck_tree_test.go::TestFsckRepairsDotEntriesAfterRename + - id: fsck-snapshot-parents name: fsck checks snapshot parents description: fsck reports a snapshot whose parent snapshot is missing, is itself, or was taken of another file, and accepts a parent of the same file