Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- `-cache-inodes`, `-cache-extents`, `-cache-negative` and `-broadcast-max-peers` set the cache and broadcast table sizes, and `/metrics` reports each limit next to its size.
- A node that hears no broadcasts from a node the registry shows mounted, for 3 `-broadcast-heartbeat` intervals (90 seconds by default), logs a warning naming it, reports `broadcast-peers` as degraded on `/health`, and counts it in `goclusterfs_broadcast_silent_peers`. A network that silently drops multicast was invisible before.
- Rename of files and directories, in one directory or between two, with `RENAME_NOREPLACE`. A rename over a file or an empty directory replaces it; a file another node has open stays readable there until its last close. `RENAME_EXCHANGE` and `RENAME_WHITEOUT` fail with `EINVAL`.
- `goclusterfs-fsck` checks the directory tree: a directory with more than one name, a `..` that does not name the directory that names it, a directory link count other than 2 plus its subdirectories, and a `..` chain that never reaches the root. Repair reports these but does not change directories.
- `goclusterfs-fsck` checks the directory tree: a directory with more than one name, a `..` that does not name the directory that names it, a directory link count other than 2 plus its subdirectories, and a `..` chain that never reaches the root. Repair points a wrong `..` at the directory that names it and reports the rest.
- `-preflight` warns when a broadcast port or the API port is in the NodePort range 30000-32767 on a host that looks like a Kubernetes node.

### Changed
Expand All @@ -88,6 +88,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Fixed

- `goclusterfs-fsck` never checked a directory's `.` entry. It now reports a missing `.` or one that names another inode, with both inode numbers. `-repair` points a wrong `.` back at the directory and a wrong `..` at the directory that names it, found by walking the names down from the root, for example after a crash in the middle of a rename that moves a directory.
- A power failure or an I/O error during mkdir or rmdir could leave the parent directory's link count off by one: the entry and the count were written under separate locks, and replay did not fix the count. Both now change in one write of the parent's inode. Replay, or the failed mkdir or rmdir itself, then sets the parent's count from its subdirectories.
- `goclusterfs-fsck` checked a directory entry's type only when the entry's inode number was below the count of inodes in use, and then only that the type was a valid value. It now reports every entry whose type differs from its inode's, which lookup, unlink and rmdir trust: an unlink could remove a directory that its entry calls a file. It also reports an entry whose inode number is past the inode table; before, a number past 2^32 wrapped and could pass as an inode in use.
- `goclusterfs-fsck` never checked a snapshot's parent: the check was an empty block. It now warns about a snapshot whose parent snapshot does not exist, is the snapshot itself, or was taken of another file. Deleting a snapshot hands its children to its parent.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Clone paths are absolute from the filesystem root, and clones can only go in the

- Linux only. Needs FUSE and a backing store that supports O_DIRECT.
- At most 64 nodes mount a volume at once, one per registry slot. A node judges another by whether its heartbeat changes, timed on its own clock, so failover does not need clocks to agree.
- `goclusterfs-fsck -repair` fixes only block refcounts, sharing that older versions left uncounted, leaked blocks, inodes no directory names, snapshot registry sectors that fail their checksum, and the superblock checksum. It reports every other problem without fixing it.
- `goclusterfs-fsck -repair` fixes only block refcounts, sharing that older versions left uncounted, leaked blocks, inodes no directory names, snapshot registry sectors that fail their checksum, a directory's wrong `.` or `..`, and the superblock checksum. It reports every other problem without fixing it.
- The inode count is set at format time and cannot change. Every `statfs` (`df`) reads the whole inode table to count free inodes: 50 MB on a 200 GB device at the default ratio.
- SFB is disabled: writes never allocate SFB slots, so every file uses full 1 MB blocks. `internal/sfb` container I/O is broken (unaligned header buffers, no data offset) and stays until it is fixed or removed.
- No hard links. Rename supports `RENAME_NOREPLACE`; `RENAME_EXCHANGE` and `RENAME_WHITEOUT` fail with `EINVAL`. Symlinks need a volume formatted by a build with symlink support.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod
- **The inode count is set at format time.** `-format` creates one inode per `-bytes-per-inode` bytes of device (default 1 MB, so one per data block), between 4096 and 16,777,216. It cannot change later.
- **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md).
- **Snapshots and clones are experimental.** Before you mount an image that has snapshots or clones with a newer build, read the upgrade notes in [CHANGELOG.md](CHANGELOG.md). They say whether to run `goclusterfs-fsck` first.
- **fsck repairs little.** `-repair` rebuilds refcounts, frees leaked blocks and inodes no directory names, fixes the superblock checksum, and rewrites snapshot registry sectors that fail their checksum. It only reports other problems. See [Filesystem Check](#filesystem-check).
- **fsck repairs little.** `-repair` rebuilds refcounts, frees leaked blocks and inodes no directory names, fixes the superblock checksum, rewrites snapshot registry sectors that fail their checksum, and points a wrong `.` or `..` at the right directory. It only reports other problems. See [Filesystem Check](#filesystem-check).
- **Cluster safety limits.** Node stalls and storage timeouts have hard limits. Read [Node Requirements](docs/OPERATIONS.md#node-requirements) and [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) before you run more than one node.

## Quick Start
Expand Down
2 changes: 1 addition & 1 deletion docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -447,7 +447,7 @@ goclusterfs-fsck -repair /dev/device
goclusterfs-mount -device /dev/device -mount /mnt/cluster
```

`-repair` fixes refcounts and the superblock checksum, frees leaked blocks and inodes no directory names, and rewrites a snapshot registry sector that fails its checksum, keeping the snapshots whose descriptors pass their own checksum. It never edits directories, and there is no `lost+found`. Restore anything else from backup. See the [fsck Guide](TROUBLESHOOTING.md#fsck-guide).
`-repair` fixes refcounts and the superblock checksum, frees leaked blocks and inodes no directory names, and rewrites a snapshot registry sector that fails its checksum, keeping the snapshots whose descriptors pass their own checksum. It edits directories only to point a wrong `.` or `..` back at the right directory, and there is no `lost+found`. Restore anything else from backup. See the [fsck Guide](TROUBLESHOOTING.md#fsck-guide).

---

Expand Down
3 changes: 2 additions & 1 deletion docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -541,8 +541,9 @@ If a node mounts the device while `-repair` runs, repair stops before its next w
- It fixes the superblock checksum, but only when every other superblock field passed its checks.
- It frees leaked blocks: blocks the bitmap marks allocated that no inode, extent tree or older small-file container maps. Writes and mkdirs that fail, interrupted truncates and unlinks, frees that gave up waiting for a lock, and small-file migrations leave them behind. It sets their refcounts to 0 first. It keeps all of them, refcounts included, when it could not read an inode or extent tree, when `-skip-phase` skipped the bitmap, inode or refcount phase, or when a node's journal still needs replay (even with `-force`). It also keeps those in bitmap sectors with a bad checksum.
- It frees inodes that no directory names, with their blocks, the way unlink does. Replay leaves them behind when it skips an unlink's records, for example after another node wrote a file that was unlinked while open there. A block that a clone or snapshot also maps loses one owner and stays. It keeps inodes a snapshot refers to, reserved inodes, directories that are not empty, and files that map older small-file slots. It keeps all of them when it could not read every inode, directory or snapshot descriptor, when `-skip-phase` skipped a phase it needs, or when a node's journal still needs replay. There is no `lost+found`, so the data of a freed file is gone: read the `-dry-run` list first.
- It points a directory's `.` back at the directory itself, and its `..` at the directory that names it, found by walking the names down from the root. It leaves a `..` alone when the directory has more than one name or the walk does not reach it, and it does not add a missing `.` or `..`.

It never edits directories. Problems not marked `[FIXABLE]` stay as they are.
It edits directories only for that. Problems not marked `[FIXABLE]` stay as they are.

### Exit Codes

Expand Down
113 changes: 104 additions & 9 deletions internal/fsck/phase6_directories.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
package fsck

import (
"context"
"encoding/binary"
"fmt"
"maps"
"slices"

"github.com/supporttools/GoClusterFS/internal/bitmap"
"github.com/supporttools/GoClusterFS/internal/dir"
"github.com/supporttools/GoClusterFS/internal/extent"
"github.com/supporttools/GoClusterFS/internal/inode"
"github.com/supporttools/GoClusterFS/internal/refcount"
"github.com/supporttools/GoClusterFS/internal/storage"
)

Expand Down Expand Up @@ -71,7 +77,7 @@ func (c *Checker) checkDirectories() error {
dirCount := 0
entryCount := 0
invalidEntries := 0
tree := dirTree{dotdot: map[uint32]uint32{}, parents: map[uint32][]uint32{}, subdirs: map[uint32]int{}, links: map[uint32]uint32{}}
tree := dirTree{dot: map[uint32]uint64{}, dotdot: map[uint32]uint32{}, parents: map[uint32][]uint32{}, subdirs: map[uint32]int{}, links: map[uint32]uint32{}}
totalInodes := sb.InodeTableSize / InodeSize

// Find all directory inodes and validate their entries
Expand Down Expand Up @@ -128,6 +134,9 @@ func (c *Checker) checkDirectories() error {
if entry.InodeNum == 0 {
continue // Deleted entry
}
if entry.Name == "." {
tree.dot[inodeNum] = entry.InodeNum
}

if entry.InodeNum >= totalInodes {
c.addProblem(PhaseDirectories, Error, fmt.Sprintf("inode %d", inodeNum),
Expand Down Expand Up @@ -177,7 +186,9 @@ func (c *Checker) checkDirectories() error {
}
}
if len(c.namesIncomplete) == 0 {
c.checkDirectoryTree(tree)
if err := c.checkDirectoryTree(tree); err != nil {
return err
}
} else {
c.logf(" Directory tree not checked: %s\n", c.namesIncomplete[0])
}
Expand Down Expand Up @@ -297,36 +308,53 @@ func parseDirChunk(chunk []byte, base int) ([]*DirEntryData, []string) {
return entries, errors
}

// dirTree is what phase 6 read of every directory: its "..", the directories that name it, its subdirectories and
// its link count.
// dirTree is what phase 6 read of every directory: its "." and "..", the directories that name it, its
// subdirectories and its link count.
type dirTree struct {
dot map[uint32]uint64
dotdot map[uint32]uint32
parents map[uint32][]uint32
subdirs map[uint32]int
links map[uint32]uint32
}

// checkDirectoryTree reports a directory that has other than one name, whose ".." does not name the directory that
// names it, whose link count is not 2 plus its subdirectories, or from which ".." never reaches the root.
func (c *Checker) checkDirectoryTree(t dirTree) {
// checkDirectoryTree reports a directory that has other than one name, whose "." does not name itself, whose ".."
// does not name the directory that names it, whose link count is not 2 plus its subdirectories, or from which ".."
// never reaches the root. Repair points a wrong "." at the directory itself and a wrong ".." at the parent found by
// walking the names down from the root.
func (c *Checker) checkDirectoryTree(t dirTree) error {
dirs := slices.Sorted(maps.Keys(t.links))
parent := t.parentsFromRoot()
var fixes []dotFix
wrong := func(d uint32, name string, cur uint64, want uint32, found bool, msg string) {
c.addProblem(PhaseDirectories, Error, fmt.Sprintf("inode %d", d), msg, found)
if found {
fixes = append(fixes, dotFix{dir: d, name: name, cur: cur, want: want, problem: c.lastProblem()})
}
}
for _, d := range dirs {
loc := fmt.Sprintf("inode %d", d)
if want := 2 + uint32(t.subdirs[d]); t.links[d] != want {
c.addProblem(PhaseDirectories, Warning, loc, fmt.Sprintf("directory has %d links, want %d: 2 plus one per subdirectory", t.links[d], want), false)
}
if dot, ok := t.dot[d]; !ok {
c.addProblem(PhaseDirectories, Error, loc, "directory has no \".\"", false)
} else if dot != uint64(d) {
wrong(d, ".", dot, d, true, fmt.Sprintf("\".\" names inode %d, not directory %d itself", dot, d))
}
dots, hasDots := t.dotdot[d]
p, reached := parent[d]
switch ps := t.parents[d]; {
case d == InodeRoot:
if dots != InodeRoot {
c.addProblem(PhaseDirectories, Error, loc, fmt.Sprintf("\"..\" of the root names inode %d, not the root", dots), false)
wrong(d, "..", uint64(dots), InodeRoot, hasDots, fmt.Sprintf("\"..\" of the root names inode %d, not the root", dots))
}
case len(ps) > 1:
c.addProblem(PhaseDirectories, Error, loc, fmt.Sprintf("directory has %d names, in directories %v; a directory has exactly one", len(ps), ps), false)
case len(ps) == 1 && !hasDots:
c.addProblem(PhaseDirectories, Error, loc, "directory has no \"..\"", false)
case len(ps) == 1 && dots != ps[0]:
c.addProblem(PhaseDirectories, Error, loc, fmt.Sprintf("\"..\" names inode %d, but directory %d names it", dots, ps[0]), false)
wrong(d, "..", uint64(dots), p, reached, fmt.Sprintf("\"..\" names inode %d, but directory %d names it", dots, ps[0]))
}
}

Expand All @@ -352,4 +380,71 @@ func (c *Checker) checkDirectoryTree(t dirTree) {
reaches[p], cut[p] = reaches[cur], !reaches[cur]
}
}
if c.opts.Mode != Repair || len(fixes) == 0 {
return nil
}
return c.repairDots(fixes)
}

// parentsFromRoot walks the names down from the root and returns, for each directory it reaches, the directory that
// names it. It does not go through a directory with more than one name.
func (t dirTree) parentsFromRoot() map[uint32]uint32 {
children := map[uint32][]uint32{}
for d, ps := range t.parents {
if len(ps) == 1 {
children[ps[0]] = append(children[ps[0]], d)
}
}
parent := map[uint32]uint32{}
for queue := []uint32{InodeRoot}; len(queue) > 0; queue = queue[1:] {
for _, d := range children[queue[0]] {
if _, seen := parent[d]; !seen && d != InodeRoot {
parent[d] = queue[0]
queue = append(queue, d)
}
}
}
return parent
}

type dotFix struct {
dir uint32
name string
cur uint64
want uint32
problem int
}

// repairDots points each wrong "." or ".." at want, if it still names the inode phase 6 read.
func (c *Checker) repairDots(fixes []dotFix) error {
if err := c.beforeWrites(); err != nil {
return err
}
if c.opts.DryRun {
for _, f := range fixes {
c.markRepaired(f.problem)
}
return nil
}
sb := c.sbData
var uuid [16]byte
tbl := inode.NewTable(c.dev, sb.InodeTableOffset, sb.InodeTableSize, nil, uuid)
mgr := extent.NewManager(c.dev, bitmap.NewRegion(c.dev, sb.BlockBitmapOffset, sb.TotalBlocks, nil, uuid), nil, sb.DataRegionOffset)
mgr.SetRefcounts(refcount.NewRegion(c.dev, sb.RefCountTableOffset, sb.RefCountTableSize, sb.TotalBlocks, nil))
tbl.SetTreeFlusher(mgr)
dirs := dir.NewManager(c.dev, tbl, mgr, nil, sb.DataRegionOffset)
for _, f := range fixes {
ino, err := tbl.Read(f.dir)
if err != nil {
return fmt.Errorf("directory %d: %w", f.dir, err)
}
pointed, err := dirs.PointIf(context.Background(), ino, f.name, uint64(f.want), dir.DT_DIR, func(cur uint64) bool { return cur == f.cur })
if err != nil {
return fmt.Errorf("directory %d %q: %w", f.dir, f.name, err)
}
if pointed {
c.markRepaired(f.problem)
}
}
return c.dev.Sync()
}
46 changes: 46 additions & 0 deletions internal/goclusterfs/fsck_tree_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package goclusterfs

import (
"context"
"fmt"
"strings"
"testing"

Expand Down Expand Up @@ -65,3 +66,48 @@ func pointDots(t *testing.T, fsys *FS, num, parent uint32) {
t.Fatal(err)
}
}

// A directory moved by rename keeps its old parent in "..", and its new parent's "." names another directory. fsck
// reports both with the inode numbers, a dry run changes nothing, and repair points them at the right directories.
func TestFsckRepairsDotEntriesAfterRename(t *testing.T) {
path := newFormattedImage(t)
fsys := openFormattedFS(t, path)
ctx := context.Background()
if err := fsys.InitRoot(ctx); err != nil {
t.Fatal(err)
}
p1, p2 := newSubdir(t, fsys, "p1"), newSubdir(t, fsys, "p2")
d := newDirIn(t, fsys, p1.inodeNum, "d")
if errno := p1.Rename(ctx, "d", p2, "e", 0); errno != 0 {
t.Fatalf("rename: %v", errno)
}
expectNoProblems(t, fsys, "after the rename")

pointDots(t, fsys, d.inodeNum, p1.inodeNum)
if _, err := fsys.dirMgr.PointIf(ctx, readInode(t, fsys, p2.inodeNum), ".", uint64(p1.inodeNum), dir.DT_DIR, func(uint64) bool { return true }); err != nil {
t.Fatal(err)
}
dotdot := fmt.Sprintf("inode %d: \"..\" names inode %d, but directory %d names it", d.inodeNum, p1.inodeNum, p2.inodeNum)
dot := fmt.Sprintf("inode %d: \".\" names inode %d, not directory %d itself", p2.inodeNum, p1.inodeNum, p2.inodeNum)

dry := repairOpts()
dry.DryRun = true
r := runFsck(t, fsys, dry)
expectProblem(t, r, dotdot+" [WOULD FIX]")
expectProblem(t, r, dot+" [WOULD FIX]")
if got := named(t, openFormattedFS(t, path), d.inodeNum, ".."); got != p1.inodeNum {
t.Fatalf("dry run changed \"..\" to %d", got)
}

r = runFsck(t, fsys, repairOpts())
expectProblem(t, r, dotdot+" [FIXED]")
expectProblem(t, r, dot+" [FIXED]")
reopened := openFormattedFS(t, path)
if got := named(t, reopened, d.inodeNum, ".."); got != p2.inodeNum {
t.Errorf("\"..\" of the moved directory names %d after repair, want %d", got, p2.inodeNum)
}
if got := named(t, reopened, p2.inodeNum, "."); got != p2.inodeNum {
t.Errorf("\".\" of the new parent names %d after repair, want %d", got, p2.inodeNum)
}
expectNoProblems(t, reopened, "after repair")
}
7 changes: 7 additions & 0 deletions tests/test-manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4361,6 +4361,13 @@ scenarios:
environments:
go: internal/goclusterfs/fsck_dir_test.go::TestFsckChecksEveryDirectoryEntry

- id: fsck-repair-dot-entries
name: fsck repairs a wrong "." or ".." after a rename
description: After a rename moves a directory, a ".." left at the old parent and a "." naming another directory are reported with both inode numbers; a dry run changes nothing and repair points them at the right directories
priority: go_only
environments:
go: internal/goclusterfs/fsck_tree_test.go::TestFsckRepairsDotEntriesAfterRename

- id: fsck-snapshot-parents
name: fsck checks snapshot parents
description: fsck reports a snapshot whose parent snapshot is missing, is itself, or was taken of another file, and accepts a parent of the same file
Expand Down
Loading