Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- The on-disk format is now version 3. Mount refuses devices formatted by earlier builds. Version 1 kept one journal for all nodes. Version 2 had room for 4096 inodes, and on devices over 128 GB its refcounts overwrote the snapshot registry and then data blocks. Copy the data off with the old build, reformat with `-format`, and copy it back.
- Devices must be at least 327 MB: the layout reserves 256 MB for 64 node journals of 4 MB and 64 MB for lock records.
- Mount also refuses version 3 devices formatted before lock records had one claim slot per node, or before each lock type had its own lock records. Older builds refuse devices this version formats. Copy the data off with the build that formatted them and reformat.
- Node clocks must agree within a few seconds. See the node requirements in `docs/OPERATIONS.md`.
- Upgrade all nodes together: unmount on every node, upgrade, then mount again. Don't mount one shared device with a mix of old and new binaries.
- Older binaries reset an inode's generation counters to 0 when they free it, so other nodes can serve stale data after the inode number is reused.
- Older binaries zero the extent-tree generation (inode bytes 188–195) when they rewrite an inode. Newer binaries then cannot read that file if it has more than two extents, because such files use an extent tree.
Expand Down Expand Up @@ -86,6 +85,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- `goclusterfs-fsck` ran for hours on a superblock with a damaged node count, reading one journal per node it claimed. It now reads at most the 64 that the node registry has.
- One failed write while a node released a lock left its claim on the device, and the node forgot it: other nodes timed out with `EIO` on every resource in that lock record until the node unmounted. The node now clears such a claim again every second until the write succeeds.
- One failed read of the snapshot registry during a snapshot create could delete other snapshots: the create counted the unreadable sector's slots as free, and a write to a sector it could not read replaced the sector with one that held only the new snapshot, losing up to 30 others. The create still reported success. Both now fail with `EIO` and leave the sector alone. A sector that stays unreadable now blocks every snapshot create; `goclusterfs-fsck` reports it but cannot repair it yet.
- Nodes judged each other's heartbeats and recovery leases by wall clock. A node mounting with its clock more than 10 seconds ahead took a live node's registry slot and stopped it; a crashed node whose clock ran ahead was replayed only once its clock's lead ran out; and a replay whose node's clock was behind looked abandoned, so lock waits for the dead node failed with `EIO`. A node now counts another as failing, dead or abandoned only after reading the same heartbeat or lease for long enough by its own clock; a mount takes another host's slot only when it held that hostname before a reboot, when `-skip-recovery` names it, or when no slot is empty, and only after watching its heartbeat stay unchanged for about 12 seconds.
- Under slow storage a node was declared dead while it was only slow: registry scans held the heartbeat back for 64 reads each, so at 1 second per I/O a peer replayed the node and it stopped. Scans now read the registry in one I/O without holding the heartbeat's lock, and registry I/O no longer waits for other I/O to the same 1 MB region.
- A node in a higher registry slot could wait 30 seconds and fail with `EIO` for a lock that a node in a lower slot kept releasing and claiming again every few milliseconds. A node that loses a claim to a lower slot now leaves a waiting mark in its own slot, and the holder gives the lock up 250 ms after it first sees a live node's mark.
- `chmod`, `chown` and `touch` could undo another node's concurrent change to the same inode: on a directory, a create's link count or mtime; on a file, the size an append inside an allocated block set. Directory attribute changes now hold the inode lock from read to write, and file attribute changes retry when the inode changed since they read it.
- After 3 failed multicast sends in total, counted over the whole mount, a node sent cache invalidations only to unicast peers until it unmounted. Now only failures in a row count: a successful send resets them, and after 3 in a row the node tries multicast again 30 seconds later.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Clone paths are absolute from the filesystem root, and clones can only go in the
## Current Limitations

- Linux only. Needs FUSE and a backing store that supports O_DIRECT.
- At most 64 nodes mount a volume at once, one per registry slot. Heartbeats compare wall clocks, so node clocks must agree within a few seconds.
- At most 64 nodes mount a volume at once, one per registry slot. A node judges another by whether its heartbeat changes, timed on its own clock, so failover does not need clocks to agree.
- `goclusterfs-fsck -repair` fixes only block refcounts, sharing that older versions left uncounted, leaked blocks, inodes no directory names, and the superblock checksum. It reports every other problem without fixing it.
- The inode count is set at format time and cannot change. Every `statfs` (`df`) reads the whole inode table to count free inodes: 50 MB on a 200 GB device at the default ratio.
- SFB is disabled: writes never allocate SFB slots, so every file uses full 1 MB blocks. `internal/sfb` container I/O is broken (unaligned header buffers, no data offset) and stays until it is fixed or removed.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod
- **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md).
- **Snapshots and clones are experimental.** Before you mount an image that has snapshots or clones with a newer build, read the upgrade notes in [CHANGELOG.md](CHANGELOG.md). They say whether to run `goclusterfs-fsck` first.
- **fsck repairs little.** `-repair` rebuilds refcounts, frees leaked blocks and inodes no directory names, and fixes the superblock checksum. It only reports other problems. See [Filesystem Check](#filesystem-check).
- **Cluster safety limits.** Node clocks, node stalls, and storage timeouts have hard limits. Read [Node Requirements](docs/OPERATIONS.md#node-requirements) and [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) before you run more than one node.
- **Cluster safety limits.** Node stalls and storage timeouts have hard limits. Read [Node Requirements](docs/OPERATIONS.md#node-requirements) and [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) before you run more than one node.

## Quick Start

Expand Down Expand Up @@ -195,7 +195,7 @@ For clustered deployments:
1. **Shared Storage**: All nodes must access the same block device (SAN, iSCSI, NVMe-oF)
2. **Network**: Nodes should be on the same subnet for multicast invalidation
3. **Mount Command**: Use identical mount options across all nodes
4. **Limits**: At most 64 nodes mount a device at once, and node clocks must agree within a few seconds (run NTP)
4. **Limits**: At most 64 nodes mount a device at once; run NTP so file times agree
5. **Format version**: This release uses on-disk format version 3, which gives each node its own journal and sizes the block bitmap, refcount table and inode table from the device. It refuses devices formatted by earlier builds; copy their data off and reformat
6. **Stalled nodes**: Other nodes declare a node dead after about 20 seconds of silence, for example a paused VM, and replay its journal. The stalled node checks its registry slot before each write, so when it resumes it refuses new writes and exits. A write it started before the stall can still land. Only storage fencing closes that gap, such as SCSI-3 persistent reservations, which GoClusterFS does not use. See [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits)

Expand Down
5 changes: 4 additions & 1 deletion cmd/goclusterfs-mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,9 @@ func main() {
nodeReg := node.NewRegistry(dev, nodeRegistryOffset)
nodeReg.SetJournalArea(sb.JournalRegionOffset, sb.JournalSlotSize())
hostname, _ := os.Hostname()
if *skipRecovery >= 0 {
nodeReg.PreferSlot(*skipRecovery)
}
slotIdx, err := nodeReg.Register(ctx, hostname)
if errors.Is(err, node.ErrNoFreeSlots) {
failf("Cannot join the cluster: all %d node slots are taken. List them with goclusterfs-fsck -list-slots %s, and free the slot of a node that will not come back with -release-slot", node.MaxNodes, *devicePath)
Expand All @@ -283,7 +286,7 @@ func main() {
}
nodeReg.SetSlotLostHandler(func(slot int, owner [16]byte) {
failf("Node registry slot %d is no longer this node's (owner %x); stopping so two nodes never share a journal. "+
"Check that the device is reachable and that node clocks agree within a few seconds, then mount again", slot, owner[:8])
"Check that the device is reachable and that this node did not stall, then mount again", slot, owner[:8])
})
if err := nodeReg.StartHeartbeat(); err != nil {
failf("Failed to start node heartbeat: %v", err)
Expand Down
2 changes: 1 addition & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ Use identical mount options and the same GoClusterFS version on all nodes:
-broadcast-secret "your-shared-secret"
```

Read [Node Requirements](OPERATIONS.md#node-requirements) and [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits) first. Node clocks must agree, and nodes must not stall.
Read [Node Requirements](OPERATIONS.md#node-requirements) and [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits) first. Nodes must not stall, and storage must fail I/O rather than queue it.

### Step 4: Verify Cluster Communication

Expand Down
17 changes: 8 additions & 9 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,10 +182,10 @@ Snapshots and clones share blocks with the file and copy a block on its first wr

### Node Requirements

- **Clocks:** Nodes judge each other's heartbeats by wall clock. A node that mounts with its clock more than about 10 seconds ahead treats live peers as failed, and can take a live peer's registry slot and replay that peer's journal. Run NTP on every node.
- **Clocks:** A node judges another node by whether its heartbeat keeps changing, timed on its own clock, so failover works even when node clocks disagree. File times use each node's clock, and `goclusterfs-fsck -release-slot` compares a slot's lease with the clock of the host it runs on. Run NTP on every node.
- **Restarts:** A restarted mount takes back the registry slot of a mount that exited on the same host (same hostname and kernel boot ID) and replays its journal before serving. In the same PID namespace it checks the old process ID; across PID namespaces it waits for the old slot's heartbeat to go stale.
- **Crashed node's locks:** Once a node is proven dead, the node replaying its journal takes over its exclusive locks and frees them after the replay, so no other node uses them mid-replay. Proven dead means its mount process is gone (same host and PID namespace), or its heartbeat is more than 20 seconds old and has not changed for 15 seconds by the observing node's own clock. A node paused for more than about 20 seconds therefore counts as dead; it exits on its next heartbeat. The dead node's read locks are freed at once. Until a failed node is replayed, other nodes wait for its locks however long that takes. Failover takes about 3 to 6 seconds when a surviving mount runs on the same host and in the same PID namespace, and up to about 35 seconds otherwise. A mount that takes a failed node's registry slot keeps that node's locks until it has replayed the slot's journal.
- **Node count:** The device has 64 registry slots, each with its own 4 MB journal. A crashed node's slot frees once another node replays its journal, or when the same host mounts again.
- **Crashed node's locks:** Once a node is proven dead, the node replaying its journal takes over its exclusive locks and frees them after the replay, so no other node uses them mid-replay. Proven dead means its mount process is gone (same host and PID namespace), or the observing node has read the same heartbeat for 20 seconds by its own clock. A node paused for more than about 20 seconds therefore counts as dead; it exits on its next heartbeat. The dead node's read locks are freed at once. Until a failed node is replayed, other nodes wait for its locks however long that takes. Failover takes about 3 to 6 seconds when a surviving mount runs on the same host and in the same PID namespace, and up to about 35 seconds otherwise. A mount that takes a failed node's registry slot keeps that node's locks until it has replayed the slot's journal.
- **Node count:** The device has 64 registry slots, each with its own 4 MB journal. A crashed node's slot frees once another node replays its journal, or when the same host mounts again. A host that rebooted after a crash takes back the slot its hostname held, and replays that journal before serving. Otherwise a mount takes another host's slot only when `-skip-recovery` names it or no slot is empty. Either way it first watches the slot's heartbeat stay unchanged for about 12 seconds.

### Known Gaps in Slot Ownership

Expand Down Expand Up @@ -297,7 +297,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no

| Limit | Why |
|-------|-----|
| Keep node clocks within 1 second (chrony or NTP) | Nodes judge each other's heartbeats by wall clock |
| Run chrony or NTP on every node | File times and `fsck -release-slot` use the wall clock; failover does not |
| No node may stall for 15 seconds or more while mounted | Peers prove a node dead after about 20 seconds of silence and take its locks |
| Storage I/O must fail, not queue, within 10 seconds | A queued write that lands after takeover corrupts metadata. Do not use multipath `queue_if_no_path`; lower iSCSI `node.session.timeo.replacement_timeout` (default 120s) |
| Unmount before you pause, migrate or snapshot a node VM | Unmounting releases its locks |
Expand Down Expand Up @@ -339,15 +339,14 @@ A mount exits with `Journal recovery for slot N failed: ...` when it cannot repl

`-skip-recovery` is for emergencies only: it throws away changes the node had committed, which may have been only partly applied. Never put it in a unit file or script. `goclusterfs-fsck -list-slots` shows each slot's journal state first.

`-skip-recovery=N` discards a journal only when the mount takes slot N. If it takes another slot, it logs `ignoring -skip-recovery=N` and replays that slot's journal as usual, so it never discards a journal you did not name. A mount takes another slot while a running node is replaying slot N: about 20 seconds after the failed mount, a running node tries to replay the damaged slot, holds it for 60 seconds, and tries again every minute. To discard slot N's journal:
`-skip-recovery=N` discards a journal only when the mount takes slot N, and it makes the mount try slot N first. It takes the slot at once if it is empty or holds this host's exited mount; after watching it for about 12 seconds if it holds another host's node whose heartbeat never changes; and after a minute if a running node is replaying it and that replay keeps failing, since the replay's lease then lapses. If it cannot take slot N, it logs `ignoring -skip-recovery=N` and replays the journal of the slot it took, so it never discards a journal you did not name. To discard slot N's journal:

1. Either mount right after the failed mount, within about 20 seconds, or first unmount on every other node and wait one minute.
2. On the same host, before it reboots, mount once with `-skip-recovery=N`. The mount logs `discarding the journal of slot N`.
3. Unmount on every node and run fsck, because discarded transactions may have been only partly applied.
1. Mount once with `-skip-recovery=N`, on any host. The mount logs `discarding the journal of slot N`.
2. Unmount on every node and run fsck, because discarded transactions may have been only partly applied.

`-skip-recovery=N` does not stop the node from replaying crashed peers' journals. `-journal=false` also skips the replay of its own slot, and leaves that journal in place for the next mount, so don't use it on a device that may have a dirty journal.

A slot belongs to its host's hostname and boot ID. If the damaged slot belongs to a node that has rebooted or will not mount again, the other nodes log `Replaying slot N ... failed` and retry every minute. Between retries, a new mount on any host can take that slot and then fail with the same error. Free the slot with fsck. Unmount the filesystem on every node first, then:
A slot belongs to its host's hostname and boot ID. If the damaged slot belongs to a node that has rebooted or will not mount again, the other nodes log `Replaying slot N ... failed`, naming `-skip-recovery=N`, and retry every minute. A mount takes such a slot only when `-skip-recovery=N` names it or no slot is empty. Discard its journal as above, or free the slot with fsck. Unmount the filesystem on every node first, then:

```bash
./goclusterfs-fsck -list-slots /dev/device # find the slot and its journal state
Expand Down
2 changes: 1 addition & 1 deletion docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,7 @@ fusermount3 -u /mnt/cluster

1. **Always unmount cleanly** before shutting down
2. **Keep journaling on** (`-journal`, on by default)
3. **Keep node clocks in sync and avoid stalls** (see [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits))
3. **Avoid stalls, and run NTP on every node** (see [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits))
4. **Run periodic fsck** during maintenance windows
5. **Maintain backups** - no filesystem is immune to hardware failure

Expand Down
7 changes: 3 additions & 4 deletions internal/fsck/slots_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ func TestReleaseSlotLease(t *testing.T) {
var late *node.Registry
t.Cleanup(fsck.SetReleaseHooks(func() {
late = node.NewRegistry(dev, sb.NodeRegistryOffset)
if got, err := late.Register(ctx, "late"); err != nil || got != slot {
if got, err := late.Register(ctx, "gone"); err != nil || got != slot {
t.Errorf("late node took slot %d (%v), want the stale slot %d", got, err, slot)
}
}, nil))
Expand Down Expand Up @@ -202,6 +202,7 @@ func TestReleaseSlotCrash(t *testing.T) {
resetFirst bool
}{{"before reset", false}, {"after reset", true}} {
t.Run(tt.name, func(t *testing.T) {
t.Cleanup(node.SetRecoveryLease(time.Second))
var sb *superblock.Superblock
dev := formattedDevice(t, func(s *superblock.Superblock) { sb = s })
ctx := context.Background()
Expand Down Expand Up @@ -236,9 +237,7 @@ func TestReleaseSlotCrash(t *testing.T) {
t.Fatalf("a peer replayed slot %d during the lease", slot)
}

editSlot(t, dev, sb, slot, func(s []byte) {
binary.LittleEndian.PutUint64(s[172:180], uint64(time.Now().Add(-61*time.Second).Unix()))
})
time.Sleep(2 * time.Second)
replays := detectFailures(peer, dev)
if len(replays) != 1 || replays[0].slot != slot {
t.Fatalf("after the lease lapsed the peer replayed %+v, want slot %d once", replays, slot)
Expand Down
Loading
Loading