diff --git a/CHANGELOG.md b/CHANGELOG.md index 024126c6..db3aac9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - The on-disk format is now version 3. Mount refuses devices formatted by earlier builds. Version 1 kept one journal for all nodes. Version 2 had room for 4096 inodes, and on devices over 128 GB its refcounts overwrote the snapshot registry and then data blocks. Copy the data off with the old build, reformat with `-format`, and copy it back. - Devices must be at least 327 MB: the layout reserves 256 MB for 64 node journals of 4 MB and 64 MB for lock records. - Mount also refuses version 3 devices formatted before lock records had one claim slot per node, or before each lock type had its own lock records. Older builds refuse devices this version formats. Copy the data off with the build that formatted them and reformat. -- Node clocks must agree within a few seconds. See the node requirements in `docs/OPERATIONS.md`. - Upgrade all nodes together: unmount on every node, upgrade, then mount again. Don't mount one shared device with a mix of old and new binaries. - Older binaries reset an inode's generation counters to 0 when they free it, so other nodes can serve stale data after the inode number is reused. - Older binaries zero the extent-tree generation (inode bytes 188–195) when they rewrite an inode. Newer binaries then cannot read that file if it has more than two extents, because such files use an extent tree. @@ -86,6 +85,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - `goclusterfs-fsck` ran for hours on a superblock with a damaged node count, reading one journal per node it claimed. It now reads at most the 64 that the node registry has. - One failed write while a node released a lock left its claim on the device, and the node forgot it: other nodes timed out with `EIO` on every resource in that lock record until the node unmounted. The node now clears such a claim again every second until the write succeeds. - One failed read of the snapshot registry during a snapshot create could delete other snapshots: the create counted the unreadable sector's slots as free, and a write to a sector it could not read replaced the sector with one that held only the new snapshot, losing up to 30 others. The create still reported success. Both now fail with `EIO` and leave the sector alone. A sector that stays unreadable now blocks every snapshot create; `goclusterfs-fsck` reports it but cannot repair it yet. +- Nodes judged each other's heartbeats and recovery leases by wall clock. A node mounting with its clock more than 10 seconds ahead took a live node's registry slot and stopped it; a crashed node whose clock ran ahead was replayed only once its clock's lead ran out; and a replay whose node's clock was behind looked abandoned, so lock waits for the dead node failed with `EIO`. A node now counts another as failing, dead or abandoned only after reading the same heartbeat or lease for long enough by its own clock; a mount takes another host's slot only when it held that hostname before a reboot, when `-skip-recovery` names it, or when no slot is empty, and only after watching its heartbeat stay unchanged for about 12 seconds. +- Under slow storage a node was declared dead while it was only slow: registry scans held the heartbeat back for 64 reads each, so at 1 second per I/O a peer replayed the node and it stopped. Scans now read the registry in one I/O without holding the heartbeat's lock, and registry I/O no longer waits for other I/O to the same 1 MB region. - A node in a higher registry slot could wait 30 seconds and fail with `EIO` for a lock that a node in a lower slot kept releasing and claiming again every few milliseconds. A node that loses a claim to a lower slot now leaves a waiting mark in its own slot, and the holder gives the lock up 250 ms after it first sees a live node's mark. - `chmod`, `chown` and `touch` could undo another node's concurrent change to the same inode: on a directory, a create's link count or mtime; on a file, the size an append inside an allocated block set. Directory attribute changes now hold the inode lock from read to write, and file attribute changes retry when the inode changed since they read it. - After 3 failed multicast sends in total, counted over the whole mount, a node sent cache invalidations only to unicast peers until it unmounted. Now only failures in a row count: a successful send resets them, and after 3 in a row the node tries multicast again 30 seconds later. diff --git a/CLAUDE.md b/CLAUDE.md index cb80185c..6b02fb56 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,7 +95,7 @@ Clone paths are absolute from the filesystem root, and clones can only go in the ## Current Limitations - Linux only. Needs FUSE and a backing store that supports O_DIRECT. -- At most 64 nodes mount a volume at once, one per registry slot. Heartbeats compare wall clocks, so node clocks must agree within a few seconds. +- At most 64 nodes mount a volume at once, one per registry slot. A node judges another by whether its heartbeat changes, timed on its own clock, so failover does not need clocks to agree. - `goclusterfs-fsck -repair` fixes only block refcounts, sharing that older versions left uncounted, leaked blocks, inodes no directory names, and the superblock checksum. It reports every other problem without fixing it. - The inode count is set at format time and cannot change. Every `statfs` (`df`) reads the whole inode table to count free inodes: 50 MB on a 200 GB device at the default ratio. - SFB is disabled: writes never allocate SFB slots, so every file uses full 1 MB blocks. `internal/sfb` container I/O is broken (unaligned header buffers, no data offset) and stays until it is fixed or removed. diff --git a/README.md b/README.md index e54b9e6a..0a3bac0b 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod - **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md). - **Snapshots and clones are experimental.** Before you mount an image that has snapshots or clones with a newer build, read the upgrade notes in [CHANGELOG.md](CHANGELOG.md). They say whether to run `goclusterfs-fsck` first. - **fsck repairs little.** `-repair` rebuilds refcounts, frees leaked blocks and inodes no directory names, and fixes the superblock checksum. It only reports other problems. See [Filesystem Check](#filesystem-check). -- **Cluster safety limits.** Node clocks, node stalls, and storage timeouts have hard limits. Read [Node Requirements](docs/OPERATIONS.md#node-requirements) and [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) before you run more than one node. +- **Cluster safety limits.** Node stalls and storage timeouts have hard limits. Read [Node Requirements](docs/OPERATIONS.md#node-requirements) and [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) before you run more than one node. ## Quick Start @@ -195,7 +195,7 @@ For clustered deployments: 1. **Shared Storage**: All nodes must access the same block device (SAN, iSCSI, NVMe-oF) 2. **Network**: Nodes should be on the same subnet for multicast invalidation 3. **Mount Command**: Use identical mount options across all nodes -4. **Limits**: At most 64 nodes mount a device at once, and node clocks must agree within a few seconds (run NTP) +4. **Limits**: At most 64 nodes mount a device at once; run NTP so file times agree 5. **Format version**: This release uses on-disk format version 3, which gives each node its own journal and sizes the block bitmap, refcount table and inode table from the device. It refuses devices formatted by earlier builds; copy their data off and reformat 6. **Stalled nodes**: Other nodes declare a node dead after about 20 seconds of silence, for example a paused VM, and replay its journal. The stalled node checks its registry slot before each write, so when it resumes it refuses new writes and exits. A write it started before the stall can still land. Only storage fencing closes that gap, such as SCSI-3 persistent reservations, which GoClusterFS does not use. See [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) diff --git a/cmd/goclusterfs-mount/main.go b/cmd/goclusterfs-mount/main.go index 9d11d8c7..08ee9578 100644 --- a/cmd/goclusterfs-mount/main.go +++ b/cmd/goclusterfs-mount/main.go @@ -274,6 +274,9 @@ func main() { nodeReg := node.NewRegistry(dev, nodeRegistryOffset) nodeReg.SetJournalArea(sb.JournalRegionOffset, sb.JournalSlotSize()) hostname, _ := os.Hostname() + if *skipRecovery >= 0 { + nodeReg.PreferSlot(*skipRecovery) + } slotIdx, err := nodeReg.Register(ctx, hostname) if errors.Is(err, node.ErrNoFreeSlots) { failf("Cannot join the cluster: all %d node slots are taken. List them with goclusterfs-fsck -list-slots %s, and free the slot of a node that will not come back with -release-slot", node.MaxNodes, *devicePath) @@ -283,7 +286,7 @@ func main() { } nodeReg.SetSlotLostHandler(func(slot int, owner [16]byte) { failf("Node registry slot %d is no longer this node's (owner %x); stopping so two nodes never share a journal. "+ - "Check that the device is reachable and that node clocks agree within a few seconds, then mount again", slot, owner[:8]) + "Check that the device is reachable and that this node did not stall, then mount again", slot, owner[:8]) }) if err := nodeReg.StartHeartbeat(); err != nil { failf("Failed to start node heartbeat: %v", err) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index e0e61514..61914abe 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -293,7 +293,7 @@ Use identical mount options and the same GoClusterFS version on all nodes: -broadcast-secret "your-shared-secret" ``` -Read [Node Requirements](OPERATIONS.md#node-requirements) and [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits) first. Node clocks must agree, and nodes must not stall. +Read [Node Requirements](OPERATIONS.md#node-requirements) and [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits) first. Nodes must not stall, and storage must fail I/O rather than queue it. ### Step 4: Verify Cluster Communication diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index ec70916d..2f41408f 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -182,10 +182,10 @@ Snapshots and clones share blocks with the file and copy a block on its first wr ### Node Requirements -- **Clocks:** Nodes judge each other's heartbeats by wall clock. A node that mounts with its clock more than about 10 seconds ahead treats live peers as failed, and can take a live peer's registry slot and replay that peer's journal. Run NTP on every node. +- **Clocks:** A node judges another node by whether its heartbeat keeps changing, timed on its own clock, so failover works even when node clocks disagree. File times use each node's clock, and `goclusterfs-fsck -release-slot` compares a slot's lease with the clock of the host it runs on. Run NTP on every node. - **Restarts:** A restarted mount takes back the registry slot of a mount that exited on the same host (same hostname and kernel boot ID) and replays its journal before serving. In the same PID namespace it checks the old process ID; across PID namespaces it waits for the old slot's heartbeat to go stale. -- **Crashed node's locks:** Once a node is proven dead, the node replaying its journal takes over its exclusive locks and frees them after the replay, so no other node uses them mid-replay. Proven dead means its mount process is gone (same host and PID namespace), or its heartbeat is more than 20 seconds old and has not changed for 15 seconds by the observing node's own clock. A node paused for more than about 20 seconds therefore counts as dead; it exits on its next heartbeat. The dead node's read locks are freed at once. Until a failed node is replayed, other nodes wait for its locks however long that takes. Failover takes about 3 to 6 seconds when a surviving mount runs on the same host and in the same PID namespace, and up to about 35 seconds otherwise. A mount that takes a failed node's registry slot keeps that node's locks until it has replayed the slot's journal. -- **Node count:** The device has 64 registry slots, each with its own 4 MB journal. A crashed node's slot frees once another node replays its journal, or when the same host mounts again. +- **Crashed node's locks:** Once a node is proven dead, the node replaying its journal takes over its exclusive locks and frees them after the replay, so no other node uses them mid-replay. Proven dead means its mount process is gone (same host and PID namespace), or the observing node has read the same heartbeat for 20 seconds by its own clock. A node paused for more than about 20 seconds therefore counts as dead; it exits on its next heartbeat. The dead node's read locks are freed at once. Until a failed node is replayed, other nodes wait for its locks however long that takes. Failover takes about 3 to 6 seconds when a surviving mount runs on the same host and in the same PID namespace, and up to about 35 seconds otherwise. A mount that takes a failed node's registry slot keeps that node's locks until it has replayed the slot's journal. +- **Node count:** The device has 64 registry slots, each with its own 4 MB journal. A crashed node's slot frees once another node replays its journal, or when the same host mounts again. A host that rebooted after a crash takes back the slot its hostname held, and replays that journal before serving. Otherwise a mount takes another host's slot only when `-skip-recovery` names it or no slot is empty. Either way it first watches the slot's heartbeat stay unchanged for about 12 seconds. ### Known Gaps in Slot Ownership @@ -297,7 +297,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no | Limit | Why | |-------|-----| -| Keep node clocks within 1 second (chrony or NTP) | Nodes judge each other's heartbeats by wall clock | +| Run chrony or NTP on every node | File times and `fsck -release-slot` use the wall clock; failover does not | | No node may stall for 15 seconds or more while mounted | Peers prove a node dead after about 20 seconds of silence and take its locks | | Storage I/O must fail, not queue, within 10 seconds | A queued write that lands after takeover corrupts metadata. Do not use multipath `queue_if_no_path`; lower iSCSI `node.session.timeo.replacement_timeout` (default 120s) | | Unmount before you pause, migrate or snapshot a node VM | Unmounting releases its locks | @@ -339,15 +339,14 @@ A mount exits with `Journal recovery for slot N failed: ...` when it cannot repl `-skip-recovery` is for emergencies only: it throws away changes the node had committed, which may have been only partly applied. Never put it in a unit file or script. `goclusterfs-fsck -list-slots` shows each slot's journal state first. -`-skip-recovery=N` discards a journal only when the mount takes slot N. If it takes another slot, it logs `ignoring -skip-recovery=N` and replays that slot's journal as usual, so it never discards a journal you did not name. A mount takes another slot while a running node is replaying slot N: about 20 seconds after the failed mount, a running node tries to replay the damaged slot, holds it for 60 seconds, and tries again every minute. To discard slot N's journal: +`-skip-recovery=N` discards a journal only when the mount takes slot N, and it makes the mount try slot N first. It takes the slot at once if it is empty or holds this host's exited mount; after watching it for about 12 seconds if it holds another host's node whose heartbeat never changes; and after a minute if a running node is replaying it and that replay keeps failing, since the replay's lease then lapses. If it cannot take slot N, it logs `ignoring -skip-recovery=N` and replays the journal of the slot it took, so it never discards a journal you did not name. To discard slot N's journal: -1. Either mount right after the failed mount, within about 20 seconds, or first unmount on every other node and wait one minute. -2. On the same host, before it reboots, mount once with `-skip-recovery=N`. The mount logs `discarding the journal of slot N`. -3. Unmount on every node and run fsck, because discarded transactions may have been only partly applied. +1. Mount once with `-skip-recovery=N`, on any host. The mount logs `discarding the journal of slot N`. +2. Unmount on every node and run fsck, because discarded transactions may have been only partly applied. `-skip-recovery=N` does not stop the node from replaying crashed peers' journals. `-journal=false` also skips the replay of its own slot, and leaves that journal in place for the next mount, so don't use it on a device that may have a dirty journal. -A slot belongs to its host's hostname and boot ID. If the damaged slot belongs to a node that has rebooted or will not mount again, the other nodes log `Replaying slot N ... failed` and retry every minute. Between retries, a new mount on any host can take that slot and then fail with the same error. Free the slot with fsck. Unmount the filesystem on every node first, then: +A slot belongs to its host's hostname and boot ID. If the damaged slot belongs to a node that has rebooted or will not mount again, the other nodes log `Replaying slot N ... failed`, naming `-skip-recovery=N`, and retry every minute. A mount takes such a slot only when `-skip-recovery=N` names it or no slot is empty. Discard its journal as above, or free the slot with fsck. Unmount the filesystem on every node first, then: ```bash ./goclusterfs-fsck -list-slots /dev/device # find the slot and its journal state diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 9e8bcbb9..d17fb686 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -475,7 +475,7 @@ fusermount3 -u /mnt/cluster 1. **Always unmount cleanly** before shutting down 2. **Keep journaling on** (`-journal`, on by default) -3. **Keep node clocks in sync and avoid stalls** (see [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits)) +3. **Avoid stalls, and run NTP on every node** (see [Distributed Lock Limits](OPERATIONS.md#distributed-lock-limits)) 4. **Run periodic fsck** during maintenance windows 5. **Maintain backups** - no filesystem is immune to hardware failure diff --git a/internal/fsck/slots_test.go b/internal/fsck/slots_test.go index 83b8b080..1d29ef25 100644 --- a/internal/fsck/slots_test.go +++ b/internal/fsck/slots_test.go @@ -157,7 +157,7 @@ func TestReleaseSlotLease(t *testing.T) { var late *node.Registry t.Cleanup(fsck.SetReleaseHooks(func() { late = node.NewRegistry(dev, sb.NodeRegistryOffset) - if got, err := late.Register(ctx, "late"); err != nil || got != slot { + if got, err := late.Register(ctx, "gone"); err != nil || got != slot { t.Errorf("late node took slot %d (%v), want the stale slot %d", got, err, slot) } }, nil)) @@ -202,6 +202,7 @@ func TestReleaseSlotCrash(t *testing.T) { resetFirst bool }{{"before reset", false}, {"after reset", true}} { t.Run(tt.name, func(t *testing.T) { + t.Cleanup(node.SetRecoveryLease(time.Second)) var sb *superblock.Superblock dev := formattedDevice(t, func(s *superblock.Superblock) { sb = s }) ctx := context.Background() @@ -236,9 +237,7 @@ func TestReleaseSlotCrash(t *testing.T) { t.Fatalf("a peer replayed slot %d during the lease", slot) } - editSlot(t, dev, sb, slot, func(s []byte) { - binary.LittleEndian.PutUint64(s[172:180], uint64(time.Now().Add(-61*time.Second).Unix())) - }) + time.Sleep(2 * time.Second) replays := detectFailures(peer, dev) if len(replays) != 1 || replays[0].slot != slot { t.Fatalf("after the lease lapsed the peer replayed %+v, want slot %d once", replays, slot) diff --git a/internal/node/clock_skew_test.go b/internal/node/clock_skew_test.go new file mode 100644 index 00000000..6f49f387 --- /dev/null +++ b/internal/node/clock_skew_test.go @@ -0,0 +1,149 @@ +package node + +import ( + "context" + "errors" + "testing" + "time" +) + +// keepWriting rewrites slot idx every 250ms as a live node whose clock is lag +// behind would: its heartbeat, or for a dead slot its recovery lease, advances. +func keepWriting(t *testing.T, reg *Registry, idx int, slot *Slot, lag time.Duration) (stop func()) { + done, stopped := make(chan struct{}), make(chan struct{}) + go func() { + defer close(stopped) + for { + select { + case <-done: + return + case <-time.After(250 * time.Millisecond): + } + if slot.State == NodeStateDead { + slot.RecoveryStart = max(time.Now().Add(-lag).Unix(), slot.RecoveryStart+1) + } else { + slot.HeartbeatSeq++ + slot.HeartbeatTime = time.Now().Add(-lag).Unix() + } + if err := reg.writeSlot(idx, slot); err != nil { + t.Error(err) + return + } + } + }() + return func() { close(done); <-stopped } +} + +func TestRegisterLeavesLiveSlotOfNodeWithSlowClock(t *testing.T) { + shortenSilentSlotWatch(t) + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + fillWithReplays(t, reg, 1) + + live := &Slot{Magic: SlotMagic, State: NodeStateActive, UUID: [16]byte{7}, HeartbeatSeq: 1, HeartbeatTime: time.Now().Add(-2 * time.Minute).Unix()} + copy(live.Hostname[:], "other-host") + if err := reg.writeSlot(0, live); err != nil { + t.Fatal(err) + } + defer keepWriting(t, reg, 0, live, 2*time.Minute)() + time.Sleep(time.Second) + + if slot, err := reg.Register(context.Background(), "new-host"); !errors.Is(err, ErrNoFreeSlots) { + t.Errorf("Register = %d, %v; want ErrNoFreeSlots: the only slot not empty or under recovery holds a live node whose clock is 2 minutes behind", slot, err) + } +} + +func TestRenewedRecoveryLeaseIsHeldWhateverTheReplayersClock(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + + failed := &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: [16]byte{5}, RecoveryOwner: [16]byte{6}, RecoveryStart: time.Now().Add(-2 * time.Minute).Unix()} + if err := reg.writeSlot(4, failed); err != nil { + t.Fatal(err) + } + silence(reg, 4, failed, recoveryLease+time.Second) + if _, state := reg.Occupant(4, true); state != OccupiedLapsed { + t.Fatalf("a lease unchanged for longer than recoveryLease: %v, want lapsed", state) + } + + stop := keepWriting(t, reg, 4, failed, 2*time.Minute) + time.Sleep(time.Second) + if _, state := reg.Occupant(4, true); state != OccupiedReplaying { + t.Errorf("a lease the replayer keeps renewing, with its clock 2 minutes behind: %v, want replaying", state) + } + _, err := reg.acquireForRecovery(4, failed, HeartbeatTimeout) + stop() + if err == nil { + t.Error("took over the recovery of a replayer that keeps renewing its lease") + } +} + +func TestCrashedNodeWithFastClockIsReplayed(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + if _, err := reg.Register(context.Background(), "observer"); err != nil { + t.Fatal(err) + } + crashed := &Slot{Magic: SlotMagic, State: NodeStateActive, UUID: [16]byte{9}, HeartbeatSeq: 7, HeartbeatTime: time.Now().Add(3 * time.Minute).Unix()} + copy(crashed.Hostname[:], "other-host") + if err := reg.writeSlot(5, crashed); err != nil { + t.Fatal(err) + } + replayed := false + reg.SetRecoveryHandler(func(info *FailedNodeInfo) error { + replayed = replayed || info.SlotIndex == 5 + return nil + }) + + silence(reg, 5, crashed, HeartbeatTimeout+time.Second) + if _, state := reg.Occupant(5, false); state != OccupiedFailing { + t.Errorf("a heartbeat 3 minutes in the future, unchanged for longer than HeartbeatTimeout: %v, want failing", state) + } + silence(reg, 5, crashed, deadAfter+time.Second) + reg.detectAndRecoverFailedNodes() + if !replayed { + t.Error("a node whose heartbeat stopped for longer than deadAfter was not replayed, because its clock was 3 minutes ahead") + } +} + +func TestSlowRegistryReadsDoNotStarveHeartbeat(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + if _, err := reg.Register(context.Background(), "slow"); err != nil { + t.Fatal(err) + } + dev.SetReadFault(func(int64) error { + time.Sleep(100 * time.Millisecond) + return nil + }) + defer dev.SetReadFault(func(int64) error { return nil }) + if err := reg.StartHeartbeat(); err != nil { + t.Fatal(err) + } + defer func() { _ = reg.Unregister(context.Background()) }() + stop := make(chan struct{}) + defer close(stop) + go func() { + for { + select { + case <-stop: + return + default: + } + _, _ = reg.GetActiveNodes() + reg.detectAndRecoverFailedNodes() + } + }() + + var worst time.Duration + for start := time.Now(); time.Since(start) < 8*time.Second; time.Sleep(100 * time.Millisecond) { + worst = max(worst, reg.sinceLastBeat()) + } + if worst >= HeartbeatTimeout/2 { + t.Errorf("with 100ms registry reads and registry scans running, the heartbeat went %v without a beat", worst.Round(100*time.Millisecond)) + } +} diff --git a/internal/node/liveness.go b/internal/node/liveness.go index f37c8a05..f8f0d132 100644 --- a/internal/node/liveness.go +++ b/internal/node/liveness.go @@ -9,19 +9,34 @@ import ( ) // recoveryLease is how long a peer may hold a dead slot's recovery before others take it over. -const recoveryLease = 60 * time.Second +var recoveryLease = 60 * time.Second -// deathMargin is added to HeartbeatTimeout before this node's own clock counts -// an unchanged heartbeat as proof that its node is dead rather than slow. -const deathMargin = HeartbeatTimeout / 2 +// SetRecoveryLease shortens the recovery lease for tests in other packages, and returns a func that restores it. +func SetRecoveryLease(d time.Duration) (restore func()) { + old := recoveryLease + recoveryLease = d + return func() { recoveryLease = old } +} + +// deadAfter is how long this node must read the same heartbeat in a slot before +// it counts the slot's node as dead rather than slow, and replays it. +const deadAfter = 2 * HeartbeatTimeout -// sighting is what failure detection last saw in a slot, timed by this node's monotonic clock. +// sighting is what this node last read in a slot: the fields a live node keeps +// changing, and since when this node's monotonic clock has seen them unchanged. type sighting struct { - uuid [16]byte - seq uint64 - beat int64 - since time.Time - changedAt time.Time + uuid [16]byte + state NodeState + seq uint64 + beat int64 + recoveryOwner [16]byte + recoveryStart int64 + since time.Time +} + +func sightingOf(slot *Slot) sighting { + return sighting{uuid: slot.UUID, state: slot.State, seq: slot.HeartbeatSeq, beat: slot.HeartbeatTime, + recoveryOwner: slot.RecoveryOwner, recoveryStart: slot.RecoveryStart} } var processStart = time.Now().UnixNano() @@ -70,18 +85,28 @@ func (r *Registry) deadLocalOwner(slot *Slot) bool { return syscall.Kill(int(slot.PID), 0) == syscall.ESRCH } -// failed reports whether a slot's node is gone: its heartbeat is older than timeout, it is a -// mount process on this host that has exited, or the peer replaying its journal let the lease lapse. -func (r *Registry) failed(slot *Slot, timeout time.Duration) bool { +// stale reports whether a slot's node is gone: it is a mount process on this +// host that has exited, or this node has read the same heartbeat in the slot for +// longer than timeout, or for a dead slot the same recovery lease for longer than +// recoveryLease. Silence is timed on this node's clock, so node clocks need not +// agree. A slot this node has not watched yet is not stale. Called with r.mu held +// after observe. +func (r *Registry) stale(slotIdx int, slot *Slot, timeout time.Duration) bool { switch slot.State { case NodeStateActive, NodeStateLeaving: - return time.Since(time.Unix(slot.HeartbeatTime, 0)) > timeout || r.deadLocalOwner(slot) + return r.deadLocalOwner(slot) || r.silentFor(slotIdx, slot) > timeout case NodeStateDead: - return time.Since(time.Unix(slot.RecoveryStart, 0)) > recoveryLease + return r.silentFor(slotIdx, slot) > recoveryLease } return false } +// heartbeatOlderThan compares a heartbeat with this node's wall clock. Only a +// slot this host wrote may be judged so, since only then did the same clock write it. +func heartbeatOlderThan(slot *Slot, d time.Duration) bool { + return time.Since(time.Unix(slot.HeartbeatTime, 0)) > d +} + // LeaseExpiry returns when the slot's node stops counting as alive: its last // heartbeat plus HeartbeatTimeout, or for a dead slot, the end of the recovery // lease of the peer replaying its journal. An empty slot returns the zero time. @@ -95,37 +120,28 @@ func (s *Slot) LeaseExpiry() time.Time { return time.Time{} } -// observe records a slot's heartbeat as failure detection sees it. Called with r.mu held. +// observe records what this node read in a slot. Called with r.mu held. func (r *Registry) observe(slotIdx int, slot *Slot) { - last, seen := r.sightings[slotIdx] - if seen && last.uuid == slot.UUID && last.seq == slot.HeartbeatSeq && last.beat == slot.HeartbeatTime { + next := sightingOf(slot) + if last, seen := r.sightings[slotIdx]; seen && last.sinceless() == next { return } - now := time.Now() - next := sighting{uuid: slot.UUID, seq: slot.HeartbeatSeq, beat: slot.HeartbeatTime, since: now} - if seen && last.uuid == slot.UUID { - next.changedAt = now - } + next.since = time.Now() r.sightings[slotIdx] = next } -// recentlyAlive reports whether this node saw the slot's heartbeat advance within -// HeartbeatTimeout, whatever the heartbeat's wall-clock time says. Called with r.mu held. -func (r *Registry) recentlyAlive(slotIdx int) bool { - last, seen := r.sightings[slotIdx] - return seen && !last.changedAt.IsZero() && time.Since(last.changedAt) < HeartbeatTimeout +func (s sighting) sinceless() sighting { + s.since = time.Time{} + return s } -// provenDead reports whether a slot's node is dead, not merely slow: its mount -// process is gone, this node has seen the same heartbeat for longer than -// HeartbeatTimeout plus deathMargin, or the slot is marked dead, which a node -// does only after proving it or fsck after releasing it. Called with r.mu held. -func (r *Registry) provenDead(slotIdx int, slot *Slot) bool { - if slot.State == NodeStateDead || r.deadLocalOwner(slot) { - return true - } +// silentFor returns how long this node has read slot unchanged. Called with r.mu held after observe. +func (r *Registry) silentFor(slotIdx int, slot *Slot) time.Duration { last, seen := r.sightings[slotIdx] - return seen && last.uuid == slot.UUID && time.Since(last.since) >= HeartbeatTimeout+deathMargin + if !seen || last.sinceless() != sightingOf(slot) { + return 0 + } + return time.Since(last.since) } // Occupancy is how the node in a registry slot stands. @@ -160,16 +176,17 @@ func (r *Registry) Occupant(slotIdx int, fresh bool) ([16]byte, Occupancy) { r.remember(slotIdx, read) slot = read } - r.mu.RLock() - defer r.mu.RUnlock() + r.mu.Lock() + defer r.mu.Unlock() + r.observe(slotIdx, slot) switch { case slot.State == NodeStateEmpty: return slot.UUID, Unoccupied - case slot.State == NodeStateDead && r.failed(slot, HeartbeatTimeout): + case slot.State == NodeStateDead && r.stale(slotIdx, slot, HeartbeatTimeout): return slot.UUID, OccupiedLapsed case slot.State == NodeStateDead: return slot.UUID, OccupiedReplaying - case r.failed(slot, HeartbeatTimeout): + case r.stale(slotIdx, slot, HeartbeatTimeout): return slot.UUID, OccupiedFailing } return slot.UUID, OccupiedLive diff --git a/internal/node/registry.go b/internal/node/registry.go index 255a2a52..7a8a1df4 100644 --- a/internal/node/registry.go +++ b/internal/node/registry.go @@ -100,6 +100,7 @@ type Registry struct { localUUID [16]byte localSlot int // -1 if not registered + preferred int // slot Register tries first, or -1 stopCh chan struct{} heartbeatWg sync.WaitGroup @@ -176,6 +177,7 @@ func NewRegistry(dev *storage.Device, registryOffset uint64) *Registry { dev: dev, offset: registryOffset, localSlot: -1, + preferred: -1, stopCh: make(chan struct{}), failureCheckInterval: FailureCheckInterval, recoveryInProgress: make(map[int]bool), @@ -281,8 +283,26 @@ func (r *Registry) Register(ctx context.Context, hostname string) (int, error) { copy(name[:], hostname) r.hostname = string(bytes.TrimRight(name[:], "\x00")) + empty := func(slot *Slot) bool { return slot.State == NodeStateEmpty } ownFailed := func(slot *Slot) bool { - return r.sameHost(slot) && (r.deadLocalOwner(slot) || r.failed(slot, HeartbeatTimeout)) + return r.sameHost(slot) && (slot.State == NodeStateActive || slot.State == NodeStateLeaving) && + (r.deadLocalOwner(slot) || heartbeatOlderThan(slot, HeartbeatTimeout)) + } + // The slot -skip-recovery names comes first: the operator chose it. + if r.preferred >= 0 { + if i, err := r.claimFirst([]int{r.preferred}, func(slot *Slot) bool { return empty(slot) || ownFailed(slot) }); i >= 0 || err != nil { + return i, err + } + // A slot under another node's recovery is taken only once its lease has lapsed unrenewed. + if _, slot, err := r.readSlotRaw(r.preferred); err == nil && slot.State != NodeStateEmpty { + watch := silentSlotWatch + if slot.State == NodeStateDead { + watch = recoveryLease + } + if i, err := r.claimWatched(r.awaitSilentSlots([]int{r.preferred}, watch)); i >= 0 || err != nil { + return i, err + } + } } if i, err := r.claimFirst(allSlots(), ownFailed); i >= 0 || err != nil { return i, err @@ -291,15 +311,125 @@ func (r *Registry) Register(ctx context.Context, hostname string) (int, error) { if i, err := r.claimFirst(append(r.awaitOtherNamespaceSlots(), allSlots()...), ownFailed); i >= 0 || err != nil { return i, err } - i, err := r.claimFirst(allSlots(), func(slot *Slot) bool { - return slot.State == NodeStateEmpty || r.failed(slot, HeartbeatTimeout) - }) + // A slot under this hostname from an earlier boot is likely this host's mount before a crash; taking it back + // replays its journal before this mount serves. Another host may share the name, so it is watched first. + if i, err := r.claimWatched(r.awaitSilentSlots(r.earlierBootSlots(), silentSlotWatch)); i >= 0 || err != nil { + return i, err + } + if i, err := r.claimFirst(allSlots(), empty); i >= 0 || err != nil { + return i, err + } + // Another host's heartbeat that looks old by this clock may be a live node's whose clock is behind, so its slot is + // taken only once the heartbeat has not changed for silentSlotWatch by this node's clock. + i, err := r.claimWatched(r.awaitSilentSlots(r.otherHostSlots(), silentSlotWatch)) if i < 0 && err == nil { err = ErrNoFreeSlots } return i, err } +// PreferSlot makes Register try slot first: it takes it if it is empty, or if +// what it watches in the slot stays unchanged: a node's heartbeat for +// silentSlotWatch, or a recovery lease for recoveryLease. Mount passes the slot +// -skip-recovery names, so the mount can take a crashed node's slot and discard +// its journal. +func (r *Registry) PreferSlot(slot int) { + r.mu.Lock() + defer r.mu.Unlock() + r.preferred = slot +} + +// silentSlotWatch is how long Register watches another host's slot before it +// takes it, if its heartbeat never changed: past HeartbeatTimeout, so a node +// that only stalled checks its slot before its next write. Tests shorten it. +var silentSlotWatch = HeartbeatTimeout + 2*HeartbeatInterval + +// earlierBootSlots returns the slots registered under this node's hostname in +// another kernel boot whose heartbeat looks old by this node's clock. +func (r *Registry) earlierBootSlots() []int { + var slots []int + for i := 0; i < MaxNodes; i++ { + _, slot, err := r.readSlotRaw(i) + if err == nil && (slot.State == NodeStateActive || slot.State == NodeStateLeaving) && !r.sameHost(slot) && + slot.Host() == r.hostname && heartbeatOlderThan(slot, HeartbeatTimeout) { + slots = append(slots, i) + } + } + return slots +} + +// otherHostSlots returns the slots of other hosts' nodes whose heartbeat looks old by this node's clock. +func (r *Registry) otherHostSlots() []int { + var slots []int + for i := 0; i < MaxNodes; i++ { + _, slot, err := r.readSlotRaw(i) + if err == nil && (slot.State == NodeStateActive || slot.State == NodeStateLeaving) && !r.sameHost(slot) && + heartbeatOlderThan(slot, HeartbeatTimeout) { + slots = append(slots, i) + } + } + return slots +} + +// watchedSlot is a slot and the bytes awaitSilentSlots read in it throughout its watch. +type watchedSlot struct { + slot int + raw []byte +} + +// afterSilentWatch, when a test sets it, runs after awaitSilentSlots ends its watch. +var afterSilentWatch func() + +// claimWatched claims the first slot that still holds the bytes watched in it, +// so a slot a peer marked dead, or whose node beat again, since is left alone. +func (r *Registry) claimWatched(slots []watchedSlot) (int, error) { + for _, w := range slots { + if err := r.claimSlot(w.slot, w.raw); err != nil { + if errors.Is(err, ErrSlotTaken) { + continue + } + return -1, err + } + slot, err := deserializeSlot(w.raw) + if err != nil { + return -1, err + } + r.took(w.slot, slot) + return w.slot, nil + } + return -1, nil +} + +// awaitSilentSlots reads slots until each has changed or for d, and returns the +// ones that never changed with the bytes they held. It times the wait on this node's clock. +func (r *Registry) awaitSilentSlots(slots []int, d time.Duration) []watchedSlot { + first := map[int][]byte{} + for _, i := range slots { + if raw, _, err := r.readSlotRaw(i); err == nil { + first[i] = raw + } + } + deadline := time.Now().Add(d) + for len(first) > 0 && time.Now().Before(deadline) { + time.Sleep(HeartbeatInterval / 4) + for i, was := range first { + if raw, _, err := r.readSlotRaw(i); err != nil || !bytes.Equal(raw, was) { + delete(first, i) + } + } + } + if afterSilentWatch != nil { + afterSilentWatch() + } + var silent []watchedSlot + for _, i := range slots { + if raw, ok := first[i]; ok { + silent = append(silent, watchedSlot{i, raw}) + } + } + return silent +} + func allSlots() []int { slots := make([]int, MaxNodes) for i := range slots { @@ -321,20 +451,26 @@ func (r *Registry) claimFirst(slots []int, claimable func(*Slot) bool) (int, err } return -1, err } - r.localSlot = i - r.beatSeq = 1 - now := time.Now() - r.lastBeat.Store(&now) - if slot.State != NodeStateEmpty { - r.previousOwner = &slot.UUID - } + r.took(i, slot) return i, nil } return -1, nil } +// took records that this node registered in slot i, which held slot. Called with r.mu held. +func (r *Registry) took(i int, slot *Slot) { + r.localSlot = i + r.beatSeq = 1 + now := time.Now() + r.lastBeat.Store(&now) + if slot.State != NodeStateEmpty { + r.previousOwner = &slot.UUID + } +} + // awaitOtherNamespaceSlots watches the live-looking slots of this host in -// another PID namespace until each heartbeats or goes stale, and returns the stale ones. +// another PID namespace until each heartbeats or goes stale, and returns the +// stale ones. The slots share this host's clock, so wall time judges them. func (r *Registry) awaitOtherNamespaceSlots() []int { type beat struct { seq uint64 @@ -343,7 +479,7 @@ func (r *Registry) awaitOtherNamespaceSlots() []int { watch := map[int]beat{} for i := 0; i < MaxNodes; i++ { _, slot, err := r.readSlotRaw(i) - if err == nil && r.sameHost(slot) && slot.PIDNamespace != r.pidNS && !r.failed(slot, HeartbeatTimeout) { + if err == nil && r.sameHost(slot) && slot.PIDNamespace != r.pidNS && !heartbeatOlderThan(slot, HeartbeatTimeout) { watch[i] = beat{slot.HeartbeatSeq, slot.HeartbeatTime} } } @@ -357,7 +493,7 @@ func (r *Registry) awaitOtherNamespaceSlots() []int { switch { case err != nil || beat{slot.HeartbeatSeq, slot.HeartbeatTime} != last: delete(watch, i) - case r.failed(slot, HeartbeatTimeout): + case heartbeatOlderThan(slot, HeartbeatTimeout): stale = append(stale, i) delete(watch, i) } @@ -600,7 +736,7 @@ func (r *Registry) casSlot(slotIdx int, expected, new []byte) (bool, error) { defer storage.PutBuffer(buf) // Read current value - if _, err := r.dev.ReadAt(buf.Bytes(), offset); err != nil { + if _, err := r.dev.ReadAtUnsharded(buf.Bytes(), offset); err != nil { return false, err } @@ -613,7 +749,7 @@ func (r *Registry) casSlot(slotIdx int, expected, new []byte) (bool, error) { // Write new value copy(buf.Bytes(), new) - if _, err := r.dev.WriteAt(buf.Bytes(), offset); err != nil { + if _, err := r.dev.WriteAtUnsharded(buf.Bytes(), offset); err != nil { return false, err } @@ -629,7 +765,7 @@ func (r *Registry) readSlotRaw(slotIdx int) ([]byte, *Slot, error) { buf := storage.GetBuffer(SlotSize) defer storage.PutBuffer(buf) - if _, err := r.dev.ReadAt(buf.Bytes(), int64(r.offset)+int64(slotIdx)*SlotSize); err != nil { + if _, err := r.dev.ReadAtUnsharded(buf.Bytes(), int64(r.offset)+int64(slotIdx)*SlotSize); err != nil { return nil, nil, err } raw := bytes.Clone(buf.Bytes()) @@ -648,7 +784,7 @@ func (r *Registry) readSlot(slotIdx int) (*Slot, error) { buf := storage.GetBuffer(SlotSize) defer storage.PutBuffer(buf) - if _, err := r.dev.ReadAt(buf.Bytes(), offset); err != nil { + if _, err := r.dev.ReadAtUnsharded(buf.Bytes(), offset); err != nil { return nil, err } @@ -669,34 +805,49 @@ func (r *Registry) writeSlot(slotIdx int, slot *Slot) error { defer storage.PutBuffer(buf) copy(buf.Bytes(), data) - _, err := r.dev.WriteAt(buf.Bytes(), offset) + _, err := r.dev.WriteAtUnsharded(buf.Bytes(), offset) return err } -// isHeartbeatStale checks if a slot's heartbeat is beyond the timeout. -func (r *Registry) isHeartbeatStale(slot *Slot) bool { - age := time.Since(time.Unix(slot.HeartbeatTime, 0)) - return age > HeartbeatTimeout -} - -// GetActiveNodes returns information about all active nodes. +// GetActiveNodes returns the active nodes whose heartbeat this node has not seen stop. func (r *Registry) GetActiveNodes() ([]*Slot, error) { - r.mu.RLock() - defer r.mu.RUnlock() - + slots, err := r.readAllSlots() + if err != nil { + return nil, err + } + r.mu.Lock() + defer r.mu.Unlock() var active []*Slot - for i := 0; i < MaxNodes; i++ { - slot, err := r.readSlot(i) - if err != nil { + for i, slot := range slots { + if slot == nil || slot.State != NodeStateActive { continue } - if slot.State == NodeStateActive && !r.isHeartbeatStale(slot) { + r.observe(i, slot) + if !r.stale(i, slot, HeartbeatTimeout) { active = append(active, slot) } } return active, nil } +// readAllSlots reads the whole registry in one I/O, without a lock, so slow +// storage costs a scan one round trip and never holds up the heartbeat. A slot +// that fails to decode is nil. +func (r *Registry) readAllSlots() ([]*Slot, error) { + buf := storage.GetBuffer(MaxNodes * SlotSize) + defer storage.PutBuffer(buf) + if _, err := r.dev.ReadAtUnsharded(buf.Bytes()[:MaxNodes*SlotSize], int64(r.offset)); err != nil { + return nil, err + } + slots := make([]*Slot, MaxNodes) + for i := range slots { + if slot, err := deserializeSlot(buf.Bytes()[i*SlotSize : (i+1)*SlotSize]); err == nil { + slots[i] = slot + } + } + return slots, nil +} + // Slots returns every node slot, with nil for empty or unreadable ones. The // error names the unreadable ones. It holds no lock, so it never delays a heartbeat. func (r *Registry) Slots() ([]*Slot, error) { @@ -837,23 +988,21 @@ func (r *Registry) detectAndRecoverFailedNodes() { return // No recovery handler registered } - for i := 0; i < MaxNodes; i++ { - if i == localSlot { - continue // Skip our own slot + slots, err := r.readAllSlots() + if err != nil { + return + } + for i, slot := range slots { + if i == localSlot || slot == nil { + continue } r.mu.Lock() - slot, err := r.readSlot(i) - if err != nil { - r.mu.Unlock() - continue - } r.remember(i, slot) r.observe(i, slot) - // Stale heartbeats need 2x timeout for consensus, and none this node saw advance lately. // Peers wait for a failed node's locks until its replay ends, so only a node proven dead, not one paused, is replayed. - if !r.failed(slot, 2*HeartbeatTimeout) || (r.recentlyAlive(i) && !r.deadLocalOwner(slot)) || !r.provenDead(i, slot) { + if !r.stale(i, slot, deadAfter) { r.mu.Unlock() continue } @@ -867,7 +1016,7 @@ func (r *Registry) detectAndRecoverFailedNodes() { r.mu.Unlock() // Try to acquire the slot for recovery - info, err := r.acquireForRecovery(i, slot, 2*HeartbeatTimeout) + info, err := r.acquireForRecovery(i, slot, deadAfter) if err != nil { r.mu.Lock() delete(r.recoveryInProgress, i) @@ -878,8 +1027,8 @@ func (r *Registry) detectAndRecoverFailedNodes() { err = r.replay(handler, i, info) done(err == nil) if err != nil { - log.Printf("[Recovery] Replaying slot %d (node %x) failed: %v; any node retries after %v", - i, info.UUID[:8], err, recoveryLease) + log.Printf("[Recovery] Replaying slot %d (node %x) failed: %v; any node retries after %v. To discard its journal instead, mount a node with -skip-recovery=%d", + i, info.UUID[:8], err, recoveryLease, i) r.mu.Lock() delete(r.recoveryInProgress, i) r.mu.Unlock() @@ -907,7 +1056,8 @@ func (r *Registry) acquireForRecovery(slotIdx int, observed *Slot, timeout time. if err != nil { return nil, err } - if slot.UUID != observed.UUID || !r.failed(slot, timeout) { + r.observe(slotIdx, slot) + if slot.UUID != observed.UUID || !r.stale(slotIdx, slot, timeout) { return nil, ErrSlotTaken } @@ -929,13 +1079,14 @@ func (r *Registry) acquireForRecovery(slotIdx int, observed *Slot, timeout time. JournalSize: slot.JournalSize, LastHeartbeat: time.Unix(slot.HeartbeatTime, 0), Hostname: slot.Host(), - CheckLease: r.leaseCheck(slotIdx, slot.UUID), + CheckLease: r.leaseCheck(slotIdx, slot.UUID, time.Now()), }, nil } // leaseCheck returns a check that fails once this node no longer holds the -// recovery lease of the slot, and renews the lease after a quarter of its term. -func (r *Registry) leaseCheck(slotIdx int, uuid [16]byte) func() error { +// recovery lease of the slot, and renews the lease after a quarter of its term +// since renewed, on this node's clock. Peers see a renewal as a change. +func (r *Registry) leaseCheck(slotIdx int, uuid [16]byte, renewed time.Time) func() error { return func() error { r.mu.Lock() defer r.mu.Unlock() @@ -947,10 +1098,11 @@ func (r *Registry) leaseCheck(slotIdx int, uuid [16]byte) func() error { if slot.State != NodeStateDead || slot.UUID != uuid || slot.RecoveryOwner != r.localUUID { return ErrLeaseLost } - if time.Since(time.Unix(slot.RecoveryStart, 0)) < recoveryLease/4 { + if time.Since(renewed) < recoveryLease/4 { return nil } - slot.RecoveryStart = time.Now().Unix() + now := time.Now() + slot.RecoveryStart = max(now.Unix(), slot.RecoveryStart+1) if success, err := r.casSlot(slotIdx, raw, serializeSlot(slot)); err != nil || !success { return ErrLeaseLost } @@ -958,6 +1110,7 @@ func (r *Registry) leaseCheck(slotIdx int, uuid [16]byte) func() error { if current, err := r.readSlot(slotIdx); err != nil || current.RecoveryOwner != r.localUUID { return ErrLeaseLost } + renewed = now return nil } } @@ -1031,44 +1184,29 @@ func (r *Registry) ReleaseSlot(slotIdx int, expected []byte, reset func() error) // GetFailedNodes returns information about nodes that have failed (stale heartbeat). // Useful for manual inspection or triggering recovery outside the normal flow. func (r *Registry) GetFailedNodes() ([]*FailedNodeInfo, error) { - r.mu.RLock() - defer r.mu.RUnlock() - + slots, err := r.readAllSlots() + if err != nil { + return nil, err + } + r.mu.Lock() + defer r.mu.Unlock() var failed []*FailedNodeInfo - for i := 0; i < MaxNodes; i++ { - slot, err := r.readSlot(i) - if err != nil { + for i, slot := range slots { + if slot == nil || slot.State == NodeStateEmpty { continue } - - isFailed := false - switch slot.State { - case NodeStateActive, NodeStateLeaving: - if r.isHeartbeatStale(slot) { - isFailed = true - } - case NodeStateDead: - isFailed = true - } - - if isFailed { - hostname := string(slot.Hostname[:]) - for j, b := range slot.Hostname { - if b == 0 { - hostname = string(slot.Hostname[:j]) - break - } - } - - failed = append(failed, &FailedNodeInfo{ - SlotIndex: i, - UUID: slot.UUID, - JournalOffset: slot.JournalOffset, - JournalSize: slot.JournalSize, - LastHeartbeat: time.Unix(slot.HeartbeatTime, 0), - Hostname: hostname, - }) + r.observe(i, slot) + if slot.State != NodeStateDead && !r.stale(i, slot, HeartbeatTimeout) { + continue } + failed = append(failed, &FailedNodeInfo{ + SlotIndex: i, + UUID: slot.UUID, + JournalOffset: slot.JournalOffset, + JournalSize: slot.JournalSize, + LastHeartbeat: time.Unix(slot.HeartbeatTime, 0), + Hostname: slot.Host(), + }) } return failed, nil } @@ -1107,7 +1245,8 @@ func (r *Registry) RecoverFailedNode(slotIdx int) error { return err } - if !r.failed(slot, HeartbeatTimeout) { + r.observe(slotIdx, slot) + if !r.stale(slotIdx, slot, HeartbeatTimeout) { r.mu.Unlock() return errors.New("node is not in a failed state") } diff --git a/internal/node/registry_fuzz_test.go b/internal/node/registry_fuzz_test.go index eb85e1f2..0d99d013 100644 --- a/internal/node/registry_fuzz_test.go +++ b/internal/node/registry_fuzz_test.go @@ -58,20 +58,25 @@ func checkSlot(t *testing.T, buf []byte, slot *Slot) { r := NewRegistry(nil, 0) _, _ = slot.Host(), slot.State.String() - failed := r.failed(slot, HeartbeatTimeout) + r.observe(0, slot) + if r.stale(0, slot, HeartbeatTimeout) && !r.deadLocalOwner(slot) { + t.Fatalf("state %s: a slot seen for the first time counted as stale", slot.State) + } expiry := slot.LeaseExpiry() switch slot.State { - case NodeStateActive, NodeStateLeaving, NodeStateDead: - if r.deadLocalOwner(slot) { - return + case NodeStateActive, NodeStateLeaving: + old := heartbeatOlderThan(slot, HeartbeatTimeout) + if gap := time.Until(expiry); (gap > time.Second && old) || (gap < -time.Second && !old) { + t.Fatalf("state %s: heartbeat old is %v but the lease ends %s from now (heartbeat %d)", + slot.State, old, gap, slot.HeartbeatTime) } - if gap := time.Until(expiry); (gap > time.Second && failed) || (gap < -time.Second && !failed) { - t.Fatalf("state %s: failed is %v but the lease ends %s from now (heartbeat %d, recovery start %d)", - slot.State, failed, gap, slot.HeartbeatTime, slot.RecoveryStart) + case NodeStateDead: + if want := time.Unix(slot.RecoveryStart, 0).Add(recoveryLease); !expiry.Equal(want) { + t.Fatalf("dead slot: lease ends %v, want recovery start %d plus the lease", expiry, slot.RecoveryStart) } default: - if failed || !expiry.IsZero() { - t.Fatalf("state %s: failed %v, lease ends %v", slot.State, failed, expiry) + if !expiry.IsZero() { + t.Fatalf("state %s: lease ends %v", slot.State, expiry) } } } diff --git a/internal/node/registry_test.go b/internal/node/registry_test.go index 64860c84..2f489c96 100644 --- a/internal/node/registry_test.go +++ b/internal/node/registry_test.go @@ -321,29 +321,6 @@ func TestRegistryGetActiveNodes(t *testing.T) { } } -func TestIsHeartbeatStale(t *testing.T) { - dev, cleanup := createTestDevice(t, 10<<20) - defer cleanup() - - reg := NewRegistry(dev, 0) - - // Fresh heartbeat - not stale - freshSlot := &Slot{ - HeartbeatTime: time.Now().Unix(), - } - if reg.isHeartbeatStale(freshSlot) { - t.Error("fresh heartbeat should not be stale") - } - - // Old heartbeat - stale - staleSlot := &Slot{ - HeartbeatTime: time.Now().Add(-HeartbeatTimeout - time.Second).Unix(), - } - if !reg.isHeartbeatStale(staleSlot) { - t.Error("old heartbeat should be stale") - } -} - func BenchmarkSlotSerialize(b *testing.B) { slot := &Slot{ Magic: SlotMagic, @@ -447,6 +424,7 @@ func TestGetFailedNodesWithStaleHeartbeat(t *testing.T) { // Create a second registry to detect the failure reg2 := NewRegistry(dev, 0) + silence(reg2, slot, staleSlot, HeartbeatTimeout+time.Second) failed, err := reg2.GetFailedNodes() if err != nil { @@ -520,6 +498,8 @@ func TestRecoverFailedNode(t *testing.T) { return nil }) + silence(reg2, failingSlot, staleSlot, deadAfter+time.Second) + // Trigger recovery on the failing node's slot err = reg2.RecoverFailedNode(failingSlot) if err != nil { @@ -631,10 +611,7 @@ func TestFailureDetectionLoop(t *testing.T) { if err != nil { t.Fatalf("writeSlot failed: %v", err) } - reg2.mu.Lock() - reg2.sightings[failingSlot] = sighting{uuid: staleSlot.UUID, seq: staleSlot.HeartbeatSeq, beat: staleSlot.HeartbeatTime, - since: time.Now().Add(-HeartbeatTimeout - deathMargin)} - reg2.mu.Unlock() + silence(reg2, failingSlot, staleSlot, deadAfter+time.Second) // Set up recovery handler recovered := make(chan *FailedNodeInfo, 1) @@ -720,8 +697,8 @@ func TestRegisterSkipsSlotUnderRecovery(t *testing.T) { if err != nil { t.Fatalf("Register: %v", err) } - if slot != 1 { - t.Errorf("Register took slot %d, want slot 1, whose recovery lease lapsed", slot) + if slot != 2 { + t.Errorf("Register took slot %d, want empty slot 2: a slot under recovery is failure detection's to judge, by watching its lease", slot) } } @@ -891,10 +868,7 @@ func TestRecoveryLeaseBlocksOthersUntilItLapses(t *testing.T) { t.Fatalf("acquire during another node's lease: err = %v, want ErrSlotTaken", err) } - failed.RecoveryStart = time.Now().Add(-2 * recoveryLease).Unix() - if err := reg.writeSlot(4, failed); err != nil { - t.Fatal(err) - } + silence(reg, 4, failed, recoveryLease+time.Second) if _, err := reg.acquireForRecovery(4, failed, HeartbeatTimeout); err != nil { t.Fatalf("acquire after the lease lapsed: %v", err) } @@ -1015,6 +989,7 @@ func TestLeaseCheckRenewsAndDetectsLoss(t *testing.T) { if err := reg.writeSlot(4, failed); err != nil { t.Fatal(err) } + silence(reg, 4, failed, HeartbeatTimeout+time.Second) info, err := reg.acquireForRecovery(4, failed, HeartbeatTimeout) if err != nil { t.Fatal(err) @@ -1023,17 +998,19 @@ func TestLeaseCheckRenewsAndDetectsLoss(t *testing.T) { t.Fatalf("CheckLease right after acquire: %v", err) } + // A replayer whose clock is behind still renews its lease to a value peers see change. held, _ := reg.readSlot(4) - held.RecoveryStart = time.Now().Add(-recoveryLease / 2).Unix() + held.RecoveryStart = time.Now().Add(time.Hour).Unix() if err := reg.writeSlot(4, held); err != nil { t.Fatal(err) } - if err := info.CheckLease(); err != nil { + if err := reg.leaseCheck(4, failed.UUID, time.Now().Add(-recoveryLease/2))(); err != nil { t.Fatalf("CheckLease halfway through the lease: %v", err) } - if renewed, _ := reg.readSlot(4); time.Since(time.Unix(renewed.RecoveryStart, 0)) > 5*time.Second { - t.Errorf("lease not renewed: started %v", time.Unix(renewed.RecoveryStart, 0)) + if renewed, _ := reg.readSlot(4); renewed.RecoveryStart <= held.RecoveryStart { + t.Errorf("lease not renewed: start %d, was %d", renewed.RecoveryStart, held.RecoveryStart) } + held, _ = reg.readSlot(4) held.RecoveryOwner = [16]byte{6} if err := reg.writeSlot(4, held); err != nil { @@ -1178,26 +1155,33 @@ func TestHeartbeatStopsWhenSlotIsMarkedDead(t *testing.T) { } } -func TestOwnClockDecidesDeathNotWallClock(t *testing.T) { - reg := &Registry{sightings: make(map[int]sighting)} - slot := &Slot{UUID: [16]byte{4}, State: NodeStateActive, HeartbeatSeq: 1, - HeartbeatTime: time.Now().Add(-time.Hour).Unix()} - - reg.observe(1, slot) - if reg.provenDead(1, slot) { - t.Error("a heartbeat first seen just now counted as dead because its wall-clock time is old") - } +// silence makes reg treat slot as read unchanged for d, as if it had watched the slot that long. +func silence(reg *Registry, idx int, slot *Slot, d time.Duration) { + reg.mu.Lock() + defer reg.mu.Unlock() + seen := sightingOf(slot) + seen.since = time.Now().Add(-d) + reg.sightings[idx] = seen +} - reg.sightings[1] = sighting{uuid: slot.UUID, seq: 1, beat: slot.HeartbeatTime, - since: time.Now().Add(-(HeartbeatTimeout + deathMargin + time.Second))} - if !reg.provenDead(1, slot) { - t.Error("a heartbeat unchanged past timeout plus margin by this node's clock did not count as dead") - } +func TestOwnClockDecidesDeathNotWallClock(t *testing.T) { + for _, beat := range []time.Duration{-time.Hour, 3 * time.Minute} { + reg := &Registry{sightings: make(map[int]sighting)} + slot := &Slot{UUID: [16]byte{4}, State: NodeStateActive, HeartbeatSeq: 1, HeartbeatTime: time.Now().Add(beat).Unix()} - slot.HeartbeatSeq++ - reg.observe(1, slot) - if reg.provenDead(1, slot) || !reg.recentlyAlive(1) { - t.Error("a heartbeat that advanced still counted as dead") + reg.observe(1, slot) + if reg.stale(1, slot, deadAfter) { + t.Errorf("heartbeat %v from now: first seen just now, it counted as dead", beat) + } + silence(reg, 1, slot, deadAfter+time.Second) + if !reg.stale(1, slot, deadAfter) { + t.Errorf("heartbeat %v from now: unchanged past deadAfter by this node's clock, it did not count as dead", beat) + } + slot.HeartbeatSeq++ + reg.observe(1, slot) + if reg.stale(1, slot, deadAfter) { + t.Errorf("heartbeat %v from now: a heartbeat that advanced still counted as dead", beat) + } } } @@ -1253,10 +1237,14 @@ func TestOccupantReportsHowSlotStands(t *testing.T) { {Magic: SlotMagic, State: NodeStateEmpty}, } want := []Occupancy{OccupiedLive, OccupiedFailing, OccupiedReplaying, OccupiedLapsed, Unoccupied} + unchanged := map[int]time.Duration{1: HeartbeatTimeout + time.Second, 3: recoveryLease + time.Second} for i, slot := range slots { if err := reg.writeSlot(i, slot); err != nil { t.Fatal(err) } + if d, ok := unchanged[i]; ok { + silence(reg, i, slot, d) + } if uuid, got := reg.Occupant(i, true); got != want[i] || uuid != slot.UUID { t.Errorf("slot %d: Occupant = %x, %d; want %x, %d", i, uuid[:1], got, slot.UUID[:1], want[i]) } @@ -1281,28 +1269,163 @@ func TestOccupantAnswersFromLastReadUnlessFresh(t *testing.T) { } } -func TestRegisterReportsPreviousOwnerOfAnyTakenSlot(t *testing.T) { - dev, cleanup := createTestDevice(t, 10<<20) - defer cleanup() - reg := NewRegistry(dev, 0) - gone := &Slot{Magic: SlotMagic, State: NodeStateActive, UUID: [16]byte{7}, HeartbeatTime: time.Now().Add(-3 * HeartbeatTimeout).Unix()} +// fillWithReplays marks every slot from first on as dead and under another node's recovery, so none is empty. +func fillWithReplays(t *testing.T, reg *Registry, first int) { + t.Helper() + for i := first; i < MaxNodes; i++ { + dead := &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: [16]byte{byte(100 + i)}, RecoveryOwner: [16]byte{7}, RecoveryStart: time.Now().Unix()} + if err := reg.writeSlot(i, dead); err != nil { + t.Fatal(err) + } + } +} + +func shortenSilentSlotWatch(t *testing.T) { + t.Helper() + old := silentSlotWatch + t.Cleanup(func() { silentSlotWatch = old }) + silentSlotWatch = 2 * time.Second +} + +// writeGoneSlot writes slot 0 as another host's node whose heartbeat stopped long ago by this clock. +func writeGoneSlot(t *testing.T, reg *Registry) *Slot { + t.Helper() + gone := &Slot{Magic: SlotMagic, State: NodeStateActive, UUID: [16]byte{7}, HeartbeatSeq: 3, HeartbeatTime: time.Now().Add(-3 * HeartbeatTimeout).Unix()} copy(gone.Hostname[:], "elsewhere") if err := reg.writeSlot(0, gone); err != nil { t.Fatal(err) } - if slot, err := reg.Register(context.Background(), "node"); err != nil || slot != 0 { - t.Fatalf("Register = %d, %v; want the failed slot 0", slot, err) + return gone +} + +func TestRegisterTakesEmptySlotBeforeAnotherHostsSilentOne(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + writeGoneSlot(t, reg) + if slot, err := reg.Register(context.Background(), "node"); err != nil || slot != 1 { + t.Fatalf("Register = %d, %v; want the empty slot 1", slot, err) } - if owner, ok := reg.PreviousOwner(); !ok || owner != gone.UUID { - t.Errorf("PreviousOwner = %x, %v; want the failed node %x of another host", owner, ok, gone.UUID) + if owner, ok := reg.PreviousOwner(); ok { + t.Errorf("PreviousOwner = %x after taking an empty slot, want none", owner) } +} - fresh := NewRegistry(dev, 0) - if _, err := fresh.Register(context.Background(), "node2"); err != nil { - t.Fatal(err) +// A host that rebooted after a crash takes back the slot it held under the same +// hostname once the slot proves silent, so its mount replays that journal. A +// live host that shares the name beats during the watch and keeps its slot. +func TestRegisterTakesBackSlotOfItsEarlierBoot(t *testing.T) { + for _, beating := range []bool{false, true} { + t.Run(map[bool]string{false: "silent", true: "another host with the name"}[beating], func(t *testing.T) { + shortenSilentSlotWatch(t) + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + earlier := &Slot{Magic: SlotMagic, State: NodeStateActive, UUID: [16]byte{7}, HeartbeatSeq: 3, + HeartbeatTime: time.Now().Add(-3 * HeartbeatTimeout).Unix(), BootID: [16]byte{0xEE}} + copy(earlier.Hostname[:], "node") + if err := reg.writeSlot(0, earlier); err != nil { + t.Fatal(err) + } + if beating { + defer keepWriting(t, reg, 0, earlier, 3*HeartbeatTimeout)() + } + slot, err := reg.Register(context.Background(), "node") + if err != nil { + t.Fatal(err) + } + owner, ok := reg.PreviousOwner() + switch { + case beating && slot == 0: + t.Error("Register took the slot of a live host that shares this hostname") + case !beating && (slot != 0 || !ok || owner != earlier.UUID): + t.Errorf("Register = %d, previous owner %x %v; want slot 0 of this hostname's earlier boot, node %x", slot, owner, ok, earlier.UUID) + } + }) } - if owner, ok := fresh.PreviousOwner(); ok { - t.Errorf("PreviousOwner = %x after taking an empty slot, want none", owner) +} + +func TestRegisterTakesAnotherHostsSilentSlotWhenNamedOrNoneIsEmpty(t *testing.T) { + for _, named := range []bool{true, false} { + t.Run(map[bool]string{true: "named", false: "none empty"}[named], func(t *testing.T) { + shortenSilentSlotWatch(t) + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + gone := writeGoneSlot(t, reg) + if named { + reg.PreferSlot(0) + } else { + fillWithReplays(t, reg, 1) + } + if slot, err := reg.Register(context.Background(), "node"); err != nil || slot != 0 { + t.Fatalf("Register = %d, %v; want slot 0, whose heartbeat never changed", slot, err) + } + if owner, ok := reg.PreviousOwner(); !ok || owner != gone.UUID { + t.Errorf("PreviousOwner = %x, %v; want the failed node %x of another host", owner, ok, gone.UUID) + } + }) + } +} + +// A named slot under another node's recovery is taken once its lease lapses +// unrenewed, and left alone while the lease is renewed. +func TestRegisterTakesNamedSlotUnderRecoveryOnlyOnceItsLeaseLapses(t *testing.T) { + for _, renewed := range []bool{false, true} { + t.Run(map[bool]string{false: "lapsed", true: "renewed"}[renewed], func(t *testing.T) { + t.Cleanup(SetRecoveryLease(2 * time.Second)) + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + dead := &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: [16]byte{5}, RecoveryOwner: [16]byte{6}, RecoveryStart: time.Now().Unix()} + if err := reg.writeSlot(0, dead); err != nil { + t.Fatal(err) + } + if renewed { + defer keepWriting(t, reg, 0, dead, 0)() + } + reg.PreferSlot(0) + slot, err := reg.Register(context.Background(), "node") + if err != nil { + t.Fatal(err) + } + if took := slot == 0; took == renewed { + t.Errorf("Register took slot %d; want slot 0 only when its lease was not renewed", slot) + } + }) + } +} + +// A slot that changes after Register watched it, because a peer marked it dead +// to replay it or its node beat again, is not taken. +func TestRegisterLeavesWatchedSlotThatChanges(t *testing.T) { + for _, tc := range []struct { + name string + change func(*Slot) + }{ + {"marked dead by a peer", func(s *Slot) { + s.State, s.RecoveryOwner, s.RecoveryStart = NodeStateDead, [16]byte{8}, time.Now().Unix() + }}, + {"its node beat again", func(s *Slot) { s.HeartbeatSeq++ }}, + } { + t.Run(tc.name, func(t *testing.T) { + shortenSilentSlotWatch(t) + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + gone := writeGoneSlot(t, reg) + reg.PreferSlot(0) + t.Cleanup(func() { afterSilentWatch = nil }) + afterSilentWatch = func() { + tc.change(gone) + if err := reg.writeSlot(0, gone); err != nil { + t.Error(err) + } + } + if slot, err := reg.Register(context.Background(), "node"); err != nil || slot == 0 { + t.Fatalf("Register = %d, %v; want an empty slot, not slot 0, which changed after the watch", slot, err) + } + }) } } diff --git a/internal/storage/device.go b/internal/storage/device.go index 3a2d8647..ee02bf6b 100644 --- a/internal/storage/device.go +++ b/internal/storage/device.go @@ -48,6 +48,8 @@ type Device struct { closeMu sync.RWMutex // Protects close; held shared during I/O // writeGate, when set, runs before every write and can refuse it. writeGate atomic.Pointer[func(offset int64) error] + // readFault, when a test sets it, runs before every read and can fail it. + readFault atomic.Pointer[func(offset int64) error] file *os.File path string size int64 @@ -192,6 +194,11 @@ func (d *Device) readAt(buf []byte, offset int64, sharded bool) (int, error) { if len(buf) == 0 { return 0, nil } + if fault := d.readFault.Load(); fault != nil { + if err := (*fault)(offset); err != nil { + return 0, err + } + } // Acquire shared close lock to prevent device close during I/O d.closeMu.RLock() @@ -251,6 +258,11 @@ func (d *Device) SetWriteGate(gate func(offset int64) error) { d.writeGate.Store(&gate) } +// SetReadFault makes fault run before every read; a read it returns an error for fails with that error. +func (d *Device) SetReadFault(fault func(offset int64) error) { + d.readFault.Store(&fault) +} + // CheckWrite runs the write gate for a write at offset. Callers that change the // device without WriteAt, such as a discard, call it first. func (d *Device) CheckWrite(offset int64) error { diff --git a/tests/integration/helpers.go b/tests/integration/helpers.go index 00fae675..45573f0a 100644 --- a/tests/integration/helpers.go +++ b/tests/integration/helpers.go @@ -263,7 +263,7 @@ func (m *MountedFS) Mount(debug bool) error { } // Wait for mount to become accessible - if err := m.WaitForMount(10 * time.Second); err != nil { + if err := m.WaitForMount(mountWait); err != nil { _ = m.Unmount() return err } @@ -296,7 +296,7 @@ func (m *MountedFS) FormatAndMount(debug bool) error { } // Wait for mount to become accessible - if err := m.WaitForMount(10 * time.Second); err != nil { + if err := m.WaitForMount(mountWait); err != nil { _ = m.Unmount() return err } @@ -307,6 +307,23 @@ func (m *MountedFS) FormatAndMount(debug bool) error { // WaitForMount waits until the filesystem is mounted and accessible. // It checks /proc/mounts to verify the FUSE mount is active. +// processExited reports whether p has exited, without waiting for it: an exited child stays a zombie until reaped. +func processExited(p *os.Process) bool { + if p == nil { + return false + } + stat, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", p.Pid)) + if err != nil { + return true + } + _, after, ok := strings.Cut(string(stat), ") ") + return ok && strings.HasPrefix(after, "Z") +} + +// mountWait is how long a mount may take: one whose -skip-recovery names +// another host's slot watches it for about 12 seconds before taking it. +const mountWait = 30 * time.Second + func (m *MountedFS) WaitForMount(timeout time.Duration) error { deadline := time.Now().Add(timeout) @@ -321,9 +338,8 @@ func (m *MountedFS) WaitForMount(timeout time.Duration) error { } } - // Check if mount command exited unexpectedly - if m.MountCmd != nil && m.MountCmd.ProcessState != nil { - return fmt.Errorf("mount process exited unexpectedly: %v", m.MountCmd.ProcessState) + if m.MountCmd != nil && (m.MountCmd.ProcessState != nil || processExited(m.MountCmd.Process)) { + return fmt.Errorf("mount process exited unexpectedly") } time.Sleep(100 * time.Millisecond) @@ -604,7 +620,7 @@ func (m *MountedFS) FormatAndMountWithSFB(debug bool) error { } // Wait for mount to become accessible - if err := m.WaitForMount(10 * time.Second); err != nil { + if err := m.WaitForMount(mountWait); err != nil { _ = m.Unmount() return err } @@ -638,7 +654,7 @@ func (m *MountedFS) MountWithSFB(debug bool) error { } // Wait for mount to become accessible - if err := m.WaitForMount(10 * time.Second); err != nil { + if err := m.WaitForMount(mountWait); err != nil { _ = m.Unmount() return err } diff --git a/tests/integration/journal_slots_test.go b/tests/integration/journal_slots_test.go index a2e5e53a..5a3c2a92 100644 --- a/tests/integration/journal_slots_test.go +++ b/tests/integration/journal_slots_test.go @@ -216,9 +216,9 @@ func TestMountRefusesDamagedJournal(t *testing.T) { } // TestSkipRecoveryDiscardsOnlyNamedSlot damages the journal of another host's -// crashed node, whose slot the mount then takes. -skip-recovery naming a -// different slot must not discard it, also on a retry, when the slot already -// carries this host's name from the failed mount. Naming the slot does. +// crashed node. A mount takes an empty slot instead, and -skip-recovery naming +// a different slot leaves the damaged journal alone. Naming the crashed slot +// makes the mount take it, once its heartbeat has not changed, and discard it. func TestSkipRecoveryDiscardsOnlyNamedSlot(t *testing.T) { cluster := NewMultiNodeCluster(t, 1, 50) mfs := cluster.Node(0) @@ -240,25 +240,20 @@ func TestSkipRecoveryDiscardsOnlyNamedSlot(t *testing.T) { if slot == other { other = "2" } - for attempt := 1; attempt <= 2; attempt++ { - out := &syncBuffer{} - mfs.Log = out - mfs.ExtraArgs = []string{"-skip-recovery=" + other} - if err := cluster.StartNode(0, false); err == nil { - t.Fatalf("attempt %d: -skip-recovery=%s mounted over slot %s's damaged journal", attempt, other, slot) - } - for _, want := range []string{ - "ignoring -skip-recovery=" + other + ": this mount took slot " + slot, - "Journal recovery for slot " + slot + " failed", - "mount with -skip-recovery=" + slot, - } { - if !strings.Contains(out.String(), want) { - t.Errorf("attempt %d: mount output lacks %q:\n%s", attempt, want, out.String()) - } - } - if !bytes.Equal(readImageSector(t, cluster.ImagePath, offset), damaged) { - t.Fatalf("attempt %d: slot %s's journal changed", attempt, slot) - } + out := &syncBuffer{} + mfs.Log = out + mfs.ExtraArgs = []string{"-skip-recovery=" + other} + if err := cluster.StartNode(0, false); err != nil { + t.Fatalf("mount with -skip-recovery=%s: %v", other, err) + } + if got := mustMatch(t, slotRe, out, "registration"); got == slot { + t.Errorf("a mount took slot %s of another host's crashed node, which -skip-recovery did not name", slot) + } + if !bytes.Equal(readImageSector(t, cluster.ImagePath, offset), damaged) { + t.Fatalf("slot %s's journal changed although -skip-recovery named slot %s", slot, other) + } + if err := cluster.StopNode(0); err != nil { + t.Fatalf("stop: %v", err) } named := &syncBuffer{} @@ -267,6 +262,9 @@ func TestSkipRecoveryDiscardsOnlyNamedSlot(t *testing.T) { if err := cluster.StartNode(0, false); err != nil { t.Fatalf("mount with -skip-recovery=%s: %v", slot, err) } + if got := mustMatch(t, slotRe, named, "registration"); got != slot { + t.Errorf("-skip-recovery=%s: the mount took slot %s", slot, got) + } if got := mustMatch(t, journalInitRe, named, "journal init"); got != offset { t.Errorf("-skip-recovery=%s started a journal at offset %s, want %s", slot, got, offset) } diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index 648daf40..bb321cc6 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -723,6 +723,68 @@ scenarios: docker: TODO # needs SIGSTOP and SIGCONT on one container's mount process with a writer through it; covered in Go and VM vm: run-vm-tests.sh::test_stalled_node_writes_stop_at_write_gate + - id: hostile-storage-latency-spikes + name: Storage latency spikes on one node + description: One node's path to the shared disk takes 100ms, 500ms, 1s and 2s per I/O for 15s each while both nodes create files; no node is declared dead, every acknowledged file reads back, and fsck finds nothing but leaks + priority: vm_only # needs a device-mapper device under one node's mount + environments: + go: internal/node/clock_skew_test.go::TestSlowRegistryReadsDoNotStarveHeartbeat + vm: run-vm-tests.sh::test_storage_latency_spikes + + - id: clock-ahead-mount-keeps-live-slot + name: A node with a fast clock does not take a live node's slot + description: A node that never mounted the volume mounts with its clock 2 minutes ahead; it takes a free slot, and the mounted node keeps running + priority: high + environments: + go: internal/node/clock_skew_test.go::TestRegisterLeavesLiveSlotOfNodeWithSlowClock + docker: TODO # containers share the host clock; covered in Go and VM + vm: run-vm-tests.sh::test_clock_ahead_mount + + - id: clock-ahead-crash-replayed + name: A crashed node with a fast clock is replayed on time + description: A node whose clock is 3 minutes ahead crashes; its peer replays it within the usual failover time, judging silence on its own clock + priority: high + environments: + go: internal/node/clock_skew_test.go::TestCrashedNodeWithFastClockIsReplayed + docker: TODO # containers share the host clock; covered in Go and VM + vm: run-vm-tests.sh::test_clock_ahead_crash_recovered + + - id: register-takes-empty-slot-first + name: Register takes an empty slot before another host's silent one + description: A mount takes an empty slot without waiting, and leaves another host's slot whose heartbeat looks old to failure detection unless -skip-recovery names it or no slot is empty + priority: go_only + environments: + go: internal/node/registry_test.go::TestRegisterTakesEmptySlotBeforeAnotherHostsSilentOne + + - id: register-takes-named-slot-under-recovery-once-lapsed + name: A named slot under recovery is taken once its lease lapses + description: -skip-recovery naming a slot that another node marked dead takes it only after its recovery lease stays unrenewed for the lease term + priority: go_only + environments: + go: internal/node/registry_test.go::TestRegisterTakesNamedSlotUnderRecoveryOnlyOnceItsLeaseLapses + + - id: register-takes-back-earlier-boot-slot + name: A rebooted host takes back its slot + description: A mount takes back the slot its hostname held in an earlier kernel boot, once the slot's heartbeat proves silent, before any empty slot, so it replays that journal before serving + priority: go_only + environments: + go: internal/node/registry_test.go::TestRegisterTakesBackSlotOfItsEarlierBoot + vm: run-vm-tests.sh::test_fsck_repair_after_power_failure + + - id: register-leaves-watched-slot-that-changes + name: Register leaves a watched slot that changed + description: A slot Register watched stay silent is not taken if a peer marks it dead or its node beats again before the claim + priority: go_only + environments: + go: internal/node/registry_test.go::TestRegisterLeavesWatchedSlotThatChanges + + - id: renewed-lease-held-whatever-clock + name: A renewed recovery lease is held whatever the replayer's clock + description: A lease unchanged past recoveryLease counts as lapsed; once the replayer renews it, peers count it as held although its start time is 2 minutes in the past + priority: go_only + environments: + go: internal/node/clock_skew_test.go::TestRenewedRecoveryLeaseIsHeldWhateverTheReplayersClock + - id: hostile-storage-short-stalls name: Short storage stalls on one node description: One node's path to the shared disk stalls for 5s and then 12s (dm suspend) while both nodes create files; no node is declared dead, every acknowledged file reads back, and fsck finds nothing but leaks @@ -3824,7 +3886,7 @@ scenarios: description: Register reports the node whose failed slot it took, from any host, so mount keeps that node's lock claims through the replay priority: go_only environments: - go: internal/node/registry_test.go::TestRegisterReportsPreviousOwnerOfAnyTakenSlot + go: internal/node/registry_test.go::TestRegisterTakesAnotherHostsSilentSlotWhenNamedOrNoneIsEmpty - id: storage-write-gate-refuses-writes name: Device write gate refuses writes diff --git a/tests/vm/scripts/run-vm-tests.sh b/tests/vm/scripts/run-vm-tests.sh index ba9a5478..08cc90ed 100755 --- a/tests/vm/scripts/run-vm-tests.sh +++ b/tests/vm/scripts/run-vm-tests.sh @@ -3192,7 +3192,7 @@ test_stalled_node_writes_stop_at_write_gate() { } # ============================================================================= -# Hostile storage +# Hostile storage and clocks # ============================================================================= # VM 1 mounts through a device-mapper device over the shared disk, so its path # alone can be slowed, stalled or failed. Both nodes create small files and @@ -3314,6 +3314,16 @@ hostile_scenario() { return $result } +hostile_delay_spikes() { + local ms + for ms in 100 500 1000 2000; do + log_info "VM 1 path: ${ms}ms per I/O for 15s" + hostile_dm_load "delay /dev/vda 0 $ms" || return 1 + sleep 15 + done + hostile_dm_load "linear /dev/vda 0" +} + hostile_stall() { log_info "VM 1 path: stalled for ${HOSTILE_STALL}s" ssh_run 1 "dmsetup suspend $HOSTILE_DM" @@ -3335,6 +3345,7 @@ hostile_flaky() { hostile_dm_load "linear /dev/vda 0" } +test_storage_latency_spikes() { hostile_scenario "latency spikes" hostile_delay_spikes survives; } test_storage_short_stalls() { HOSTILE_STALL=5 hostile_scenario "5s stall" hostile_stall survives || return 1 HOSTILE_STALL=12 hostile_scenario "12s stall" hostile_stall survives @@ -3344,6 +3355,63 @@ test_storage_short_eio() { HOSTILE_EIO=2 hostile_scenario "2s of EIO" hostile_ei test_storage_long_eio() { HOSTILE_EIO=10 hostile_scenario "10s of EIO" hostile_eio fenced; } test_storage_flaky_eio() { hostile_scenario "flaky EIO" hostile_flaky survives; } +# A node that never mounted the volume, with its clock 2 minutes ahead, must +# take a free slot, not the slot of a node that is mounted and alive. +test_clock_ahead_mount() { + ensure_cluster_healthy 2 || return 1 + unmount_goclusterfs 2 || { log_error "VM 2: unmount failed"; return 1; } + local marker="hostile: clock ahead $(date +%s)" result=0 + mark_vm_logs "$marker" + ssh_run 2 "hostname newnode; date -u -s '$(date -u -d @$(( $(date +%s) + 120 )) '+%Y-%m-%d %H:%M:%S')' > /dev/null" + setup_goclusterfs 2 false || result=1 + sleep 5 + local slot1 slot2 + slot1=$(grep -o 'Registered as node [0-9]*' "$(get_vm_mount_log 1)" | tail -1) + slot2=$(sed -n "/$marker/,\$p" "$(get_vm_mount_log 2)" | grep -o 'Registered as node [0-9]*' | tail -1) + log_info "VM 1: $slot1; VM 2 with its clock 2 minutes ahead: $slot2" + if [[ -z "$(mount_pids 1)" ]]; then + log_error "VM 1 stopped after VM 2 mounted with a fast clock" + sed -n "/$marker/,\$p" "$(get_vm_mount_log 1)" | grep -E 'ERROR|slot' | head -5 + result=1 + fi + if [[ "$slot1" == "$slot2" ]]; then + log_error "VM 2 took VM 1's registry slot" + result=1 + fi + ssh_run 2 "hostname localhost; date -u -s '$(date -u '+%Y-%m-%d %H:%M:%S')' > /dev/null" + if [[ -z "$(mount_pids 1)" ]]; then + ssh_run 1 "fusermount3 -uz ${GOCLUSTERFS_MOUNT}" || true + setup_goclusterfs 1 false || result=1 + fi + unmount_goclusterfs 2 || crash_goclusterfs 2 || true + setup_goclusterfs 2 false || result=1 + return $result +} + +# A node whose clock jumped 3 minutes ahead and then crashed must still be +# replayed within the usual failover time. +test_clock_ahead_crash_recovered() { + ensure_cluster_healthy 2 || return 1 + local marker="hostile: clock ahead crash $(date +%s)" result=0 waited=0 + mark_vm_logs "$marker" + ssh_run 1 "date -u -s '$(date -u -d @$(( $(date +%s) + 180 )) '+%Y-%m-%d %H:%M:%S')' > /dev/null" + sleep 3 + crash_goclusterfs 1 || result=1 + while ! hostile_replayed_since "$marker" && [[ $waited -lt 60 ]]; do + sleep 1 + waited=$((waited + 1)) + done + if ! hostile_replayed_since "$marker"; then + log_error "VM 2 did not replay crashed VM 1 within 60s; VM 1's clock was 3 minutes ahead" + result=1 + else + log_info "VM 2 replayed VM 1 ${waited}s after the crash" + fi + ssh_run 1 "date -u -s '$(date -u '+%Y-%m-%d %H:%M:%S')' > /dev/null; fusermount3 -uz ${GOCLUSTERFS_MOUNT}" || true + setup_goclusterfs 1 false || result=1 + return $result +} + # A power failure while files are written may leave blocks that no file maps. # With every node unmounted, fsck -repair frees any it finds, the check after # it is clean, and no file changes. @@ -4173,11 +4241,14 @@ main() { run_test "Snapshot Create/Delete Power Failure" test_snapshot_create_delete_power_failure run_test "TRIM Reuse Power Failure" test_trim_reuse_power_failure run_test "Stalled Node Writes Stop At Write Gate" test_stalled_node_writes_stop_at_write_gate + run_test "Hostile: Storage Latency Spikes" test_storage_latency_spikes run_test "Hostile: Short Storage Stalls" test_storage_short_stalls run_test "Hostile: Long Storage Stall" test_storage_long_stall run_test "Hostile: Short EIO Burst" test_storage_short_eio run_test "Hostile: Long EIO Burst" test_storage_long_eio run_test "Hostile: Flaky EIO" test_storage_flaky_eio + run_test "Hostile: Clock Ahead Mount" test_clock_ahead_mount + run_test "Hostile: Clock Ahead Crash Recovered" test_clock_ahead_crash_recovered # Summary log_info "=========================================="