Skip to content
Merged
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- The README quick start now uses the real binary names and flags.
- Create, mkdir, clone and snapshot failed with "No space left on device" when the lock wait for the new inode gave up or its slot could not be read. They now fail with `EIO`. `ENOSPC` means the inode table is full.
- Directory entries went only into the first 4 KB of each 1 MB directory block, so a block held about 250 short names or 15 long ones, and a busy directory took a new block every few creates. Entries now fill the whole block. A lookup reads each block in one I/O, and an add or remove writes only the 4 KB it changed.
- A node that other nodes declared dead while it was stalled, for example a paused VM, kept writing when it resumed. A node now checks before every device write that it still owns its registry slot, and refuses the write and exits if it does not. A write it started before the stall can still land.
- A node stalled for 15 seconds while it held a lock, for example a paused VM, lost the lock to other nodes, which could then write the same metadata. Other nodes now wait for a failed node's locks until it is proven dead and its journal is replayed, with no time limit, and give up only if the replay fails or is abandoned, or after 30 seconds if the waiting operation already holds another lock the replay may need. An overwrite of existing file data instead lets go of its locks and retries, so it does not fail. Failure detection now replays only nodes proven dead.
- A mount that took the registry slot of a failed node on another host did not keep that node's locks while it replayed the node's journal.
- Two nodes replaying dead nodes whose locks shared a record waited on each other until the lock timeout.
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ For clustered deployments:
3. **Mount Command**: Use identical mount options across all nodes
4. **Limits**: At most 64 nodes mount a device at once, and node clocks must agree within a few seconds (run NTP)
5. **Format version**: This release uses on-disk format version 3, which gives each node its own journal and sizes the block bitmap, refcount table and inode table from the device. It refuses devices formatted by earlier builds; copy their data off and reformat
6. **Stalled nodes**: Other nodes declare a node dead after about 20 seconds of silence, for example a paused VM, and replay its journal. The stalled node checks its registry slot before each write, so when it resumes it refuses new writes and exits. A write it started before the stall can still land. Only storage fencing closes that gap, such as SCSI-3 persistent reservations, which GoClusterFS does not use. See [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits)

Format the device once, from one node, with `-format`. Then mount it on every node without `-format`:

Expand Down
1 change: 1 addition & 0 deletions cmd/mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,7 @@ func main() {
if err := nodeReg.StartHeartbeat(); err != nil {
log.Fatalf("Failed to start node heartbeat: %v", err)
}
dev.SetWriteGate(nodeReg.WriteGate)
nodeUUID := nodeReg.LocalUUID()
log.Printf("Registered as node %d (UUID: %x)", slotIdx, nodeUUID[:8])
journalOffset, journalSize := nodeReg.LocalJournal()
Expand Down
9 changes: 7 additions & 2 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ Snapshots and clones share blocks with the file and copy a block on its first wr
### Known Gaps in Slot Ownership

- Slot claims use a read-compare-write on the device, then re-read the slot after 100 ms. Two nodes can still both keep a slot if one node's write lands more than 100 ms after its read, for example during a storage stall.
- A node paused for more than 10 seconds (VM freeze, `SIGSTOP`, a stalled path) looks failed to a new mount, which can take its slot and replay and reset its journal. Running nodes wait until it is proven dead (about 20 seconds, see above). The paused node exits on its next heartbeat after it resumes, up to 1 second later, and can write to the journal region in that second.
- A node paused for more than 10 seconds (VM freeze, `SIGSTOP`, a stalled path) looks failed to a new mount, which can take its slot and replay and reset its journal. Running nodes wait until it is proven dead (about 20 seconds, see above). When the paused node resumes, it refuses further writes and exits, but writes it started before the pause can still land.
- A restarted container comes back in a new PID namespace, so its mount cannot prove by PID that the previous incarnation exited. It waits up to about 12 seconds for the old slot's heartbeat to go stale, then replays that journal before serving.

### Adding a New Node
Expand Down Expand Up @@ -238,7 +238,12 @@ Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts,

**When another node's claim stops counting.** A claim counts until the registry slot with the same number is empty: its node unmounted cleanly, or another node replayed its journal and freed the slot. A node checks this by reading the registry slot from the device after it reads the record, so it cannot miss a node that registered in the meantime. A torn slot follows the same rule. After its replay, a mount clears the claims in its own slot that an earlier occupant left. How old a claim is does not matter, so a lock that a node never releases, for example after a bug, stays held until that node unmounts. The node logs a warning for each lock it has held for more than 60 seconds. A replay that fails or gets stuck keeps the dead node's locks held until a later replay succeeds; waits for them fail with `EIO` once its recovery lease lapses, about a minute after the replay last made progress. A mount whose own journal replay gives up waiting for another node's lock retries every 1 to 30 seconds and logs who holds the lock each time. Any other replay failure, such as a record that does not decode or an I/O error, fails the mount at once with the `-skip-recovery` hint.

**The hazard: a stalled node keeps writing.** If a node stops for about 20 seconds while it holds a lock, other nodes prove it dead, replay its journal and take its locks. When the stalled node resumes, it exits on its next heartbeat, up to 1 second later, but it can finish the writes it had started. No write path checks that the lock is still held, and there is no fencing token. Two nodes then write the same metadata, which can double-allocate blocks or lose directory entries.
**The hazard: a stalled node's last writes.** If a node stops for about 20 seconds while it holds a lock, other nodes prove it dead, replay its journal and take its locks. Before every device write, a node checks that it wrote its heartbeat less than 5 seconds ago. If not, it writes a heartbeat first, and if its registry slot is no longer its own, it refuses the write and exits. Some writes still land after the takeover:

- a write that passed the check just before the stall, and I/O the kernel or the storage queued during it;
- writes in the second after the takeover, if the stalled node resumed while the other node was marking its slot dead.

Two nodes can then write the same metadata, which can double-allocate blocks or lose directory entries. Only fencing at the storage closes this gap, for example SCSI-3 persistent reservations (or NVMe reservations), with which surviving nodes revoke a dead node's access to the device. GoClusterFS does not use them.

Common causes of a stall:

Expand Down
135 changes: 100 additions & 35 deletions internal/node/registry.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ import (
"errors"
"fmt"
"log"
"math"
"os"
"sync"
"sync/atomic"
"time"

"github.com/supporttools/GoClusterFS/internal/storage"
Expand Down Expand Up @@ -116,6 +118,13 @@ type Registry struct {
seenMu sync.Mutex
seen map[int]*Slot

// lastBeat is when this node last wrote its own slot; beatSeq is its last heartbeat number.
lastBeat atomic.Pointer[time.Time]
beatSeq uint64
// lost is set once this node's slot was taken over or its heartbeats failed.
lost atomic.Bool
ownMu sync.Mutex

// Cross-node recovery
recoveryHandler RecoveryHandler
failureCheckInterval time.Duration // How often to scan for failed nodes
Expand All @@ -130,6 +139,7 @@ var (
ErrInvalidSlot = errors.New("invalid slot data")
ErrRecoveryInProgress = errors.New("recovery already in progress for this node")
ErrLeaseLost = errors.New("another node took over this slot's recovery")
ErrSlotLost = errors.New("this node no longer owns its registry slot")
)

// FailedNodeInfo contains information about a failed node for recovery.
Expand Down Expand Up @@ -307,6 +317,9 @@ func (r *Registry) claimFirst(slots []int, claimable func(*Slot) bool) (int, err
return -1, err
}
r.localSlot = i
r.beatSeq = 1
now := time.Now()
r.lastBeat.Store(&now)
if slot.State != NodeStateEmpty {
r.previousOwner = &slot.UUID
}
Expand Down Expand Up @@ -428,49 +441,101 @@ func (r *Registry) heartbeatLoop(slotIdx int) {
ticker := time.NewTicker(HeartbeatInterval)
defer ticker.Stop()

seq := uint64(1)
lost := false
lastBeat := time.Now()

for {
select {
case <-r.stopCh:
return
case <-ticker.C:
r.mu.Lock()
slot, err := r.readSlot(slotIdx)
ours := err == nil && slot.UUID == r.localUUID && slot.State != NodeStateDead
takenOver := err == nil && !ours
if ours {
slot.HeartbeatSeq = seq
slot.HeartbeatTime = time.Now().Unix()
if r.writeSlot(slotIdx, slot) == nil {
lastBeat = time.Now()
}
seq++
}
onSlotLost := r.onSlotLost
r.mu.Unlock()
_ = r.beat(slotIdx)
}
}
}

// Peers prove a silent node dead and take its locks; a node that cannot heartbeat must stop.
failing := time.Since(lastBeat) > HeartbeatTimeout/2
if (takenOver || failing) && !lost {
lost = true
owner := r.localUUID
if takenOver {
owner = slot.UUID
} else {
log.Printf("ERROR: heartbeats to node registry slot %d have failed for %v", slotIdx, time.Since(lastBeat))
}
if onSlotLost != nil {
onSlotLost(slotIdx, owner)
} else {
log.Printf("ERROR: node registry slot %d is no longer this node's (owner %x); "+
"this node no longer owns its journal region", slotIdx, owner[:8])
}
}
// beat writes a heartbeat while slotIdx is still this node's. Peers prove a
// silent node dead and take its locks, so a node whose slot was taken over, or
// whose heartbeats have failed for HeartbeatTimeout/2, must stop: beat tells the
// slot-lost handler once and returns ErrSlotLost.
func (r *Registry) beat(slotIdx int) error {
r.mu.Lock()
slot, err := r.readSlot(slotIdx)
ours := err == nil && slot.UUID == r.localUUID && slot.State != NodeStateDead
takenOver := err == nil && !ours
if ours {
r.beatSeq++
slot.HeartbeatSeq = r.beatSeq
slot.HeartbeatTime = time.Now().Unix()
if r.writeSlot(slotIdx, slot) == nil {
now := time.Now()
r.lastBeat.Store(&now)
}
}
onSlotLost := r.onSlotLost
r.mu.Unlock()

sinceBeat := r.sinceLastBeat()
if !takenOver && sinceBeat <= HeartbeatTimeout/2 {
return nil
}
if r.lost.CompareAndSwap(false, true) {
owner := r.localUUID
if takenOver {
owner = slot.UUID
} else {
log.Printf("ERROR: heartbeats to node registry slot %d have failed for %v", slotIdx, sinceBeat.Round(time.Millisecond))
}
if onSlotLost != nil {
onSlotLost(slotIdx, owner)
} else {
log.Printf("ERROR: node registry slot %d is no longer this node's (owner %x); "+
"this node no longer owns its journal region", slotIdx, owner[:8])
}
}
return ErrSlotLost
}

// CheckOwnership returns nil while this node still owns its registry slot and
// ErrSlotLost once it may not, so a node that stalled past its failover stops
// writing. A heartbeat written within HeartbeatTimeout/2 proves ownership
// without I/O, because peers declare a node dead only after 2*HeartbeatTimeout
// of silence. Otherwise it writes a heartbeat first, as the heartbeat loop would.
func (r *Registry) CheckOwnership() error {
if r.lost.Load() {
return ErrSlotLost
}
if r.sinceLastBeat() < HeartbeatTimeout/2 {
return nil
}
r.ownMu.Lock()
defer r.ownMu.Unlock()
if r.sinceLastBeat() < HeartbeatTimeout/2 {
return nil
}
r.mu.RLock()
slotIdx := r.localSlot
r.mu.RUnlock()
if slotIdx < 0 {
return ErrNotRegistered
}
log.Printf("[WARN] node: the last heartbeat to registry slot %d was %v ago; checking the slot before a write",
slotIdx, r.sinceLastBeat().Round(time.Millisecond))
return r.beat(slotIdx)
}

// WriteGate is a device write gate: it passes writes to the registry, which
// the heartbeat itself must make, and checks ownership before any other write.
func (r *Registry) WriteGate(offset int64) error {
if offset >= int64(r.offset) && offset < int64(r.offset)+MaxNodes*SlotSize {
return nil
}
return r.CheckOwnership()
}

func (r *Registry) sinceLastBeat() time.Duration {
t := r.lastBeat.Load()
if t == nil {
return math.MaxInt64
}
return max(time.Since(*t), time.Now().Round(0).Sub(t.Round(0)))
}

// claimSlot replaces the slot bytes Register judged claimable with this node's
Expand Down
101 changes: 101 additions & 0 deletions internal/node/registry_test.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
package node

import (
"bytes"
"context"
"errors"
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"

Expand Down Expand Up @@ -1323,3 +1326,101 @@ func TestFailedReplayKeepsDeadNodesLocks(t *testing.T) {
t.Errorf("after a failed replay slot 2 is %s with node %x; want it still dead with the failed node", slot.State, slot.UUID[:1])
}
}

func TestCheckOwnershipWritesHeartbeatAfterPause(t *testing.T) {
dev, cleanup := createTestDevice(t, 10<<20)
defer cleanup()
reg := NewRegistry(dev, 0)
slot, err := reg.Register(context.Background(), "node")
if err != nil {
t.Fatal(err)
}
if err := reg.CheckOwnership(); err != nil {
t.Fatalf("right after Register: %v", err)
}

paused := time.Now().Add(-HeartbeatTimeout)
reg.lastBeat.Store(&paused)
before, err := reg.readSlot(slot)
if err != nil {
t.Fatal(err)
}
var logs bytes.Buffer
log.SetOutput(&logs)
err = reg.CheckOwnership()
log.SetOutput(os.Stderr)
if err != nil {
t.Fatalf("slot still ours after a pause: %v", err)
}
if !strings.Contains(logs.String(), "checking the slot before a write") {
t.Errorf("the check after a pause logged nothing:\n%s", logs.String())
}
after, err := reg.readSlot(slot)
if err != nil {
t.Fatal(err)
}
if after.HeartbeatSeq <= before.HeartbeatSeq {
t.Errorf("heartbeat %d after the check, want it past %d", after.HeartbeatSeq, before.HeartbeatSeq)
}
}

func TestCheckOwnershipFailsOnceSlotIsTakenOver(t *testing.T) {
dev, cleanup := createTestDevice(t, 10<<20)
defer cleanup()
reg := NewRegistry(dev, 0)
slot, err := reg.Register(context.Background(), "node")
if err != nil {
t.Fatal(err)
}
var lost []string
reg.SetSlotLostHandler(func(slot int, owner [16]byte) { lost = append(lost, fmt.Sprintf("%d %x", slot, owner[:1])) })

if err := reg.writeSlot(slot, &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: reg.LocalUUID(), RecoveryOwner: [16]byte{9}, RecoveryStart: time.Now().Unix()}); err != nil {
t.Fatal(err)
}
if err := reg.CheckOwnership(); err != nil {
t.Fatalf("with a recent heartbeat the check needs no read: %v", err)
}
paused := time.Now().Add(-HeartbeatTimeout)
reg.lastBeat.Store(&paused)
for range 2 {
if err := reg.CheckOwnership(); !errors.Is(err, ErrSlotLost) {
t.Fatalf("after peers marked the slot dead: %v, want ErrSlotLost", err)
}
}
uuid := reg.LocalUUID()
if want := []string{fmt.Sprintf("%d %x", slot, uuid[:1])}; fmt.Sprint(lost) != fmt.Sprint(want) {
t.Errorf("slot-lost handler calls %v, want %v", lost, want)
}
}

func TestWriteGateRefusesWritesOnceSlotIsLost(t *testing.T) {
dev, cleanup := createTestDevice(t, 10<<20)
defer cleanup()
reg := NewRegistry(dev, 0)
slot, err := reg.Register(context.Background(), "node")
if err != nil {
t.Fatal(err)
}
dev.SetWriteGate(reg.WriteGate)
data := storage.NewAlignedBuffer(storage.SectorSize)
copy(data.Bytes(), bytes.Repeat([]byte{'x'}, storage.SectorSize))
if _, err := dev.WriteAt(data.Bytes(), 1<<20); err != nil {
t.Fatalf("write while the slot is ours: %v", err)
}

if err := reg.writeSlot(slot, &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: reg.LocalUUID(), RecoveryOwner: [16]byte{9}, RecoveryStart: time.Now().Unix()}); err != nil {
t.Fatalf("a registry write must pass the gate: %v", err)
}
paused := time.Now().Add(-HeartbeatTimeout)
reg.lastBeat.Store(&paused)
if _, err := dev.WriteAt(data.Bytes(), 2<<20); !errors.Is(err, ErrSlotLost) {
t.Fatalf("write after peers marked the slot dead: %v, want ErrSlotLost", err)
}
if _, err := dev.ReadAt(data.Bytes(), 2<<20); err != nil {
t.Fatal(err)
}
if !bytes.Equal(data.Bytes(), make([]byte, storage.SectorSize)) {
t.Error("the refused write reached the device")
}
}
27 changes: 24 additions & 3 deletions internal/storage/device.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"io"
"os"
"sync"
"sync/atomic"
"syscall"
"unsafe"
)
Expand Down Expand Up @@ -44,9 +45,11 @@ const (
type Device struct {
ioShards [NumIOShards]sync.Mutex // Sharded locks for parallel I/O
closeMu sync.RWMutex // Protects close; held shared during I/O
file *os.File
path string
size int64
// writeGate, when set, runs before every write and can refuse it.
writeGate atomic.Pointer[func(offset int64) error]
file *os.File
path string
size int64
}

// shardIndex returns the shard index for the given offset.
Expand Down Expand Up @@ -220,6 +223,9 @@ func (d *Device) writeAt(buf []byte, offset int64, sharded bool) (int, error) {
if len(buf) == 0 {
return 0, nil
}
if err := d.CheckWrite(offset); err != nil {
return 0, err
}

// Acquire shared close lock to prevent device close during I/O
d.closeMu.RLock()
Expand All @@ -233,6 +239,21 @@ func (d *Device) writeAt(buf []byte, offset int64, sharded bool) (int, error) {
return d.file.WriteAt(buf, offset)
}

// SetWriteGate makes every write call gate with its offset first, and fail
// with gate's error instead of writing.
func (d *Device) SetWriteGate(gate func(offset int64) error) {
d.writeGate.Store(&gate)
}

// CheckWrite runs the write gate for a write at offset. Callers that change the
// device without WriteAt, such as a discard, call it first.
func (d *Device) CheckWrite(offset int64) error {
if gate := d.writeGate.Load(); gate != nil {
return (*gate)(offset)
}
return nil
}

// Sync ensures all written data is persisted to the device.
// This is a global operation that does not block concurrent I/O.
func (d *Device) Sync() error {
Expand Down
Loading
Loading