diff --git a/CHANGELOG.md b/CHANGELOG.md index 31cce96e..efb2c192 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -85,6 +85,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - The README quick start now uses the real binary names and flags. - Create, mkdir, clone and snapshot failed with "No space left on device" when the lock wait for the new inode gave up or its slot could not be read. They now fail with `EIO`. `ENOSPC` means the inode table is full. - Directory entries went only into the first 4 KB of each 1 MB directory block, so a block held about 250 short names or 15 long ones, and a busy directory took a new block every few creates. Entries now fill the whole block. A lookup reads each block in one I/O, and an add or remove writes only the 4 KB it changed. +- A node that other nodes declared dead while it was stalled, for example a paused VM, kept writing when it resumed. A node now checks before every device write that it still owns its registry slot, and refuses the write and exits if it does not. A write it started before the stall can still land. - A node stalled for 15 seconds while it held a lock, for example a paused VM, lost the lock to other nodes, which could then write the same metadata. Other nodes now wait for a failed node's locks until it is proven dead and its journal is replayed, with no time limit, and give up only if the replay fails or is abandoned, or after 30 seconds if the waiting operation already holds another lock the replay may need. An overwrite of existing file data instead lets go of its locks and retries, so it does not fail. Failure detection now replays only nodes proven dead. - A mount that took the registry slot of a failed node on another host did not keep that node's locks while it replayed the node's journal. - Two nodes replaying dead nodes whose locks shared a record waited on each other until the lock timeout. diff --git a/README.md b/README.md index 65465262..3132316d 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,7 @@ For clustered deployments: 3. **Mount Command**: Use identical mount options across all nodes 4. **Limits**: At most 64 nodes mount a device at once, and node clocks must agree within a few seconds (run NTP) 5. **Format version**: This release uses on-disk format version 3, which gives each node its own journal and sizes the block bitmap, refcount table and inode table from the device. It refuses devices formatted by earlier builds; copy their data off and reformat +6. **Stalled nodes**: Other nodes declare a node dead after about 20 seconds of silence, for example a paused VM, and replay its journal. The stalled node checks its registry slot before each write, so when it resumes it refuses new writes and exits. A write it started before the stall can still land. Only storage fencing closes that gap, such as SCSI-3 persistent reservations, which GoClusterFS does not use. See [Distributed Lock Limits](docs/OPERATIONS.md#distributed-lock-limits) Format the device once, from one node, with `-format`. Then mount it on every node without `-format`: diff --git a/cmd/mount/main.go b/cmd/mount/main.go index 2ebf8269..f8e4f38d 100644 --- a/cmd/mount/main.go +++ b/cmd/mount/main.go @@ -234,6 +234,7 @@ func main() { if err := nodeReg.StartHeartbeat(); err != nil { log.Fatalf("Failed to start node heartbeat: %v", err) } + dev.SetWriteGate(nodeReg.WriteGate) nodeUUID := nodeReg.LocalUUID() log.Printf("Registered as node %d (UUID: %x)", slotIdx, nodeUUID[:8]) journalOffset, journalSize := nodeReg.LocalJournal() diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 6bce7d93..e22ff0a6 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -162,7 +162,7 @@ Snapshots and clones share blocks with the file and copy a block on its first wr ### Known Gaps in Slot Ownership - Slot claims use a read-compare-write on the device, then re-read the slot after 100 ms. Two nodes can still both keep a slot if one node's write lands more than 100 ms after its read, for example during a storage stall. -- A node paused for more than 10 seconds (VM freeze, `SIGSTOP`, a stalled path) looks failed to a new mount, which can take its slot and replay and reset its journal. Running nodes wait until it is proven dead (about 20 seconds, see above). The paused node exits on its next heartbeat after it resumes, up to 1 second later, and can write to the journal region in that second. +- A node paused for more than 10 seconds (VM freeze, `SIGSTOP`, a stalled path) looks failed to a new mount, which can take its slot and replay and reset its journal. Running nodes wait until it is proven dead (about 20 seconds, see above). When the paused node resumes, it refuses further writes and exits, but writes it started before the pause can still land. - A restarted container comes back in a new PID namespace, so its mount cannot prove by PID that the previous incarnation exited. It waits up to about 12 seconds for the old slot's heartbeat to go stale, then replays that journal before serving. ### Adding a New Node @@ -238,7 +238,12 @@ Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts, **When another node's claim stops counting.** A claim counts until the registry slot with the same number is empty: its node unmounted cleanly, or another node replayed its journal and freed the slot. A node checks this by reading the registry slot from the device after it reads the record, so it cannot miss a node that registered in the meantime. A torn slot follows the same rule. After its replay, a mount clears the claims in its own slot that an earlier occupant left. How old a claim is does not matter, so a lock that a node never releases, for example after a bug, stays held until that node unmounts. The node logs a warning for each lock it has held for more than 60 seconds. A replay that fails or gets stuck keeps the dead node's locks held until a later replay succeeds; waits for them fail with `EIO` once its recovery lease lapses, about a minute after the replay last made progress. A mount whose own journal replay gives up waiting for another node's lock retries every 1 to 30 seconds and logs who holds the lock each time. Any other replay failure, such as a record that does not decode or an I/O error, fails the mount at once with the `-skip-recovery` hint. -**The hazard: a stalled node keeps writing.** If a node stops for about 20 seconds while it holds a lock, other nodes prove it dead, replay its journal and take its locks. When the stalled node resumes, it exits on its next heartbeat, up to 1 second later, but it can finish the writes it had started. No write path checks that the lock is still held, and there is no fencing token. Two nodes then write the same metadata, which can double-allocate blocks or lose directory entries. +**The hazard: a stalled node's last writes.** If a node stops for about 20 seconds while it holds a lock, other nodes prove it dead, replay its journal and take its locks. Before every device write, a node checks that it wrote its heartbeat less than 5 seconds ago. If not, it writes a heartbeat first, and if its registry slot is no longer its own, it refuses the write and exits. Some writes still land after the takeover: + +- a write that passed the check just before the stall, and I/O the kernel or the storage queued during it; +- writes in the second after the takeover, if the stalled node resumed while the other node was marking its slot dead. + +Two nodes can then write the same metadata, which can double-allocate blocks or lose directory entries. Only fencing at the storage closes this gap, for example SCSI-3 persistent reservations (or NVMe reservations), with which surviving nodes revoke a dead node's access to the device. GoClusterFS does not use them. Common causes of a stall: diff --git a/internal/node/registry.go b/internal/node/registry.go index b1fced75..4a56ce06 100644 --- a/internal/node/registry.go +++ b/internal/node/registry.go @@ -14,8 +14,10 @@ import ( "errors" "fmt" "log" + "math" "os" "sync" + "sync/atomic" "time" "github.com/supporttools/GoClusterFS/internal/storage" @@ -116,6 +118,13 @@ type Registry struct { seenMu sync.Mutex seen map[int]*Slot + // lastBeat is when this node last wrote its own slot; beatSeq is its last heartbeat number. + lastBeat atomic.Pointer[time.Time] + beatSeq uint64 + // lost is set once this node's slot was taken over or its heartbeats failed. + lost atomic.Bool + ownMu sync.Mutex + // Cross-node recovery recoveryHandler RecoveryHandler failureCheckInterval time.Duration // How often to scan for failed nodes @@ -130,6 +139,7 @@ var ( ErrInvalidSlot = errors.New("invalid slot data") ErrRecoveryInProgress = errors.New("recovery already in progress for this node") ErrLeaseLost = errors.New("another node took over this slot's recovery") + ErrSlotLost = errors.New("this node no longer owns its registry slot") ) // FailedNodeInfo contains information about a failed node for recovery. @@ -307,6 +317,9 @@ func (r *Registry) claimFirst(slots []int, claimable func(*Slot) bool) (int, err return -1, err } r.localSlot = i + r.beatSeq = 1 + now := time.Now() + r.lastBeat.Store(&now) if slot.State != NodeStateEmpty { r.previousOwner = &slot.UUID } @@ -428,49 +441,101 @@ func (r *Registry) heartbeatLoop(slotIdx int) { ticker := time.NewTicker(HeartbeatInterval) defer ticker.Stop() - seq := uint64(1) - lost := false - lastBeat := time.Now() - for { select { case <-r.stopCh: return case <-ticker.C: - r.mu.Lock() - slot, err := r.readSlot(slotIdx) - ours := err == nil && slot.UUID == r.localUUID && slot.State != NodeStateDead - takenOver := err == nil && !ours - if ours { - slot.HeartbeatSeq = seq - slot.HeartbeatTime = time.Now().Unix() - if r.writeSlot(slotIdx, slot) == nil { - lastBeat = time.Now() - } - seq++ - } - onSlotLost := r.onSlotLost - r.mu.Unlock() + _ = r.beat(slotIdx) + } + } +} - // Peers prove a silent node dead and take its locks; a node that cannot heartbeat must stop. - failing := time.Since(lastBeat) > HeartbeatTimeout/2 - if (takenOver || failing) && !lost { - lost = true - owner := r.localUUID - if takenOver { - owner = slot.UUID - } else { - log.Printf("ERROR: heartbeats to node registry slot %d have failed for %v", slotIdx, time.Since(lastBeat)) - } - if onSlotLost != nil { - onSlotLost(slotIdx, owner) - } else { - log.Printf("ERROR: node registry slot %d is no longer this node's (owner %x); "+ - "this node no longer owns its journal region", slotIdx, owner[:8]) - } - } +// beat writes a heartbeat while slotIdx is still this node's. Peers prove a +// silent node dead and take its locks, so a node whose slot was taken over, or +// whose heartbeats have failed for HeartbeatTimeout/2, must stop: beat tells the +// slot-lost handler once and returns ErrSlotLost. +func (r *Registry) beat(slotIdx int) error { + r.mu.Lock() + slot, err := r.readSlot(slotIdx) + ours := err == nil && slot.UUID == r.localUUID && slot.State != NodeStateDead + takenOver := err == nil && !ours + if ours { + r.beatSeq++ + slot.HeartbeatSeq = r.beatSeq + slot.HeartbeatTime = time.Now().Unix() + if r.writeSlot(slotIdx, slot) == nil { + now := time.Now() + r.lastBeat.Store(&now) + } + } + onSlotLost := r.onSlotLost + r.mu.Unlock() + + sinceBeat := r.sinceLastBeat() + if !takenOver && sinceBeat <= HeartbeatTimeout/2 { + return nil + } + if r.lost.CompareAndSwap(false, true) { + owner := r.localUUID + if takenOver { + owner = slot.UUID + } else { + log.Printf("ERROR: heartbeats to node registry slot %d have failed for %v", slotIdx, sinceBeat.Round(time.Millisecond)) } + if onSlotLost != nil { + onSlotLost(slotIdx, owner) + } else { + log.Printf("ERROR: node registry slot %d is no longer this node's (owner %x); "+ + "this node no longer owns its journal region", slotIdx, owner[:8]) + } + } + return ErrSlotLost +} + +// CheckOwnership returns nil while this node still owns its registry slot and +// ErrSlotLost once it may not, so a node that stalled past its failover stops +// writing. A heartbeat written within HeartbeatTimeout/2 proves ownership +// without I/O, because peers declare a node dead only after 2*HeartbeatTimeout +// of silence. Otherwise it writes a heartbeat first, as the heartbeat loop would. +func (r *Registry) CheckOwnership() error { + if r.lost.Load() { + return ErrSlotLost + } + if r.sinceLastBeat() < HeartbeatTimeout/2 { + return nil + } + r.ownMu.Lock() + defer r.ownMu.Unlock() + if r.sinceLastBeat() < HeartbeatTimeout/2 { + return nil + } + r.mu.RLock() + slotIdx := r.localSlot + r.mu.RUnlock() + if slotIdx < 0 { + return ErrNotRegistered + } + log.Printf("[WARN] node: the last heartbeat to registry slot %d was %v ago; checking the slot before a write", + slotIdx, r.sinceLastBeat().Round(time.Millisecond)) + return r.beat(slotIdx) +} + +// WriteGate is a device write gate: it passes writes to the registry, which +// the heartbeat itself must make, and checks ownership before any other write. +func (r *Registry) WriteGate(offset int64) error { + if offset >= int64(r.offset) && offset < int64(r.offset)+MaxNodes*SlotSize { + return nil + } + return r.CheckOwnership() +} + +func (r *Registry) sinceLastBeat() time.Duration { + t := r.lastBeat.Load() + if t == nil { + return math.MaxInt64 } + return max(time.Since(*t), time.Now().Round(0).Sub(t.Round(0))) } // claimSlot replaces the slot bytes Register judged claimable with this node's diff --git a/internal/node/registry_test.go b/internal/node/registry_test.go index 3e42b8fb..08879381 100644 --- a/internal/node/registry_test.go +++ b/internal/node/registry_test.go @@ -1,12 +1,15 @@ package node import ( + "bytes" "context" "errors" "fmt" + "log" "os" "os/exec" "path/filepath" + "strings" "testing" "time" @@ -1323,3 +1326,101 @@ func TestFailedReplayKeepsDeadNodesLocks(t *testing.T) { t.Errorf("after a failed replay slot 2 is %s with node %x; want it still dead with the failed node", slot.State, slot.UUID[:1]) } } + +func TestCheckOwnershipWritesHeartbeatAfterPause(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + slot, err := reg.Register(context.Background(), "node") + if err != nil { + t.Fatal(err) + } + if err := reg.CheckOwnership(); err != nil { + t.Fatalf("right after Register: %v", err) + } + + paused := time.Now().Add(-HeartbeatTimeout) + reg.lastBeat.Store(&paused) + before, err := reg.readSlot(slot) + if err != nil { + t.Fatal(err) + } + var logs bytes.Buffer + log.SetOutput(&logs) + err = reg.CheckOwnership() + log.SetOutput(os.Stderr) + if err != nil { + t.Fatalf("slot still ours after a pause: %v", err) + } + if !strings.Contains(logs.String(), "checking the slot before a write") { + t.Errorf("the check after a pause logged nothing:\n%s", logs.String()) + } + after, err := reg.readSlot(slot) + if err != nil { + t.Fatal(err) + } + if after.HeartbeatSeq <= before.HeartbeatSeq { + t.Errorf("heartbeat %d after the check, want it past %d", after.HeartbeatSeq, before.HeartbeatSeq) + } +} + +func TestCheckOwnershipFailsOnceSlotIsTakenOver(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + slot, err := reg.Register(context.Background(), "node") + if err != nil { + t.Fatal(err) + } + var lost []string + reg.SetSlotLostHandler(func(slot int, owner [16]byte) { lost = append(lost, fmt.Sprintf("%d %x", slot, owner[:1])) }) + + if err := reg.writeSlot(slot, &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: reg.LocalUUID(), RecoveryOwner: [16]byte{9}, RecoveryStart: time.Now().Unix()}); err != nil { + t.Fatal(err) + } + if err := reg.CheckOwnership(); err != nil { + t.Fatalf("with a recent heartbeat the check needs no read: %v", err) + } + paused := time.Now().Add(-HeartbeatTimeout) + reg.lastBeat.Store(&paused) + for range 2 { + if err := reg.CheckOwnership(); !errors.Is(err, ErrSlotLost) { + t.Fatalf("after peers marked the slot dead: %v, want ErrSlotLost", err) + } + } + uuid := reg.LocalUUID() + if want := []string{fmt.Sprintf("%d %x", slot, uuid[:1])}; fmt.Sprint(lost) != fmt.Sprint(want) { + t.Errorf("slot-lost handler calls %v, want %v", lost, want) + } +} + +func TestWriteGateRefusesWritesOnceSlotIsLost(t *testing.T) { + dev, cleanup := createTestDevice(t, 10<<20) + defer cleanup() + reg := NewRegistry(dev, 0) + slot, err := reg.Register(context.Background(), "node") + if err != nil { + t.Fatal(err) + } + dev.SetWriteGate(reg.WriteGate) + data := storage.NewAlignedBuffer(storage.SectorSize) + copy(data.Bytes(), bytes.Repeat([]byte{'x'}, storage.SectorSize)) + if _, err := dev.WriteAt(data.Bytes(), 1<<20); err != nil { + t.Fatalf("write while the slot is ours: %v", err) + } + + if err := reg.writeSlot(slot, &Slot{Magic: SlotMagic, State: NodeStateDead, UUID: reg.LocalUUID(), RecoveryOwner: [16]byte{9}, RecoveryStart: time.Now().Unix()}); err != nil { + t.Fatalf("a registry write must pass the gate: %v", err) + } + paused := time.Now().Add(-HeartbeatTimeout) + reg.lastBeat.Store(&paused) + if _, err := dev.WriteAt(data.Bytes(), 2<<20); !errors.Is(err, ErrSlotLost) { + t.Fatalf("write after peers marked the slot dead: %v, want ErrSlotLost", err) + } + if _, err := dev.ReadAt(data.Bytes(), 2<<20); err != nil { + t.Fatal(err) + } + if !bytes.Equal(data.Bytes(), make([]byte, storage.SectorSize)) { + t.Error("the refused write reached the device") + } +} diff --git a/internal/storage/device.go b/internal/storage/device.go index 8f1367e2..b1b4e6c8 100644 --- a/internal/storage/device.go +++ b/internal/storage/device.go @@ -11,6 +11,7 @@ import ( "io" "os" "sync" + "sync/atomic" "syscall" "unsafe" ) @@ -44,9 +45,11 @@ const ( type Device struct { ioShards [NumIOShards]sync.Mutex // Sharded locks for parallel I/O closeMu sync.RWMutex // Protects close; held shared during I/O - file *os.File - path string - size int64 + // writeGate, when set, runs before every write and can refuse it. + writeGate atomic.Pointer[func(offset int64) error] + file *os.File + path string + size int64 } // shardIndex returns the shard index for the given offset. @@ -220,6 +223,9 @@ func (d *Device) writeAt(buf []byte, offset int64, sharded bool) (int, error) { if len(buf) == 0 { return 0, nil } + if err := d.CheckWrite(offset); err != nil { + return 0, err + } // Acquire shared close lock to prevent device close during I/O d.closeMu.RLock() @@ -233,6 +239,21 @@ func (d *Device) writeAt(buf []byte, offset int64, sharded bool) (int, error) { return d.file.WriteAt(buf, offset) } +// SetWriteGate makes every write call gate with its offset first, and fail +// with gate's error instead of writing. +func (d *Device) SetWriteGate(gate func(offset int64) error) { + d.writeGate.Store(&gate) +} + +// CheckWrite runs the write gate for a write at offset. Callers that change the +// device without WriteAt, such as a discard, call it first. +func (d *Device) CheckWrite(offset int64) error { + if gate := d.writeGate.Load(); gate != nil { + return (*gate)(offset) + } + return nil +} + // Sync ensures all written data is persisted to the device. // This is a global operation that does not block concurrent I/O. func (d *Device) Sync() error { diff --git a/internal/storage/device_test.go b/internal/storage/device_test.go index b44eabc7..8bf5b57b 100644 --- a/internal/storage/device_test.go +++ b/internal/storage/device_test.go @@ -1,6 +1,8 @@ package storage import ( + "bytes" + "errors" "os" "path/filepath" "sync" @@ -369,3 +371,42 @@ func TestMultiShardIO(t *testing.T) { } } } + +func TestWriteGateRefusesWrites(t *testing.T) { + tmpDir := os.Getenv("GOCLUSTERFS_TEST_DIR") + if tmpDir == "" { + tmpDir = t.TempDir() + } + path := filepath.Join(tmpDir, "test_gate.bin") + if err := os.WriteFile(path, make([]byte, 2*SectorSize), 0o644); err != nil { + t.Fatal(err) + } + defer func() { _ = os.Remove(path) }() + dev, err := Open(path) + if err != nil { + t.Skipf("O_DIRECT not supported: %v", err) + } + defer func() { _ = dev.Close() }() + + refused := errors.New("refused") + dev.SetWriteGate(func(offset int64) error { + if offset >= SectorSize { + return refused + } + return nil + }) + buf := NewAlignedBuffer(SectorSize) + copy(buf.Bytes(), bytes.Repeat([]byte{'x'}, SectorSize)) + if _, err := dev.WriteAt(buf.Bytes(), 0); err != nil { + t.Fatalf("write the gate allows: %v", err) + } + if _, err := dev.WriteAtUnsharded(buf.Bytes(), SectorSize); !errors.Is(err, refused) { + t.Fatalf("write the gate refuses = %v, want %v", err, refused) + } + if _, err := dev.ReadAt(buf.Bytes(), SectorSize); err != nil { + t.Fatal(err) + } + if !bytes.Equal(buf.Bytes(), make([]byte, SectorSize)) { + t.Error("a refused write reached the device") + } +} diff --git a/internal/trim/trim.go b/internal/trim/trim.go index 7d3cfcb7..f35bf8ef 100644 --- a/internal/trim/trim.go +++ b/internal/trim/trim.go @@ -367,6 +367,9 @@ func (m *Manager) issueDiscard(r blockRange) error { // startBlock is relative to data region, add DataOffset byteOffset := m.config.DataOffset + r.startBlock*BlockSize byteLength := r.count * BlockSize + if err := m.dev.CheckWrite(int64(byteOffset)); err != nil { + return err + } if m.config.Discard != nil { return m.config.Discard(byteOffset, byteLength) } diff --git a/internal/trim/trim_test.go b/internal/trim/trim_test.go index d1f9794b..e029f22e 100644 --- a/internal/trim/trim_test.go +++ b/internal/trim/trim_test.go @@ -445,3 +445,23 @@ func TestRefusedFlushCountsAsError(t *testing.T) { t.Errorf("Stats = %+v, want 1 error and nothing discarded", got) } } + +func TestDiscardRefusedByWriteGate(t *testing.T) { + dev, cleanup := createTestDevice(t) + defer cleanup() + dev.SetWriteGate(func(int64) error { return errors.New("node lost its registry slot") }) + + discarded := false + m := NewManager(dev, Config{BatchSize: 1000, MaxAge: time.Hour, Enabled: true, Discard: func(offset, length uint64) error { + discarded = true + return nil + }}) + m.Start() + m.Queue(5) + if err := m.Stop(); err != nil { + t.Fatal(err) + } + if discarded { + t.Error("a discard ran though the write gate refused it") + } +} diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index 401d38cc..90dd73e6 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -2981,6 +2981,41 @@ scenarios: environments: go: internal/node/registry_test.go::TestRegisterReportsPreviousOwnerOfAnyTakenSlot + - id: storage-write-gate-refuses-writes + name: Device write gate refuses writes + description: A write the gate refuses fails with the gate's error and does not reach the device + priority: go_only + environments: + go: internal/storage/device_test.go::TestWriteGateRefusesWrites + + - id: trim-discard-refused-by-write-gate + name: Discard refused by the write gate + description: A queued discard does not run when the device's write gate refuses it + priority: go_only + environments: + go: internal/trim/trim_test.go::TestDiscardRefusedByWriteGate + + - id: registry-check-ownership-after-pause + name: Ownership check after a pause writes a heartbeat + description: When the last heartbeat is old, CheckOwnership writes a heartbeat and passes while the slot is still the node's + priority: go_only + environments: + go: internal/node/registry_test.go::TestCheckOwnershipWritesHeartbeatAfterPause + + - id: registry-check-ownership-after-takeover + name: Ownership check fails once the slot is taken over + description: Once peers marked the slot dead, CheckOwnership returns ErrSlotLost and calls the slot-lost handler once + priority: go_only + environments: + go: internal/node/registry_test.go::TestCheckOwnershipFailsOnceSlotIsTakenOver + + - id: registry-write-gate-after-takeover + name: Write gate refuses data writes once the slot is lost + description: With the registry's write gate on the device, a node whose slot was marked dead while it was paused cannot write outside the registry, and the write does not reach the device + priority: go_only + environments: + go: internal/node/registry_test.go::TestWriteGateRefusesWritesOnceSlotIsLost + - id: lock-leaked-shared-count-goes-stale name: Leaked read claim goes stale description: New readers do not refresh another node's read claim, so a read claim leaked by a stopped node stops blocking writers once it is 15 seconds old