diff --git a/CHANGELOG.md b/CHANGELOG.md index 93b2bdf1..60716878 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -85,3 +85,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - The README quick start now uses the real binary names and flags. - Create, mkdir, clone and snapshot failed with "No space left on device" when the lock wait for the new inode gave up or its slot could not be read. They now fail with `EIO`. `ENOSPC` means the inode table is full. - Directory entries went only into the first 4 KB of each 1 MB directory block, so a block held about 250 short names or 15 long ones, and a busy directory took a new block every few creates. Entries now fill the whole block. A lookup reads each block in one I/O, and an add or remove writes only the 4 KB it changed. +- A lock held for more than 15 seconds by one operation, or by a node replaying a crashed node's journal, went stale, and another node could take it while it was still in use. The holding node now writes its claims again every few seconds in the background. diff --git a/cmd/mount/main.go b/cmd/mount/main.go index bc1f977b..88dc044a 100644 --- a/cmd/mount/main.go +++ b/cmd/mount/main.go @@ -240,6 +240,7 @@ func main() { // Create lock manager for distributed locking lockMgr := lock.NewManager(dev, lockOffset, lockSize, nodeUUID, slotIdx) + lockMgr.Start() nodeReg.SetOwnerDeadHandler(func(owner [16]byte, slot int) func() { return adoptLocks(lockMgr, owner, slot) }) releasePrevious := func() {} if owner, ok := nodeReg.DeadPreviousOwner(); ok { diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 435e5751..52452ff7 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -236,7 +236,7 @@ Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts, **When claims collide.** The node in the lower registry slot keeps its claim and waits for the other claims to go. A node that sees a claim in a lower slot clears its own and waits, reading the record without writing until that claim is gone. Waits only point from lower to higher slots, so two nodes never wait on each other. A claim kept for a journal replay counts as lower than every slot, so the replaying node is never stuck behind a waiter. A slot that fails its checksum, such as one torn by a crash in the middle of a write, counts as a claim. -**How a lock goes stale.** A claim stores the wall-clock time its node wrote it. Other nodes ignore a claim older than 15 seconds (`StaleThreshold`), judged by their own clocks. This is how locks held by a crashed node get freed. A node writes its claim again when another operation on the same node joins a hold more than 5 seconds old, and before it reuses a kept lock that old; a single long operation is not renewed. A torn slot has no readable time. Each node treats it as a claim until it has read it torn for 15 seconds, or until its node is proven dead and the node that replays its journal clears it. +**How a lock goes stale.** A claim stores the wall-clock time its node wrote it. Other nodes ignore a claim older than 15 seconds (`StaleThreshold`), judged by their own clocks. This is how locks held by a crashed node get freed. While a node holds a lock, it writes its claim again in the background every 2.5 to 4 seconds, so a long operation never looks stale to other nodes. It does the same for the claims it took over from a dead node until their replay ends. A torn slot has no readable time. Each node treats it as a claim until it has read it torn for 15 seconds, or until its node is proven dead and the node that replays its journal clears it. **The hazard: a stalled node keeps writing.** If a node stops for more than 15 seconds while it holds a lock, another node takes the lock. When the stalled node resumes, it finishes its writes. No write path checks that the lock is still held, and there is no fencing token. The stalled node is not told it lost the lock; it holds it in memory until it releases it. Two nodes then write the same metadata, which can double-allocate blocks or lose directory entries. @@ -253,9 +253,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no **Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves. -**Read locks are not renewed.** A node whose overlapping lookups keep a directory read-locked for more than 15 seconds can lose that lock to another node's writer. Its readers may then see a half-written directory. Two writers still cannot overlap. - -**Locks share records.** The lock region has 256 records, and every lock maps to one of them, so unrelated files and the block bitmap can share a record. A node holding one of those locks blocks other nodes from all of them. If two nodes each hold one record and wait for the other's, both wait until one record is 15 seconds old; the takeover then lets both proceed, with the same risk as a stall. This needs nested locks whose records cross, roughly 1 in 65,000 pairs of overlapping operations. +**Locks share records.** The lock region has 256 records, and every lock maps to one of them, so unrelated files and the block bitmap can share a record. A node holding one of those locks blocks other nodes from all of them. If two nodes each hold one record and wait for the other's, neither claim goes stale while both nodes run. The first of the two waits gives up after 30 seconds and its operation fails with `EIO`; the other then proceeds. This needs nested locks whose records cross, roughly 1 in 65,000 pairs of overlapping operations. **Safe operating limits:** diff --git a/internal/goclusterfs/fs.go b/internal/goclusterfs/fs.go index 8993c336..c7461631 100644 --- a/internal/goclusterfs/fs.go +++ b/internal/goclusterfs/fs.go @@ -315,6 +315,7 @@ func (f *FS) Close() error { } // Other nodes would otherwise wait for these holds to go stale. if f.lockMgr != nil { + f.lockMgr.Stop() f.lockMgr.ReleaseCached() f.lockMgr.LogStats() } diff --git a/internal/lock/lock.go b/internal/lock/lock.go index 18c18a2d..cc247471 100644 --- a/internal/lock/lock.go +++ b/internal/lock/lock.go @@ -145,6 +145,8 @@ type Manager struct { tornCount atomic.Int32 // slotWrite, when a test sets it, replaces the device write of a slot. slotWrite func(buf []byte, offset int64) (int, error) + // stopRenew and renewDone run the renewer between Start and Stop. + stopRenew, renewDone chan struct{} // Callback for cache invalidation onAcquire OnAcquireFunc @@ -182,6 +184,7 @@ type heldLock struct { type heldRecord struct { mode LockMode + key lockKey // the key this node's slot names keys int since time.Time // when this node claimed the record renewed time.Time // when this node last wrote its slot @@ -207,6 +210,9 @@ func covers(held, want LockMode) bool { // relying on it, so other nodes never see a slot in use as stale. var renewAfter = StaleThreshold / 3 +// staleAfter is StaleThreshold; tests shorten it. +var staleAfter = StaleThreshold + // acquireTimeout is how long Acquire waits at most; tests shorten it. var acquireTimeout = DefaultTimeout @@ -504,7 +510,7 @@ func (m *Manager) attemptAcquire(key lockKey, recordID uint64, mode LockMode, gi m.countAcquire(key.Type, func(c *acquireCounts) *atomic.Uint64 { return &c.claimed }) m.mu.Lock() - m.records[recordID] = &heldRecord{mode: mode, keys: 1, since: time.Now(), renewed: written} + m.records[recordID] = &heldRecord{mode: mode, key: key, keys: 1, since: time.Now(), renewed: written} m.addHeldKeyLocked(key, recordID, mode, gid) m.mu.Unlock() return true, true, nil diff --git a/internal/lock/reclaim.go b/internal/lock/reclaim.go index 3a4d75da..facdd87a 100644 --- a/internal/lock/reclaim.go +++ b/internal/lock/reclaim.go @@ -53,13 +53,14 @@ func (m *Manager) adoptRecord(recordID uint64, owner [16]byte, ownerSlot int) (b case err != nil: dead = &claimSlot{owner: owner} } + now := time.Now() dead.mode, dead.adopted = LockModeExclusive, true - dead.acquireTime = time.Now().UnixNano() + dead.acquireTime = now.UnixNano() if err := m.writeSlot(recordID, ownerSlot, dead); err != nil { return false, err } m.mu.Lock() - m.adopted[recordID] = adoptedSlot{slot: ownerSlot, owner: owner} + m.adopted[recordID] = adoptedSlot{slot: ownerSlot, owner: owner, renewed: now} m.mu.Unlock() return true, nil } diff --git a/internal/lock/renew.go b/internal/lock/renew.go new file mode 100644 index 00000000..9f87f720 --- /dev/null +++ b/internal/lock/renew.go @@ -0,0 +1,137 @@ +package lock + +import ( + "log" + "time" + + "github.com/supporttools/GoClusterFS/internal/storage" +) + +// Start runs a renewer that rewrites this node's held slots and the slots it +// adopted for a replay, so peers never see them as stale however long an +// operation or a replay runs. Stop ends it. +func (m *Manager) Start() { + m.mu.Lock() + defer m.mu.Unlock() + if m.stopRenew != nil { + return + } + m.stopRenew, m.renewDone = make(chan struct{}), make(chan struct{}) + go m.renewLoop(m.stopRenew, m.renewDone) +} + +// Stop ends the renewer and waits for it to finish. +func (m *Manager) Stop() { + m.mu.Lock() + stop, done := m.stopRenew, m.renewDone + m.stopRenew, m.renewDone = nil, nil + m.mu.Unlock() + if stop != nil { + close(stop) + <-done + } +} + +func (m *Manager) renewLoop(stop <-chan struct{}, done chan<- struct{}) { + defer close(done) + tick := time.NewTicker(renewAfter / 4) + defer tick.Stop() + for { + select { + case <-stop: + return + case <-tick.C: + m.renewDue() + } + } +} + +// renewDue rewrites the slots this node wrote more than renewAfter/2 ago, so +// they stay younger than renewAfter, which also spares joins an inline renewal. +func (m *Manager) renewDue() { + m.mu.Lock() + var held, adopted []uint64 + for recordID, rec := range m.records { + if age(rec.renewed) > renewAfter/2 { + held = append(held, recordID) + } + } + for recordID, a := range m.adopted { + if age(a.renewed) > renewAfter/2 { + adopted = append(adopted, recordID) + } + } + m.mu.Unlock() + + for _, recordID := range held { + if err := m.renewHeld(recordID); err != nil { + log.Printf("[WARN] lock: renewing record %d: %v", recordID, err) + } + } + for _, recordID := range adopted { + if err := m.renewAdopted(recordID); err != nil { + log.Printf("[WARN] lock: renewing adopted record %d: %v", recordID, err) + } + } +} + +func (m *Manager) renewHeld(recordID uint64) error { + gate := m.recordGate(recordID) + gate.Lock() + defer gate.Unlock() + + m.mu.Lock() + rec, ok := m.records[recordID] + due := ok && age(rec.renewed) > renewAfter/2 + m.mu.Unlock() + if !due { + return nil + } + return m.renewRecord(recordID, rec.key, rec) +} + +// renewAdopted rewrites a slot AdoptOwner kept, unless it no longer holds that +// adoption. An adoption in this node's own slot that a hold or claim uses is +// renewed with that hold or claim. +func (m *Manager) renewAdopted(recordID uint64) error { + gate := m.recordGate(recordID) + gate.Lock() + defer gate.Unlock() + + m.mu.Lock() + a, ok := m.adopted[recordID] + m.mu.Unlock() + if !ok || age(a.renewed) <= renewAfter/2 { + return nil + } + now := time.Now() + if a.slot == m.nodeSlot { + if _, inUse := m.ownSlotUse(recordID); inUse { + return nil + } + if err := m.writeSlot(recordID, a.slot, m.ownClaim(recordID, lockKey{}, LockModeExclusive, now)); err != nil { + return err + } + } else { + buf := storage.GetBuffer(SlotSize) + defer storage.PutBuffer(buf) + if _, err := m.dev.ReadAtUnsharded(buf.Bytes(), m.slotOffset(recordID, a.slot)); err != nil { + return err + } + s, err := decodeSlot(buf.Bytes()) + if err != nil || s == nil || !s.adopted || s.owner != a.owner { + return nil + } + s.acquireTime = now.UnixNano() + if err := m.writeSlot(recordID, a.slot, s); err != nil { + return err + } + } + m.mu.Lock() + if cur, ok := m.adopted[recordID]; ok { + cur.renewed = now + m.adopted[recordID] = cur + } + m.mu.Unlock() + return nil +} diff --git a/internal/lock/renew_test.go b/internal/lock/renew_test.go new file mode 100644 index 00000000..8ab1dee2 --- /dev/null +++ b/internal/lock/renew_test.go @@ -0,0 +1,57 @@ +package lock + +import ( + "testing" + "time" +) + +func shortenStaleness(t *testing.T) { + t.Helper() + oldStale, oldRenew := staleAfter, renewAfter + t.Cleanup(func() { staleAfter, renewAfter = oldStale, oldRenew }) + staleAfter, renewAfter = 300*time.Millisecond, 100*time.Millisecond +} + +func TestLongHoldStaysFreshWithoutNewAcquires(t *testing.T) { + shortenStaleness(t) + dev, cleanup := createTestDevice(t, 1<<20) + defer cleanup() + holder := newTestNode(dev, 0, 1<<20, 1) + peer := newTestNode(dev, 0, 1<<20, 2) + holder.Start() + defer holder.Stop() + + requireAcquired(t, holder, LockTypeInode, 2, LockModeExclusive) + time.Sleep(3 * staleAfter) + requireBlocked(t, peer, LockTypeInode, 2, LockModeExclusive) + + if err := holder.Release(LockTypeInode, 2); err != nil { + t.Fatal(err) + } + holder.ReleaseCached() + requireAcquired(t, peer, LockTypeInode, 2, LockModeExclusive) +} + +func TestAdoptedSlotsStayFreshUntilReleased(t *testing.T) { + shortenStaleness(t) + dev, cleanup := createTestDevice(t, reclaimRegion) + defer cleanup() + dead := NewManager(dev, 0, reclaimRegion, [16]byte{1}, 1) + replayer := NewManager(dev, 0, reclaimRegion, [16]byte{3}, 3) + peer := NewManager(dev, 0, reclaimRegion, [16]byte{4}, 4) + replayer.Start() + defer replayer.Stop() + + requireAcquired(t, dead, LockTypeInode, 2, LockModeExclusive) + adopted, err := replayer.AdoptOwner([16]byte{1}, 1) + if err != nil || len(adopted) != 1 { + t.Fatalf("AdoptOwner = %v, %v; want the dead node's record", adopted, err) + } + time.Sleep(3 * staleAfter) + requireBlocked(t, peer, LockTypeInode, 2, LockModeExclusive) + + if err := replayer.ReleaseAdopted(adopted); err != nil { + t.Fatal(err) + } + requireAcquired(t, peer, LockTypeInode, 2, LockModeExclusive) +} diff --git a/internal/lock/slots.go b/internal/lock/slots.go index 4af131d3..79d5c9a4 100644 --- a/internal/lock/slots.go +++ b/internal/lock/slots.go @@ -47,8 +47,9 @@ type claimSlot struct { } type adoptedSlot struct { - slot int - owner [16]byte + slot int + owner [16]byte + renewed time.Time } type slotRef struct { @@ -105,7 +106,7 @@ func isZeroSlot(buf []byte) bool { } func (s *claimSlot) stale() bool { - return time.Since(time.Unix(0, s.acquireTime)) > StaleThreshold + return time.Since(time.Unix(0, s.acquireTime)) > staleAfter } func rowSlot(row []byte, slot int) []byte { diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index 6933a5ce..22ff53e6 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -2760,6 +2760,20 @@ scenarios: environments: go: internal/goclusterfs/lock_commit_test.go::TestWriteWhenInodeAndBlockShareLockRecord + - id: lock-long-hold-stays-fresh + name: Long lock hold stays fresh + description: A node holding a lock longer than the stale threshold, with no new acquires, keeps other nodes out because a background renewer rewrites its claim slot + priority: go_only + environments: + go: internal/lock/renew_test.go::TestLongHoldStaysFreshWithoutNewAcquires + + - id: lock-adopted-slots-stay-fresh + name: Adopted lock slots stay fresh during a long replay + description: Claims a replaying node took over from a dead node keep other nodes out past the stale threshold until ReleaseAdopted + priority: go_only + environments: + go: internal/lock/renew_test.go::TestAdoptedSlotsStayFreshUntilReleased + - id: lock-leaked-shared-count-goes-stale name: Leaked read claim goes stale description: New readers do not refresh another node's read claim, so a read claim leaked by a stopped node stops blocking writers once it is 15 seconds old