Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- The README quick start now uses the real binary names and flags.
- Create, mkdir, clone and snapshot failed with "No space left on device" when the lock wait for the new inode gave up or its slot could not be read. They now fail with `EIO`. `ENOSPC` means the inode table is full.
- Directory entries went only into the first 4 KB of each 1 MB directory block, so a block held about 250 short names or 15 long ones, and a busy directory took a new block every few creates. Entries now fill the whole block. A lookup reads each block in one I/O, and an add or remove writes only the 4 KB it changed.
- A lock held for more than 15 seconds by one operation, or by a node replaying a crashed node's journal, went stale, and another node could take it while it was still in use. The holding node now writes its claims again every few seconds in the background.
1 change: 1 addition & 0 deletions cmd/mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,7 @@ func main() {

// Create lock manager for distributed locking
lockMgr := lock.NewManager(dev, lockOffset, lockSize, nodeUUID, slotIdx)
lockMgr.Start()
nodeReg.SetOwnerDeadHandler(func(owner [16]byte, slot int) func() { return adoptLocks(lockMgr, owner, slot) })
releasePrevious := func() {}
if owner, ok := nodeReg.DeadPreviousOwner(); ok {
Expand Down
6 changes: 2 additions & 4 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts,

**When claims collide.** The node in the lower registry slot keeps its claim and waits for the other claims to go. A node that sees a claim in a lower slot clears its own and waits, reading the record without writing until that claim is gone. Waits only point from lower to higher slots, so two nodes never wait on each other. A claim kept for a journal replay counts as lower than every slot, so the replaying node is never stuck behind a waiter. A slot that fails its checksum, such as one torn by a crash in the middle of a write, counts as a claim.

**How a lock goes stale.** A claim stores the wall-clock time its node wrote it. Other nodes ignore a claim older than 15 seconds (`StaleThreshold`), judged by their own clocks. This is how locks held by a crashed node get freed. A node writes its claim again when another operation on the same node joins a hold more than 5 seconds old, and before it reuses a kept lock that old; a single long operation is not renewed. A torn slot has no readable time. Each node treats it as a claim until it has read it torn for 15 seconds, or until its node is proven dead and the node that replays its journal clears it.
**How a lock goes stale.** A claim stores the wall-clock time its node wrote it. Other nodes ignore a claim older than 15 seconds (`StaleThreshold`), judged by their own clocks. This is how locks held by a crashed node get freed. While a node holds a lock, it writes its claim again in the background every 2.5 to 4 seconds, so a long operation never looks stale to other nodes. It does the same for the claims it took over from a dead node until their replay ends. A torn slot has no readable time. Each node treats it as a claim until it has read it torn for 15 seconds, or until its node is proven dead and the node that replays its journal clears it.

**The hazard: a stalled node keeps writing.** If a node stops for more than 15 seconds while it holds a lock, another node takes the lock. When the stalled node resumes, it finishes its writes. No write path checks that the lock is still held, and there is no fencing token. The stalled node is not told it lost the lock; it holds it in memory until it releases it. Two nodes then write the same metadata, which can double-allocate blocks or lose directory entries.

Expand All @@ -253,9 +253,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no

**Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves.

**Read locks are not renewed.** A node whose overlapping lookups keep a directory read-locked for more than 15 seconds can lose that lock to another node's writer. Its readers may then see a half-written directory. Two writers still cannot overlap.

**Locks share records.** The lock region has 256 records, and every lock maps to one of them, so unrelated files and the block bitmap can share a record. A node holding one of those locks blocks other nodes from all of them. If two nodes each hold one record and wait for the other's, both wait until one record is 15 seconds old; the takeover then lets both proceed, with the same risk as a stall. This needs nested locks whose records cross, roughly 1 in 65,000 pairs of overlapping operations.
**Locks share records.** The lock region has 256 records, and every lock maps to one of them, so unrelated files and the block bitmap can share a record. A node holding one of those locks blocks other nodes from all of them. If two nodes each hold one record and wait for the other's, neither claim goes stale while both nodes run. The first of the two waits gives up after 30 seconds and its operation fails with `EIO`; the other then proceeds. This needs nested locks whose records cross, roughly 1 in 65,000 pairs of overlapping operations.

**Safe operating limits:**

Expand Down
1 change: 1 addition & 0 deletions internal/goclusterfs/fs.go
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ func (f *FS) Close() error {
}
// Other nodes would otherwise wait for these holds to go stale.
if f.lockMgr != nil {
f.lockMgr.Stop()
f.lockMgr.ReleaseCached()
f.lockMgr.LogStats()
}
Expand Down
8 changes: 7 additions & 1 deletion internal/lock/lock.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ type Manager struct {
tornCount atomic.Int32
// slotWrite, when a test sets it, replaces the device write of a slot.
slotWrite func(buf []byte, offset int64) (int, error)
// stopRenew and renewDone run the renewer between Start and Stop.
stopRenew, renewDone chan struct{}

// Callback for cache invalidation
onAcquire OnAcquireFunc
Expand Down Expand Up @@ -182,6 +184,7 @@ type heldLock struct {

type heldRecord struct {
mode LockMode
key lockKey // the key this node's slot names
keys int
since time.Time // when this node claimed the record
renewed time.Time // when this node last wrote its slot
Expand All @@ -207,6 +210,9 @@ func covers(held, want LockMode) bool {
// relying on it, so other nodes never see a slot in use as stale.
var renewAfter = StaleThreshold / 3

// staleAfter is StaleThreshold; tests shorten it.
var staleAfter = StaleThreshold

// acquireTimeout is how long Acquire waits at most; tests shorten it.
var acquireTimeout = DefaultTimeout

Expand Down Expand Up @@ -504,7 +510,7 @@ func (m *Manager) attemptAcquire(key lockKey, recordID uint64, mode LockMode, gi
m.countAcquire(key.Type, func(c *acquireCounts) *atomic.Uint64 { return &c.claimed })

m.mu.Lock()
m.records[recordID] = &heldRecord{mode: mode, keys: 1, since: time.Now(), renewed: written}
m.records[recordID] = &heldRecord{mode: mode, key: key, keys: 1, since: time.Now(), renewed: written}
m.addHeldKeyLocked(key, recordID, mode, gid)
m.mu.Unlock()
return true, true, nil
Expand Down
5 changes: 3 additions & 2 deletions internal/lock/reclaim.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,14 @@ func (m *Manager) adoptRecord(recordID uint64, owner [16]byte, ownerSlot int) (b
case err != nil:
dead = &claimSlot{owner: owner}
}
now := time.Now()
dead.mode, dead.adopted = LockModeExclusive, true
dead.acquireTime = time.Now().UnixNano()
dead.acquireTime = now.UnixNano()
if err := m.writeSlot(recordID, ownerSlot, dead); err != nil {
return false, err
}
m.mu.Lock()
m.adopted[recordID] = adoptedSlot{slot: ownerSlot, owner: owner}
m.adopted[recordID] = adoptedSlot{slot: ownerSlot, owner: owner, renewed: now}
m.mu.Unlock()
return true, nil
}
Expand Down
137 changes: 137 additions & 0 deletions internal/lock/renew.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
package lock

import (
"log"
"time"

"github.com/supporttools/GoClusterFS/internal/storage"
)

// Start runs a renewer that rewrites this node's held slots and the slots it
// adopted for a replay, so peers never see them as stale however long an
// operation or a replay runs. Stop ends it.
func (m *Manager) Start() {
m.mu.Lock()
defer m.mu.Unlock()
if m.stopRenew != nil {
return
}
m.stopRenew, m.renewDone = make(chan struct{}), make(chan struct{})
go m.renewLoop(m.stopRenew, m.renewDone)
}

// Stop ends the renewer and waits for it to finish.
func (m *Manager) Stop() {
m.mu.Lock()
stop, done := m.stopRenew, m.renewDone
m.stopRenew, m.renewDone = nil, nil
m.mu.Unlock()
if stop != nil {
close(stop)
<-done
}
}

func (m *Manager) renewLoop(stop <-chan struct{}, done chan<- struct{}) {
defer close(done)
tick := time.NewTicker(renewAfter / 4)
defer tick.Stop()
for {
select {
case <-stop:
return
case <-tick.C:
m.renewDue()
}
}
}

// renewDue rewrites the slots this node wrote more than renewAfter/2 ago, so
// they stay younger than renewAfter, which also spares joins an inline renewal.
func (m *Manager) renewDue() {
m.mu.Lock()
var held, adopted []uint64
for recordID, rec := range m.records {
if age(rec.renewed) > renewAfter/2 {
held = append(held, recordID)
}
}
for recordID, a := range m.adopted {
if age(a.renewed) > renewAfter/2 {
adopted = append(adopted, recordID)
}
}
m.mu.Unlock()

for _, recordID := range held {
if err := m.renewHeld(recordID); err != nil {
log.Printf("[WARN] lock: renewing record %d: %v", recordID, err)
}
}
for _, recordID := range adopted {
if err := m.renewAdopted(recordID); err != nil {
log.Printf("[WARN] lock: renewing adopted record %d: %v", recordID, err)
}
}
}

func (m *Manager) renewHeld(recordID uint64) error {
gate := m.recordGate(recordID)
gate.Lock()
defer gate.Unlock()

m.mu.Lock()
rec, ok := m.records[recordID]
due := ok && age(rec.renewed) > renewAfter/2
m.mu.Unlock()
if !due {
return nil
}
return m.renewRecord(recordID, rec.key, rec)
}

// renewAdopted rewrites a slot AdoptOwner kept, unless it no longer holds that
// adoption. An adoption in this node's own slot that a hold or claim uses is
// renewed with that hold or claim.
func (m *Manager) renewAdopted(recordID uint64) error {
gate := m.recordGate(recordID)
gate.Lock()
defer gate.Unlock()

m.mu.Lock()
a, ok := m.adopted[recordID]
m.mu.Unlock()
if !ok || age(a.renewed) <= renewAfter/2 {
return nil
}
now := time.Now()
if a.slot == m.nodeSlot {
if _, inUse := m.ownSlotUse(recordID); inUse {
return nil
}
if err := m.writeSlot(recordID, a.slot, m.ownClaim(recordID, lockKey{}, LockModeExclusive, now)); err != nil {
return err
}
} else {
buf := storage.GetBuffer(SlotSize)
defer storage.PutBuffer(buf)
if _, err := m.dev.ReadAtUnsharded(buf.Bytes(), m.slotOffset(recordID, a.slot)); err != nil {
return err
}
s, err := decodeSlot(buf.Bytes())
if err != nil || s == nil || !s.adopted || s.owner != a.owner {
return nil
}
s.acquireTime = now.UnixNano()
if err := m.writeSlot(recordID, a.slot, s); err != nil {
return err
}
}
m.mu.Lock()
if cur, ok := m.adopted[recordID]; ok {
cur.renewed = now
m.adopted[recordID] = cur
}
m.mu.Unlock()
return nil
}
57 changes: 57 additions & 0 deletions internal/lock/renew_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package lock

import (
"testing"
"time"
)

func shortenStaleness(t *testing.T) {
t.Helper()
oldStale, oldRenew := staleAfter, renewAfter
t.Cleanup(func() { staleAfter, renewAfter = oldStale, oldRenew })
staleAfter, renewAfter = 300*time.Millisecond, 100*time.Millisecond
}

func TestLongHoldStaysFreshWithoutNewAcquires(t *testing.T) {
shortenStaleness(t)
dev, cleanup := createTestDevice(t, 1<<20)
defer cleanup()
holder := newTestNode(dev, 0, 1<<20, 1)
peer := newTestNode(dev, 0, 1<<20, 2)
holder.Start()
defer holder.Stop()

requireAcquired(t, holder, LockTypeInode, 2, LockModeExclusive)
time.Sleep(3 * staleAfter)
requireBlocked(t, peer, LockTypeInode, 2, LockModeExclusive)

if err := holder.Release(LockTypeInode, 2); err != nil {
t.Fatal(err)
}
holder.ReleaseCached()
requireAcquired(t, peer, LockTypeInode, 2, LockModeExclusive)
}

func TestAdoptedSlotsStayFreshUntilReleased(t *testing.T) {
shortenStaleness(t)
dev, cleanup := createTestDevice(t, reclaimRegion)
defer cleanup()
dead := NewManager(dev, 0, reclaimRegion, [16]byte{1}, 1)
replayer := NewManager(dev, 0, reclaimRegion, [16]byte{3}, 3)
peer := NewManager(dev, 0, reclaimRegion, [16]byte{4}, 4)
replayer.Start()
defer replayer.Stop()

requireAcquired(t, dead, LockTypeInode, 2, LockModeExclusive)
adopted, err := replayer.AdoptOwner([16]byte{1}, 1)
if err != nil || len(adopted) != 1 {
t.Fatalf("AdoptOwner = %v, %v; want the dead node's record", adopted, err)
}
time.Sleep(3 * staleAfter)
requireBlocked(t, peer, LockTypeInode, 2, LockModeExclusive)

if err := replayer.ReleaseAdopted(adopted); err != nil {
t.Fatal(err)
}
requireAcquired(t, peer, LockTypeInode, 2, LockModeExclusive)
}
7 changes: 4 additions & 3 deletions internal/lock/slots.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@ type claimSlot struct {
}

type adoptedSlot struct {
slot int
owner [16]byte
slot int
owner [16]byte
renewed time.Time
}

type slotRef struct {
Expand Down Expand Up @@ -105,7 +106,7 @@ func isZeroSlot(buf []byte) bool {
}

func (s *claimSlot) stale() bool {
return time.Since(time.Unix(0, s.acquireTime)) > StaleThreshold
return time.Since(time.Unix(0, s.acquireTime)) > staleAfter
}

func rowSlot(row []byte, slot int) []byte {
Expand Down
14 changes: 14 additions & 0 deletions tests/test-manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2760,6 +2760,20 @@ scenarios:
environments:
go: internal/goclusterfs/lock_commit_test.go::TestWriteWhenInodeAndBlockShareLockRecord

- id: lock-long-hold-stays-fresh
name: Long lock hold stays fresh
description: A node holding a lock longer than the stale threshold, with no new acquires, keeps other nodes out because a background renewer rewrites its claim slot
priority: go_only
environments:
go: internal/lock/renew_test.go::TestLongHoldStaysFreshWithoutNewAcquires

- id: lock-adopted-slots-stay-fresh
name: Adopted lock slots stay fresh during a long replay
description: Claims a replaying node took over from a dead node keep other nodes out past the stale threshold until ReleaseAdopted
priority: go_only
environments:
go: internal/lock/renew_test.go::TestAdoptedSlotsStayFreshUntilReleased

- id: lock-leaked-shared-count-goes-stale
name: Leaked read claim goes stale
description: New readers do not refresh another node's read claim, so a read claim leaked by a stopped node stops blocking writers once it is 15 seconds old
Expand Down
Loading