From a4c6c71dd39f3b6cd52efccdd8703045ec73d00f Mon Sep 17 00:00:00 2001 From: Matthew Mattox Date: Sat, 14 Nov 2026 00:27:05 -0600 Subject: [PATCH] fix(lock): a node that loses to a lower slot leaves a waiting mark so holders give it a turn A node in a higher slot that lost a claim cleared its slot and only polled. A lower slot that released a lock and claimed it again right after its cached hold expired never saw the waiter, and its turn restarted with every claim, so the higher slot could wait until the 30 s timeout. The loser now writes a waiting mark, which conflicts with no claim. A holder that wins a claim and sees a live node's mark ends its turn 250 ms after it first saw one, even across fresh claims, then yields for YieldGap. Marks count only while their node is alive in the registry (or, without a registry, until they are stale), and the waiter clears its mark when its Acquire ends. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 + docs/OPERATIONS.md | 4 +- internal/lock/fairness_test.go | 116 +++++++++++++++++++++++++++++++++ internal/lock/lock.go | 52 +++++++++++---- internal/lock/slots.go | 96 ++++++++++++++++++++++----- tests/test-manifest.yaml | 14 ++++ 6 files changed, 255 insertions(+), 29 deletions(-) create mode 100644 internal/lock/fairness_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 1136020b..c587a06e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - Older binaries reset an inode's generation counters to 0 when they free it, so other nodes can serve stale data after the inode number is reused. - Older binaries zero the extent-tree generation (inode bytes 188–195) when they rewrite an inode. Newer binaries then cannot read that file if it has more than two extents, because such files use an extent tree. - Directory entries now fill all 4 KB chunks of a directory block. When an older binary removes the first entry of a later chunk, it grows the last entry of the chunk before it across the boundary. This version then stops listing that entry, and fsck reports the directory block. + - Older binaries read a newer node's waiting mark in a lock record as a held lock. - Run `goclusterfs-fsck -repair` before the first mount after upgrading an image that has snapshots or clones. Unmount the device on every node first. - Older versions marked a file's blocks shared, on snapshot or clone, without counting the owners. Until `goclusterfs-fsck -repair` counts them, this version never frees such blocks, so deleting those files leaks their space, and it refuses to snapshot or clone them (`ENOTSUP`). In-place writes to an old clone still change its source. - Snapshots now record the birth generation of the file they were taken of, so a later file that reuses the inode number cannot list, clone or delete them. Snapshots older versions made do not record it, and some captured no data, so after the upgrade they are orphans: `getfattr -n user.snapshots` no longer lists them, `GET /api/v1/snapshots` lists them with `orphan: true`, and `DELETE /api/v1/snapshots/` without `path` deletes them. Older binaries keep the birth generation in inode bytes 196–203 only while they do not rewrite the inode, so upgrade all nodes together. @@ -76,6 +77,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed +- A node in a higher registry slot could wait 30 seconds and fail with `EIO` for a lock that a node in a lower slot kept releasing and claiming again every few milliseconds. A node that loses a claim to a lower slot now leaves a waiting mark in its own slot, and the holder gives the lock up 250 ms after it first sees a live node's mark. - `chmod`, `chown` and `touch` could undo another node's concurrent change to the same inode: on a directory, a create's link count or mtime; on a file, the size an append inside an allocated block set. Directory attribute changes now hold the inode lock from read to write, and file attribute changes retry when the inode changed since they read it. - After 3 failed multicast sends in total, counted over the whole mount, a node sent cache invalidations only to unicast peers until it unmounted. Now only failures in a row count: a successful send resets them, and after 3 in a row the node tries multicast again 30 seconds later. - `chmod` cleared a file's type bits, so the file could no longer be opened ("Is a directory") and fsck reported an invalid file type. `chmod`, `chown` and `touch` now also work on directories and the mount root, which returned "Operation not supported". diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 6796ca6a..d096a102 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -267,7 +267,7 @@ Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts, **How a node takes a lock.** Each of the 256 lock records has one 4 KB claim slot per registry slot, and a node writes only its own slot. To take a lock, a node writes its claim into its slot, then reads all 64 slots of the record in one read. It holds the lock if no other slot conflicts: for a write lock, any other claim; for a read lock, another node's write claim. Of two nodes that claim at the same time, the one whose write finished second reads the record after the other's write finished, so it sees that claim and does not take the lock. This holds however slow the writes are. It relies on the shared device showing a finished write to every later read, from any node. -**When claims collide.** The node in the lower registry slot keeps its claim and waits for the other claims to go. A node that sees a claim in a lower slot clears its own and waits, reading the record without writing until that claim is gone. Waits only point from lower to higher slots, so two nodes never wait on each other. A claim kept for a journal replay counts as lower than every slot, so the replaying node is never stuck behind a waiter. When two nodes replay dead nodes whose claims share a record, the replay of the lower dead slot goes first. A slot that fails its checksum, such as one torn by a crash in the middle of a write, counts as a claim. +**When claims collide.** The node in the lower registry slot keeps its claim and waits for the other claims to go. A node that sees a claim in a lower slot replaces its own with a waiting mark and waits, reading the record without writing until that claim is gone. A mark blocks no one; it only tells holders that the node is waiting. Waits only point from lower to higher slots, so two nodes never wait on each other. A claim kept for a journal replay counts as lower than every slot, so the replaying node is never stuck behind a waiter. When two nodes replay dead nodes whose claims share a record, the replay of the lower dead slot goes first. A slot that fails its checksum, such as one torn by a crash in the middle of a write, counts as a claim. **When another node's claim stops counting.** A claim counts until the registry slot with the same number is empty: its node unmounted cleanly, or another node replayed its journal and freed the slot. A node checks this by reading the registry slot from the device after it reads the record, so it cannot miss a node that registered in the meantime. A torn slot follows the same rule. After its replay, a mount clears the claims in its own slot that an earlier occupant left. How old a claim is does not matter, so a lock that a node never releases, for example after a bug, stays held until that node unmounts. The node logs a warning for each lock it has held for more than 60 seconds. A replay that fails or gets stuck keeps the dead node's locks held until a later replay succeeds; waits for them fail with `EIO` once its recovery lease lapses, about a minute after the replay last made progress. A mount whose own journal replay gives up waiting for another node's lock retries every 1 to 30 seconds and logs who holds the lock each time. Any other replay failure, such as a record that does not decode or an I/O error, fails the mount at once with the `-skip-recovery` hint. @@ -287,7 +287,7 @@ Common causes of a stall: Lock slot reads and writes do not take the device's per-region I/O lock, so a node's own data and journal writes to the same region cannot delay them. The lock manager serializes each record's I/O itself. A release whose write fails leaves the claim in place until the node unmounts or is replayed. -**Nodes keep released locks briefly.** After its last user releases a lock, a node keeps it for 10 ms (`CacheGrace`), so back-to-back work on the same node does not write and read the record again. A node reuses a lock it claimed for at most 250 ms (`HoldTurn`). After that, new work on that node waits up to another 250 ms for the current holders to finish. The node then releases the lock and waits 50 ms (`YieldGap`) before taking it again, so another node gets a turn. Another node that wants the lock waits up to about 35 ms while the holder is idle, and up to about 300 ms while the holder keeps using it. A clean unmount releases kept locks at once. A crashed node's last locks, kept or held, stay held until another node proves it dead and replays its journal (see [Node Requirements](#node-requirements)). A lock wait gives up after 30 seconds (`DefaultTimeout`) while the holder is alive. A wait for a failed holder has no time limit: it logs a warning that names the node, and it ends when the replay does. It fails with `EIO` only if the replay fails or is abandoned, which shows as a lapsed recovery lease, with one exception. An operation that already holds a lock and then waits for a failed node's lock keeps the 30-second limit and fails with `EIO`, because the replay may need the lock it holds. An overwrite of a file's existing data does not: it lets go of its locks at once, waits for the block's lock holding none, and retries, so it does not fail. Operations that already changed something first, such as a create or a write that allocated blocks, keep the 30-second limit. For example, a node holds a directory's lock and waits for a block lock the dead node held, while the dead node's journal changes that directory. The replay's own waits have no limit, so it finishes once the operation gives up. The warning for this case names the record and the dead node. +**Nodes keep released locks briefly.** After its last user releases a lock, a node keeps it for 10 ms (`CacheGrace`), so back-to-back work on the same node does not write and read the record again. A node reuses a lock it claimed for at most 250 ms (`HoldTurn`). After that, new work on that node waits up to another 250 ms for the current holders to finish. The node then releases the lock and waits 50 ms (`YieldGap`) before taking it again, so another node gets a turn. A node that claims a lock and finds another node's waiting mark gives it up the same way 250 ms after it first saw the mark, even if it released and claimed the lock again in between. Without this, a node in a lower slot that keeps taking a lock back every few milliseconds would shut out a waiting node in a higher slot. A mark counts only while its node is alive in the registry. Another node that wants the lock waits up to about 35 ms while the holder is idle, and up to about 300 ms while the holder keeps using it. A clean unmount releases kept locks at once. A crashed node's last locks, kept or held, stay held until another node proves it dead and replays its journal (see [Node Requirements](#node-requirements)). A lock wait gives up after 30 seconds (`DefaultTimeout`) while the holder is alive. A wait for a failed holder has no time limit: it logs a warning that names the node, and it ends when the replay does. It fails with `EIO` only if the replay fails or is abandoned, which shows as a lapsed recovery lease, with one exception. An operation that already holds a lock and then waits for a failed node's lock keeps the 30-second limit and fails with `EIO`, because the replay may need the lock it holds. An overwrite of a file's existing data does not: it lets go of its locks at once, waits for the block's lock holding none, and retries, so it does not fail. Operations that already changed something first, such as a create or a write that allocated blocks, keep the 30-second limit. For example, a node holds a directory's lock and waits for a block lock the dead node held, while the dead node's journal changes that directory. The replay's own waits have no limit, so it finishes once the operation gives up. The warning for this case names the record and the dead node. **Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves. diff --git a/internal/lock/fairness_test.go b/internal/lock/fairness_test.go new file mode 100644 index 00000000..e0b7038c --- /dev/null +++ b/internal/lock/fairness_test.go @@ -0,0 +1,116 @@ +package lock + +import ( + "slices" + "sync" + "testing" + "time" +) + +// reclaimAfterExpiry holds the lock briefly, releases it, and claims it again as +// soon as m's cached hold is gone from disk, until stop closes. +func reclaimAfterExpiry(t *testing.T, m *Manager, lt LockType, id uint64, stop <-chan struct{}) { + recordID := m.hashToRecord(lt, id) + for { + select { + case <-stop: + return + default: + } + if err := acquireWithin(m, 5*time.Second, lt, id, LockModeExclusive); err != nil { + t.Errorf("node %d Acquire: %v", m.nodeSlot, err) + return + } + time.Sleep(time.Millisecond) + release(t, m, lt, id) + for { + m.mu.Lock() + _, kept := m.records[recordID] + m.mu.Unlock() + if !kept { + break + } + time.Sleep(100 * time.Microsecond) + } + } +} + +func TestHigherSlotGetsLockFromLowerSlotThatKeepsReclaimingIt(t *testing.T) { + dev, cleanup := createTestDevice(t, 1<<20) + defer cleanup() + low := newTestNode(dev, 0, 1<<20, 1) + high := newTestNode(dev, 0, 1<<20, 2) + + stop := make(chan struct{}) + var wg sync.WaitGroup + wg.Go(func() { reclaimAfterExpiry(t, low, LockTypeInode, 3, stop) }) + defer func() { close(stop); wg.Wait() }() + time.Sleep(20 * time.Millisecond) + + var waits []time.Duration + for range 10 { + start := time.Now() + if err := acquireWithin(high, 5*time.Second, LockTypeInode, 3, LockModeExclusive); err != nil { + t.Fatalf("higher slot never got the lock: %v", err) + } + waits = append(waits, time.Since(start)) + release(t, high, LockTypeInode, 3) + time.Sleep(20 * time.Millisecond) + } + if longest := slices.Max(waits); longest > 2*time.Second { + t.Errorf("higher slot waited up to %v, want under 2s: %v", longest, waits) + } +} + +func TestHolderYieldsOnlyToLiveWaiters(t *testing.T) { + cases := []struct { + name string + view bool + state OwnerState + owner byte + age time.Duration + yields bool + }{ + {name: "live waiter", view: true, state: OwnerAlive, owner: 5, yields: true}, + {name: "failing waiter", view: true, state: OwnerFailing, owner: 5}, + {name: "replayed waiter", view: true, state: OwnerReleased, owner: 5}, + {name: "mark of the slot's earlier node", view: true, state: OwnerAlive, owner: 9}, + {name: "fresh mark without registry", yields: true}, + {name: "stale mark without registry", age: pastStale}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + defer func(d time.Duration) { holdTurn = d }(holdTurn) + holdTurn = 50 * time.Millisecond + dev, cleanup := createTestDevice(t, 1<<20) + defer cleanup() + holder := newTestNode(dev, 0, 1<<20, 1) + if tc.view { + reg := newFakeRegistry() + reg.set(5, tc.state) + holder.SetOwnerView(reg.view) + } + mark := &claimSlot{mode: modeWaiting, lockType: LockTypeInode, resourceID: 3, owner: [16]byte{tc.owner}, acquireTime: time.Now().Add(-tc.age).UnixNano()} + if err := holder.writeSlot(holder.hashToRecord(LockTypeInode, 3), 5, mark); err != nil { + t.Fatal(err) + } + + stop := make(chan struct{}) + var wg sync.WaitGroup + wg.Go(func() { reclaimAfterExpiry(t, holder, LockTypeInode, 3, stop) }) + time.Sleep(4 * holdTurn) + close(stop) + wg.Wait() + + var blocked uint64 + for _, ts := range holder.Stats().Types { + if ts.Name == "inode" { + blocked = ts.Blocked + } + } + if yielded := blocked > 0; yielded != tc.yields { + t.Errorf("holder yielded: %v (%d blocked acquires), want %v", yielded, blocked, tc.yields) + } + }) + } +} diff --git a/internal/lock/lock.go b/internal/lock/lock.go index c0efca20..c724e33a 100644 --- a/internal/lock/lock.go +++ b/internal/lock/lock.go @@ -116,6 +116,10 @@ const ( LockModeNone LockMode = 0 LockModeShared LockMode = 1 // Multiple readers allowed LockModeExclusive LockMode = 2 // Single writer, no readers + + // modeWaiting marks the slot of a node that lost a claim to a lower slot and + // still wants the record. It conflicts with no claim. + modeWaiting LockMode = 3 ) // OnAcquireFunc is called when a lock is successfully acquired. @@ -141,6 +145,9 @@ type Manager struct { gates map[uint64]*sync.Mutex // yieldUntil holds records this node gave up and will not claim again before then. yieldUntil map[uint64]time.Time + // contested holds when this node, in its current run of claims on a record, + // first saw another node's waiting mark there. + contested map[uint64]time.Time // claims tracks the Acquire calls waiting for a record this node does not hold. claims map[uint64]*pendingClaim // adopted maps each record AdoptOwner kept for a dead node to that node's slot. @@ -226,12 +233,14 @@ type heldRecord struct { } // pendingClaim counts the local Acquire calls waiting for a record. When -// written is set, this node's slot holds a claim in mode that it has not won yet. -// behindLower means the last claim lost to a lower slot. +// written is set, this node's slot holds a claim in mode that it has not won yet; +// when marked is set, it holds a waiting mark. behindLower means the last claim +// lost to a lower slot. type pendingClaim struct { waiters int mode LockMode written time.Time + marked time.Time behindLower bool } @@ -285,6 +294,7 @@ func NewManager(dev *storage.Device, offset, size uint64, nodeUUID [16]byte, nod records: make(map[uint64]*heldRecord), gates: make(map[uint64]*sync.Mutex), yieldUntil: make(map[uint64]time.Time), + contested: make(map[uint64]time.Time), claims: make(map[uint64]*pendingClaim), adopted: make(map[uint64]adoptedSlot), tornSince: make(map[slotRef]tornSighting), @@ -400,7 +410,7 @@ func (m *Manager) recordClaimTime(d time.Duration) { // LogStats logs how this node's acquires were served, how often claims // collided, and how long claims took. func (m *Manager) LogStats() { - log.Printf("lock: %d slot writes, %d row reads; %d claims cleared after a collision, %d kept to wait, %d tries waited for a lower slot", + log.Printf("lock: %d slot writes, %d row reads; %d claims lost to a lower slot, %d kept to wait, %d tries waited for a lower slot", m.slotWrites.Load(), m.rowReads.Load(), m.conflicts.Load(), m.waits.Load(), m.deferred.Load()) for typ := range m.acquires { c := &m.acquires[typ] @@ -582,14 +592,13 @@ func (m *Manager) attemptAcquire(key lockKey, recordID uint64, mode LockMode, gi } return false, false, blocker{}, nil } - if age(rec.since) > holdTurn && !m.goroutineHoldsRecordLocked(recordID, gid) && + if m.turnOverLocked(recordID, rec) && !m.goroutineHoldsRecordLocked(recordID, gid) && (waited < holdTurn || !m.goroutineHoldsAnyLocked(gid)) { if rec.keys > 0 { m.mu.Unlock() return false, false, blocker{}, nil } - delete(m.records, recordID) - m.yieldUntil[recordID] = time.Now().Add(yieldGap) + m.endTurnLocked(recordID) m.mu.Unlock() return false, false, blocker{}, m.releaseRecord(recordID) } @@ -648,6 +657,23 @@ func (m *Manager) giveUp(err error, lockType LockType, resourceID uint64, mode L return fmt.Errorf("%w: %s", err, reason) } +// turnOverLocked reports whether this node has had its turn on recordID: it has +// held the record for holdTurn, or kept claiming it again for holdTurn since it +// saw another node waiting. +func (m *Manager) turnOverLocked(recordID uint64, rec *heldRecord) bool { + seen, waited := m.contested[recordID] + return age(rec.since) > holdTurn || waited && age(seen) > holdTurn +} + +// endTurnLocked drops this node's hold on recordID, which the caller releases +// on disk, and keeps this node from claiming it again for yieldGap so a waiting +// node gets it. +func (m *Manager) endTurnLocked(recordID uint64) { + delete(m.records, recordID) + delete(m.contested, recordID) + m.yieldUntil[recordID] = time.Now().Add(yieldGap) +} + func (m *Manager) reuseHeldKeyLocked(key lockKey, mode LockMode, gid uint64) bool { held, ok := m.heldLocks[key] if !ok { @@ -657,7 +683,7 @@ func (m *Manager) reuseHeldKeyLocked(key lockKey, mode LockMode, gid uint64) boo if !covers(rec.mode, mode) || age(rec.renewed) > renewAfter { return false } - if age(rec.since) > holdTurn && !m.goroutineHoldsRecordLocked(held.RecordID, gid) { + if m.turnOverLocked(held.RecordID, rec) && !m.goroutineHoldsRecordLocked(held.RecordID, gid) { return false } held.refCount++ @@ -757,10 +783,8 @@ func (m *Manager) dropRecordKey(recordID uint64) error { m.mu.Unlock() return nil } - if age(rec.since) > holdTurn { - // Held long enough: give other nodes a turn instead of keeping it. - delete(m.records, recordID) - m.yieldUntil[recordID] = time.Now().Add(yieldGap) + if m.turnOverLocked(recordID, rec) { + m.endTurnLocked(recordID) m.mu.Unlock() return m.releaseRecord(recordID) } @@ -784,7 +808,11 @@ func (m *Manager) expireCached(recordID uint64, gen uint64) { m.mu.Unlock() return } - delete(m.records, recordID) + if m.turnOverLocked(recordID, rec) { + m.endTurnLocked(recordID) + } else { + delete(m.records, recordID) + } m.mu.Unlock() if err := m.releaseRecord(recordID); err != nil { diff --git a/internal/lock/slots.go b/internal/lock/slots.go index b7dbf030..2d2a1607 100644 --- a/internal/lock/slots.go +++ b/internal/lock/slots.go @@ -20,11 +20,15 @@ import ( // This holds whatever the I/O delays. // // When claims collide, the claimant in the lower slot keeps its slot and waits; -// any claimant that sees a conflict in a lower slot clears its own and retries. -// Waits only point from lower to higher slots, so they never form a cycle. A -// claimant that cleared its slot reads the row before writing again, and waits -// without writing while the lower slot is still there, so it does not get in -// the way of that slot's claim. +// any claimant that sees a conflict in a lower slot replaces its claim with a +// waiting mark and retries. A mark conflicts with no claim, so waits only point +// from lower to higher slots and never form a cycle. A claimant that lost reads +// the row before writing again, and waits without writing while the lower slot +// is still there, so it does not get in the way of that slot's claim. +// +// A node that wins a claim and sees a live node's mark ends its turn holdTurn +// after it first saw one, even if it keeps claiming the record again, so a +// higher slot gets the record while a lower one keeps using it. const ( SlotsPerRecord = 64 SlotSize = storage.SectorSize @@ -165,7 +169,7 @@ func (m *Manager) rowConflicts(recordID uint64, row []byte, mode LockMode, ignor torn = rowSlot(row, slot) } tornTooLong := m.tornTooLong(slotRef{recordID, slot}, torn) - if err == nil && (s == nil || (mode != LockModeExclusive && s.mode != LockModeExclusive)) { + if err == nil && (s == nil || s.mode == modeWaiting || (mode != LockModeExclusive && s.mode != LockModeExclusive)) { continue } adopted := s != nil && s.adopted @@ -193,6 +197,49 @@ func (m *Manager) rowConflicts(recordID uint64, row []byte, mode LockMode, ignor return conflict, lower, by } +// othersWaiting reports whether row holds a waiting mark of a live node other +// than this one. With an owner view a mark counts while its writer holds its +// registry slot alive; without one, until it is older than StaleThreshold. +func (m *Manager) othersWaiting(row []byte) bool { + m.mu.Lock() + view := m.ownerView + m.mu.Unlock() + for slot := range SlotsPerRecord { + if slot == m.nodeSlot { + continue + } + s, err := decodeSlot(rowSlot(row, slot)) + if err != nil || s == nil || s.mode != modeWaiting { + continue + } + if view == nil { + if !s.stale() { + return true + } + continue + } + if occupant, state := view(slot, false); state == OwnerAlive && occupant == s.owner { + return true + } + } + return false +} + +// markWaiting replaces this node's claim on recordID with a waiting mark and +// returns when it wrote it, or keeps an adoption there as releaseRecord does. +// It runs under the record gate. +func (m *Manager) markWaiting(recordID uint64, key lockKey) (time.Time, error) { + m.mu.Lock() + adopted := m.ownAdoptedLocked(recordID) + m.mu.Unlock() + if adopted { + return time.Time{}, m.releaseRecord(recordID) + } + now := time.Now() + mark := &claimSlot{mode: modeWaiting, lockType: key.Type, resourceID: key.ResourceID, owner: m.nodeUUID, acquireTime: now.UnixNano()} + return now, m.writeSlot(recordID, m.nodeSlot, mark) +} + // tornTooLong reports whether ref has read torn, with the same bytes, for // tornExpiry; torn is nil when the slot reads whole. Changed bytes mean a node // is writing the slot, so they start the wait again. @@ -249,7 +296,7 @@ func (m *Manager) addClaimWaiterLocked(recordID uint64) { } // dropClaimWaiter ends one Acquire call's wait. The last waiter to leave -// clears a slot this node wrote for the record but never won. +// clears a claim or mark this node wrote for the record but never won. func (m *Manager) dropClaimWaiter(recordID uint64) { m.mu.Lock() c := m.claims[recordID] @@ -258,7 +305,7 @@ func (m *Manager) dropClaimWaiter(recordID uint64) { m.mu.Unlock() return } - if c.written.IsZero() { + if c.written.IsZero() && c.marked.IsZero() { delete(m.claims, recordID) m.mu.Unlock() return @@ -276,7 +323,7 @@ func (m *Manager) dropClaimWaiter(recordID uint64) { } delete(m.claims, recordID) m.mu.Unlock() - if !c.written.IsZero() { + if !c.written.IsZero() || !c.marked.IsZero() { _ = m.releaseRecord(recordID) } } @@ -292,7 +339,7 @@ func (m *Manager) claim(recordID uint64, key lockKey, mode LockMode) (written ti if a, ok := m.adopted[recordID]; ok { ignore = a.slot } - behindLower := c.behindLower + behindLower, marked := c.behindLower, c.marked m.mu.Unlock() if behindLower { @@ -302,6 +349,15 @@ func (m *Manager) claim(recordID uint64, key lockKey, mode LockMode) (written ti } if _, lower, by := m.rowConflicts(recordID, row, mode, ignore); lower { m.deferred.Add(1) + if !marked.IsZero() && age(marked) > renewAfter { + at, err := m.markWaiting(recordID, key) + m.mu.Lock() + c.marked = at + m.mu.Unlock() + if err != nil { + return time.Time{}, false, by, err + } + } return time.Time{}, false, by, nil } m.mu.Lock() @@ -313,7 +369,7 @@ func (m *Manager) claim(recordID uint64, key lockKey, mode LockMode) (written ti if rewrite { err := m.writeSlot(recordID, m.nodeSlot, m.ownClaim(recordID, key, mode, start)) m.mu.Lock() - c.written, c.mode = start, mode + c.written, c.mode, c.marked = start, mode, time.Time{} m.mu.Unlock() if err != nil { return time.Time{}, false, blocker{}, err @@ -330,19 +386,26 @@ func (m *Manager) claim(recordID uint64, key lockKey, mode LockMode) (written ti conflict, lower, by := m.rowConflicts(recordID, row, mode, ignore) switch { case !conflict: + waited := m.othersWaiting(row) m.mu.Lock() written = c.written c.written = time.Time{} + if _, seen := m.contested[recordID]; waited && !seen { + m.contested[recordID] = time.Now() + } else if !waited { + delete(m.contested, recordID) + } m.mu.Unlock() return written, true, blocker{}, nil case lower: m.conflicts.Add(1) - if err := m.releaseRecord(recordID); err != nil { - return time.Time{}, false, by, err - } + at, err := m.markWaiting(recordID, key) m.mu.Lock() - c.written, c.behindLower = time.Time{}, true + c.written, c.marked, c.behindLower = time.Time{}, at, true m.mu.Unlock() + if err != nil { + return time.Time{}, false, by, err + } return time.Time{}, false, by, nil default: m.waits.Add(1) @@ -387,6 +450,9 @@ func (m *Manager) describeHolder(recordID uint64) string { switch { case err != nil: holders = append(holders, fmt.Sprintf("slot %d (torn)", slot)) + case s != nil && s.mode == modeWaiting: + holders = append(holders, fmt.Sprintf("slot %d node %x waiting, marked %v ago", + slot, s.owner[:8], time.Since(time.Unix(0, s.acquireTime)).Round(time.Millisecond))) case s != nil: adopted := "" if s.adopted { diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index 1ba1eed4..31752cf0 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -3509,6 +3509,20 @@ scenarios: environments: go: internal/lock/safety_test.go::TestBusyNodeYieldsCachedRecord + - id: lock-higher-slot-not-starved + name: Higher slot gets a lock a lower slot keeps claiming again + description: A lower slot releases a lock and claims it again as soon as its cached hold expires; a higher slot waiting for it still gets it within 2 seconds each time + priority: go_only + environments: + go: internal/lock/fairness_test.go::TestHigherSlotGetsLockFromLowerSlotThatKeepsReclaimingIt + + - id: lock-yield-only-to-live-waiters + name: Holder yields only to a live node's waiting mark + description: A holder ends its turn early for a waiting mark whose node is alive in the registry, or is fresh without a registry; it ignores marks of failing, replayed or replaced nodes and stale marks + priority: go_only + environments: + go: internal/lock/fairness_test.go::TestHolderYieldsOnlyToLiveWaiters + - id: lock-concurrent-dir-reads-create name: Concurrent directory reads do not delay create or unlink description: Two nodes taking a shared claim on one directory at once leave no reader count behind, so a create or unlink there finishes within 2s