Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- Older binaries reset an inode's generation counters to 0 when they free it, so other nodes can serve stale data after the inode number is reused.
- Older binaries zero the extent-tree generation (inode bytes 188–195) when they rewrite an inode. Newer binaries then cannot read that file if it has more than two extents, because such files use an extent tree.
- Directory entries now fill all 4 KB chunks of a directory block. When an older binary removes the first entry of a later chunk, it grows the last entry of the chunk before it across the boundary. This version then stops listing that entry, and fsck reports the directory block.
- Older binaries read a newer node's waiting mark in a lock record as a held lock.
- Run `goclusterfs-fsck -repair` before the first mount after upgrading an image that has snapshots or clones. Unmount the device on every node first.
- Older versions marked a file's blocks shared, on snapshot or clone, without counting the owners. Until `goclusterfs-fsck -repair` counts them, this version never frees such blocks, so deleting those files leaks their space, and it refuses to snapshot or clone them (`ENOTSUP`). In-place writes to an old clone still change its source.
- Snapshots now record the birth generation of the file they were taken of, so a later file that reuses the inode number cannot list, clone or delete them. Snapshots older versions made do not record it, and some captured no data, so after the upgrade they are orphans: `getfattr -n user.snapshots` no longer lists them, `GET /api/v1/snapshots` lists them with `orphan: true`, and `DELETE /api/v1/snapshots/<name>` without `path` deletes them. Older binaries keep the birth generation in inode bytes 196–203 only while they do not rewrite the inode, so upgrade all nodes together.
Expand Down Expand Up @@ -76,6 +77,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Fixed

- A node in a higher registry slot could wait 30 seconds and fail with `EIO` for a lock that a node in a lower slot kept releasing and claiming again every few milliseconds. A node that loses a claim to a lower slot now leaves a waiting mark in its own slot, and the holder gives the lock up 250 ms after it first sees a live node's mark.
- `chmod`, `chown` and `touch` could undo another node's concurrent change to the same inode: on a directory, a create's link count or mtime; on a file, the size an append inside an allocated block set. Directory attribute changes now hold the inode lock from read to write, and file attribute changes retry when the inode changed since they read it.
- After 3 failed multicast sends in total, counted over the whole mount, a node sent cache invalidations only to unicast peers until it unmounted. Now only failures in a row count: a successful send resets them, and after 3 in a row the node tries multicast again 30 seconds later.
- `chmod` cleared a file's type bits, so the file could no longer be opened ("Is a directory") and fsck reported an invalid file type. `chmod`, `chown` and `touch` now also work on directories and the mount root, which returned "Operation not supported".
Expand Down
4 changes: 2 additions & 2 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts,

**How a node takes a lock.** Each of the 256 lock records has one 4 KB claim slot per registry slot, and a node writes only its own slot. To take a lock, a node writes its claim into its slot, then reads all 64 slots of the record in one read. It holds the lock if no other slot conflicts: for a write lock, any other claim; for a read lock, another node's write claim. Of two nodes that claim at the same time, the one whose write finished second reads the record after the other's write finished, so it sees that claim and does not take the lock. This holds however slow the writes are. It relies on the shared device showing a finished write to every later read, from any node.

**When claims collide.** The node in the lower registry slot keeps its claim and waits for the other claims to go. A node that sees a claim in a lower slot clears its own and waits, reading the record without writing until that claim is gone. Waits only point from lower to higher slots, so two nodes never wait on each other. A claim kept for a journal replay counts as lower than every slot, so the replaying node is never stuck behind a waiter. When two nodes replay dead nodes whose claims share a record, the replay of the lower dead slot goes first. A slot that fails its checksum, such as one torn by a crash in the middle of a write, counts as a claim.
**When claims collide.** The node in the lower registry slot keeps its claim and waits for the other claims to go. A node that sees a claim in a lower slot replaces its own with a waiting mark and waits, reading the record without writing until that claim is gone. A mark blocks no one; it only tells holders that the node is waiting. Waits only point from lower to higher slots, so two nodes never wait on each other. A claim kept for a journal replay counts as lower than every slot, so the replaying node is never stuck behind a waiter. When two nodes replay dead nodes whose claims share a record, the replay of the lower dead slot goes first. A slot that fails its checksum, such as one torn by a crash in the middle of a write, counts as a claim.

**When another node's claim stops counting.** A claim counts until the registry slot with the same number is empty: its node unmounted cleanly, or another node replayed its journal and freed the slot. A node checks this by reading the registry slot from the device after it reads the record, so it cannot miss a node that registered in the meantime. A torn slot follows the same rule. After its replay, a mount clears the claims in its own slot that an earlier occupant left. How old a claim is does not matter, so a lock that a node never releases, for example after a bug, stays held until that node unmounts. The node logs a warning for each lock it has held for more than 60 seconds. A replay that fails or gets stuck keeps the dead node's locks held until a later replay succeeds; waits for them fail with `EIO` once its recovery lease lapses, about a minute after the replay last made progress. A mount whose own journal replay gives up waiting for another node's lock retries every 1 to 30 seconds and logs who holds the lock each time. Any other replay failure, such as a record that does not decode or an I/O error, fails the mount at once with the `-skip-recovery` hint.

Expand All @@ -287,7 +287,7 @@ Common causes of a stall:

Lock slot reads and writes do not take the device's per-region I/O lock, so a node's own data and journal writes to the same region cannot delay them. The lock manager serializes each record's I/O itself. A release whose write fails leaves the claim in place until the node unmounts or is replayed.

**Nodes keep released locks briefly.** After its last user releases a lock, a node keeps it for 10 ms (`CacheGrace`), so back-to-back work on the same node does not write and read the record again. A node reuses a lock it claimed for at most 250 ms (`HoldTurn`). After that, new work on that node waits up to another 250 ms for the current holders to finish. The node then releases the lock and waits 50 ms (`YieldGap`) before taking it again, so another node gets a turn. Another node that wants the lock waits up to about 35 ms while the holder is idle, and up to about 300 ms while the holder keeps using it. A clean unmount releases kept locks at once. A crashed node's last locks, kept or held, stay held until another node proves it dead and replays its journal (see [Node Requirements](#node-requirements)). A lock wait gives up after 30 seconds (`DefaultTimeout`) while the holder is alive. A wait for a failed holder has no time limit: it logs a warning that names the node, and it ends when the replay does. It fails with `EIO` only if the replay fails or is abandoned, which shows as a lapsed recovery lease, with one exception. An operation that already holds a lock and then waits for a failed node's lock keeps the 30-second limit and fails with `EIO`, because the replay may need the lock it holds. An overwrite of a file's existing data does not: it lets go of its locks at once, waits for the block's lock holding none, and retries, so it does not fail. Operations that already changed something first, such as a create or a write that allocated blocks, keep the 30-second limit. For example, a node holds a directory's lock and waits for a block lock the dead node held, while the dead node's journal changes that directory. The replay's own waits have no limit, so it finishes once the operation gives up. The warning for this case names the record and the dead node.
**Nodes keep released locks briefly.** After its last user releases a lock, a node keeps it for 10 ms (`CacheGrace`), so back-to-back work on the same node does not write and read the record again. A node reuses a lock it claimed for at most 250 ms (`HoldTurn`). After that, new work on that node waits up to another 250 ms for the current holders to finish. The node then releases the lock and waits 50 ms (`YieldGap`) before taking it again, so another node gets a turn. A node that claims a lock and finds another node's waiting mark gives it up the same way 250 ms after it first saw the mark, even if it released and claimed the lock again in between. Without this, a node in a lower slot that keeps taking a lock back every few milliseconds would shut out a waiting node in a higher slot. A mark counts only while its node is alive in the registry. Another node that wants the lock waits up to about 35 ms while the holder is idle, and up to about 300 ms while the holder keeps using it. A clean unmount releases kept locks at once. A crashed node's last locks, kept or held, stay held until another node proves it dead and replays its journal (see [Node Requirements](#node-requirements)). A lock wait gives up after 30 seconds (`DefaultTimeout`) while the holder is alive. A wait for a failed holder has no time limit: it logs a warning that names the node, and it ends when the replay does. It fails with `EIO` only if the replay fails or is abandoned, which shows as a lapsed recovery lease, with one exception. An operation that already holds a lock and then waits for a failed node's lock keeps the 30-second limit and fails with `EIO`, because the replay may need the lock it holds. An overwrite of a file's existing data does not: it lets go of its locks at once, waits for the block's lock holding none, and retries, so it does not fail. Operations that already changed something first, such as a create or a write that allocated blocks, keep the 30-second limit. For example, a node holds a directory's lock and waits for a block lock the dead node held, while the dead node's journal changes that directory. The replay's own waits have no limit, so it finishes once the operation gives up. The warning for this case names the record and the dead node.

**Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves.

Expand Down
116 changes: 116 additions & 0 deletions internal/lock/fairness_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
package lock

import (
"slices"
"sync"
"testing"
"time"
)

// reclaimAfterExpiry holds the lock briefly, releases it, and claims it again as
// soon as m's cached hold is gone from disk, until stop closes.
func reclaimAfterExpiry(t *testing.T, m *Manager, lt LockType, id uint64, stop <-chan struct{}) {
recordID := m.hashToRecord(lt, id)
for {
select {
case <-stop:
return
default:
}
if err := acquireWithin(m, 5*time.Second, lt, id, LockModeExclusive); err != nil {
t.Errorf("node %d Acquire: %v", m.nodeSlot, err)
return
}
time.Sleep(time.Millisecond)
release(t, m, lt, id)
for {
m.mu.Lock()
_, kept := m.records[recordID]
m.mu.Unlock()
if !kept {
break
}
time.Sleep(100 * time.Microsecond)
}
}
}

func TestHigherSlotGetsLockFromLowerSlotThatKeepsReclaimingIt(t *testing.T) {
dev, cleanup := createTestDevice(t, 1<<20)
defer cleanup()
low := newTestNode(dev, 0, 1<<20, 1)
high := newTestNode(dev, 0, 1<<20, 2)

stop := make(chan struct{})
var wg sync.WaitGroup
wg.Go(func() { reclaimAfterExpiry(t, low, LockTypeInode, 3, stop) })
defer func() { close(stop); wg.Wait() }()
time.Sleep(20 * time.Millisecond)

var waits []time.Duration
for range 10 {
start := time.Now()
if err := acquireWithin(high, 5*time.Second, LockTypeInode, 3, LockModeExclusive); err != nil {
t.Fatalf("higher slot never got the lock: %v", err)
}
waits = append(waits, time.Since(start))
release(t, high, LockTypeInode, 3)
time.Sleep(20 * time.Millisecond)
}
if longest := slices.Max(waits); longest > 2*time.Second {
t.Errorf("higher slot waited up to %v, want under 2s: %v", longest, waits)
}
}

func TestHolderYieldsOnlyToLiveWaiters(t *testing.T) {
cases := []struct {
name string
view bool
state OwnerState
owner byte
age time.Duration
yields bool
}{
{name: "live waiter", view: true, state: OwnerAlive, owner: 5, yields: true},
{name: "failing waiter", view: true, state: OwnerFailing, owner: 5},
{name: "replayed waiter", view: true, state: OwnerReleased, owner: 5},
{name: "mark of the slot's earlier node", view: true, state: OwnerAlive, owner: 9},
{name: "fresh mark without registry", yields: true},
{name: "stale mark without registry", age: pastStale},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
defer func(d time.Duration) { holdTurn = d }(holdTurn)
holdTurn = 50 * time.Millisecond
dev, cleanup := createTestDevice(t, 1<<20)
defer cleanup()
holder := newTestNode(dev, 0, 1<<20, 1)
if tc.view {
reg := newFakeRegistry()
reg.set(5, tc.state)
holder.SetOwnerView(reg.view)
}
mark := &claimSlot{mode: modeWaiting, lockType: LockTypeInode, resourceID: 3, owner: [16]byte{tc.owner}, acquireTime: time.Now().Add(-tc.age).UnixNano()}
if err := holder.writeSlot(holder.hashToRecord(LockTypeInode, 3), 5, mark); err != nil {
t.Fatal(err)
}

stop := make(chan struct{})
var wg sync.WaitGroup
wg.Go(func() { reclaimAfterExpiry(t, holder, LockTypeInode, 3, stop) })
time.Sleep(4 * holdTurn)
close(stop)
wg.Wait()

var blocked uint64
for _, ts := range holder.Stats().Types {
if ts.Name == "inode" {
blocked = ts.Blocked
}
}
if yielded := blocked > 0; yielded != tc.yields {
t.Errorf("holder yielded: %v (%d blocked acquires), want %v", yielded, blocked, tc.yields)
}
})
}
}
Loading
Loading