Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
24c07f5
fix(lock): stop hash collisions and same-node races from creating dou…
mattmattox Oct 24, 2026
78daa43
docs(operations): document lock stall hazard and safe operating limits
mattmattox Oct 24, 2026
8fc8444
fix(fs): wire the lock manager into mounted filesystems
mattmattox Oct 24, 2026
ff31c15
test: verify first-write lost updates from a restarted reader
mattmattox Oct 24, 2026
f6ddae8
test(lock): age records through a test helper now that writeRecord is…
mattmattox Oct 24, 2026
39b2046
fix(lock): stop same-node writers waiting on their own readers; renew…
mattmattox Oct 24, 2026
91169e8
fix(fs): drop the inode lock before freeing blocks and releasing tree…
mattmattox Oct 24, 2026
17d6982
fix(lock): stop hash collisions and same-node races from creating dou…
mattmattox Oct 24, 2026
001fbd1
docs(operations): document lock stall hazard and safe operating limits
mattmattox Oct 24, 2026
4d3c5a8
fix(fs): wire the lock manager into mounted filesystems
mattmattox Oct 24, 2026
de7683d
test: verify first-write lost updates from a restarted reader
mattmattox Oct 24, 2026
fbb6455
test(lock): age records through a test helper now that writeRecord is…
mattmattox Oct 24, 2026
6e9e615
fix(lock): stop same-node writers waiting on their own readers; renew…
mattmattox Oct 24, 2026
4a50602
fix(fs): drop the inode lock before freeing blocks and releasing tree…
mattmattox Oct 24, 2026
fd638fd
fix(fs): clear unwritten flags outside the inode lock
mattmattox Oct 24, 2026
b88d32d
chore: record previous branch tip after rebase onto main
mattmattox Oct 24, 2026
b6f67aa
Merge origin/main into claude/fix-lock-double-holders-9658e1
mattmattox Oct 24, 2026
7750075
fix(lock): wait for same-node readers again; never hold a local mutex…
mattmattox Oct 24, 2026
e8f5aac
docs(operations): note that read locks are not renewed
mattmattox Oct 24, 2026
437148c
Merge origin/main into claude/fix-lock-double-holders-9658e1
mattmattox Oct 24, 2026
81279c6
fix(fs): keep blocks a mid-write persist committed when the final ino…
mattmattox Oct 24, 2026
dc566d4
Merge origin/main into claude/fix-lock-double-holders-9658e1
mattmattox Oct 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ func main() {
}

// Create filesystem with journal, TRIM, SFB, and refcount for crash consistency
gfs := goclusterfs.NewWithComponents(dev, bm, inodeTbl, extentMgr, dirMgr, cacheMgr, j, trimMgr, sfbMgr, refcountReg, nodeUUID, dataOffset)
gfs := goclusterfs.NewWithComponents(dev, bm, inodeTbl, extentMgr, dirMgr, cacheMgr, j, trimMgr, sfbMgr, refcountReg, lockMgr, nodeUUID, dataOffset)

// Set up cross-node recovery handler for failed node journal replay
recoveryCtx := &journal.RecoveryContext{
Expand Down
36 changes: 35 additions & 1 deletion docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,12 +241,46 @@ rm /mnt/fs/.snapshots/snapshot_name
iscsiadm -m node -T <iqn> -p <ip> --logout
```

### Distributed Lock Limits

Nodes coordinate metadata writes (block bitmap, inodes, directories, refcounts, SFB containers, snapshots) through lock records on the shared device. Read this before you run more than one node.

**How a lock goes stale.** A lock record stores the wall-clock time it was taken. Nothing renews it. Any other node may take over a lock that is older than 15 seconds (`StaleThreshold`), judged by that node's own clock. This is how locks held by a crashed node get freed.

**The hazard: a stalled node keeps writing.** If a node stops for more than 15 seconds while it holds a lock, another node takes the lock. When the stalled node resumes, it finishes its writes. No write path checks that the lock is still held, and there is no fencing token. The node only learns it lost the lock when it releases it (`ErrNotLockOwner`), and callers ignore that error. Two nodes then write the same metadata, which can double-allocate blocks or lose directory entries.

Common causes of a stall:

- VM pause, suspend, live migration, or a memory snapshot
- `docker pause`, cgroup freezer, `SIGSTOP`, or a debugger on the mount process
- Heavy swapping or a long GC pause
- Storage I/O that queues instead of failing, for example a multipath or iSCSI path failover

**Lock takeover is not atomic.** Taking a lock is a read followed by a write, not an atomic compare-and-swap. Two nodes that read a free record within one disk round trip of each other can both take it. This is rare but more likely on busy locks such as the block bitmap. The same race can lose an update to a read lock's reader count. The lock then looks held until it is 15 seconds old, so writers to that directory or file can stall for up to 15 seconds.

**Read locks are not renewed.** A node whose overlapping lookups keep a directory read-locked for more than 15 seconds can lose that lock to another node's writer. Its readers may then see a half-written directory. Two writers still cannot overlap.

**Locks share records.** The lock region has 256 records, and every lock maps to one of them, so unrelated files and the block bitmap can share a record. A node holding one of those locks blocks other nodes from all of them. If two nodes each hold one record and wait for the other's, both wait until one record is 15 seconds old; the takeover then lets both proceed, with the same risk as a stall. This needs nested locks whose records cross, roughly 1 in 65,000 pairs of overlapping operations.

**Safe operating limits:**

| Limit | Why |
|-------|-----|
| Keep node clocks within 1 second (chrony or NTP) | Clock skew S shortens the stale threshold to 15s − S |
| No node may stall for 10 seconds or more while mounted | Leaves margin below the 15s threshold |
| Storage I/O must fail, not queue, within 10 seconds | A queued write that lands after takeover corrupts metadata. Do not use multipath `queue_if_no_path`; lower iSCSI `node.session.timeo.replacement_timeout` (default 120s) |
| Unmount before you pause, migrate or snapshot a node VM | Unmounting releases its locks |

If a node did stall for more than 15 seconds while mounted (hung-task warnings in `dmesg`, a VM pause, a path failover), stop its mount process before it does more writes, then check the data from a healthy node.

What is safe today: a node that crashes or is killed and stays down, and a clean unmount. Other nodes take over a crashed node's locks 15 seconds after it acquired them.

### Handling Node Failure

When a node fails unexpectedly:

1. **Other nodes continue operating** - Distributed locking handles failed node
2. **Lock cleanup** - Stale locks are detected via heartbeat timeout
2. **Lock cleanup** - Other nodes take over the failed node's locks once they are 15 seconds old (lock age, not node heartbeat)
3. **No immediate action required** - System is self-healing

**When failed node recovers**:
Expand Down
12 changes: 6 additions & 6 deletions internal/bitmap/bitmap.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,6 @@ func (r *Region) Alloc(ctx context.Context, count int) ([]uint64, error) {
return nil, nil
}

r.mu.Lock()
defer r.mu.Unlock()

// Acquire distributed lock if available
if r.lockMgr != nil {
if err := r.lockMgr.Acquire(ctx, lock.LockTypeBlockBitmap, 0, lock.LockModeExclusive); err != nil {
Expand All @@ -100,6 +97,9 @@ func (r *Region) Alloc(ctx context.Context, count int) ([]uint64, error) {
defer r.lockMgr.Release(lock.LockTypeBlockBitmap, 0)
}

r.mu.Lock()
defer r.mu.Unlock()

allocated := make([]uint64, 0, count)

// Scan through sectors looking for free blocks
Expand Down Expand Up @@ -170,9 +170,6 @@ func (r *Region) Free(ctx context.Context, blockNum uint64) error {
return ErrBlockOutOfRange
}

r.mu.Lock()
defer r.mu.Unlock()

// Acquire distributed lock if available
if r.lockMgr != nil {
if err := r.lockMgr.Acquire(ctx, lock.LockTypeBlockBitmap, 0, lock.LockModeExclusive); err != nil {
Expand All @@ -181,6 +178,9 @@ func (r *Region) Free(ctx context.Context, blockNum uint64) error {
defer r.lockMgr.Release(lock.LockTypeBlockBitmap, 0)
}

r.mu.Lock()
defer r.mu.Unlock()

sectorIdx := blockNum / BitsPerSector
bitOffset := blockNum % BitsPerSector
byteIdx := bitOffset / 8
Expand Down
Loading
Loading