Skip to content

fix(lock): give up a lock wait after 30s and log who holds the lock - #54

Merged
mattmattox merged 1 commit into
mainfrom
claude/lock-wait-limit
Nov 10, 2026
Merged

mattmattox merged 1 commit into
mainfrom
claude/lock-wait-limit

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Unblocks #45. It detaches FUSE handlers' contexts from interrupts, so on main a lock wait whose holder never releases would last until SIGKILL. Acquire now also gives up after DefaultTimeout (30s) and logs at WARN the lock key and the record's holder, mode and age. Context cancellation still ends a wait as before.

This is only the wait limit from #29; nothing else from #29 is included.

Tested on .68: the two new tests pass 3x with -race, and each fails with its change removed. Lint and unit -race (20 packages) pass, and integration -short is 98/98.

🤖 Generated with Claude Code

Acquire ended a wait only when the caller's context was done. A FUSE handler
that detaches its context from request interrupts (#45) would then wait
forever on a lock that is never released. Acquire now also gives up after
DefaultTimeout (30s), and logs at WARN the lock key and the record's holder,
mode and age, so operators can see which node held it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit c052c84 into main Nov 10, 2026
10 checks passed
mattmattox added a commit that referenced this pull request Nov 10, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 10, 2026
* test: interrupt FS operations while they wait for cluster locks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): finish operations whose FUSE request is interrupted

The kernel sends FUSE_INTERRUPT whenever the calling process gets a
signal, and Go programs signal themselves for preemption all the time.
go-fuse cancels the request context, lock.Acquire gives up, and the
operation returned EIO, after side effects such as a committed journal
intent. Every FUSE handler now runs on a context that interrupts do not
cancel, so the operation finishes. Honoring an interrupt with EINTR
would need rollback of partial work, breaks retries of unlink, rmdir
and O_EXCL create, and C programs rarely retry it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: signal the creating threads directly and track the interrupt tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: describe interrupted file operations and the 30s lock wait limit

File operations ignore interrupts like an NFS hard mount; a lock wait that cannot finish fails with EIO after DefaultTimeout (from #29).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: the lock wait timeout log names the holder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): detach only the handlers that pass ctx on

Statfs, Getattr, Open, Read and Fsync never use their context, so they cannot wait on a lock; ineffassign flagged the dead assignment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: an interrupted write gives up with EIO after DefaultTimeout

Needs #29: on main a lock wait has no time limit, so this test hangs.
Merge it into #45 once #29 is on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: track the lock timeout test in the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: match the lock wait timeout log from #54

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 12, 2026
Main's #54 (30s wait limit, lock.Stats) conflicted with #29's lock.go.
The merge resolves it by replacing #29's claim path:

- Each lock record is a row of 64 claim slots of 4KB, one per registry
  slot. A claim writes the node's own slot, then reads the row; it wins
  if no other live slot conflicts. No timing bound is needed.
- When claims collide, the lower slot keeps its claim and waits; a node
  that loses to a lower slot clears its slot and reads before writing
  again. Waits point only to higher slots, so they cannot form a cycle.
- AdoptOwner rewrites a dead node's write and torn slots with a fresh
  time and ReleaseAdopted clears them after the replay; the dead node's
  read slots are cleared at once.
- The superblock records the lock slot size (offset 264). Mount refuses
  v3 images formatted before slots and says to reformat. The lock region
  is 64MB and the smallest device 327MB.
- Kept from #29: cached holds and yield, renewal, record gates, bounded
  retries, unsharded lock I/O and the unmount stats. Dropped: the settle
  wait, late-claim freeze, reader bits, background verifier and the
  write-deadline storage calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant