Skip to content

fix(snapshot): order in-place writes with snapshots on other nodes - #42

Merged
mattmattox merged 4 commits into
mainfrom
fix-cow-snapshot-race
Oct 24, 2026
Merged

mattmattox merged 4 commits into
mainfrom
fix-cow-snapshot-race

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

The race is real. On main, the new deterministic test fails: node A pauses after deciding to write a block in place, node B snapshots the file, and the snapshot then shows A's data. That holds both for a never-shared block and for one shared earlier. The real-binary stress test also caught it on main (3 failures in 25 runs).

The remaining stress failures (1 in 20 on this branch) follow a refcount-lock "not the lock owner" event. That is #29's double-holder race losing a snapshot's count, so the stress test skips in -short mode and in Docker/VM until #29. Nothing in internal/lock changed.

Cost: one shared inode lock round-trip per in-place block write.

Not changed here: writes give up with EAGAIN after 3 retries under frequent snapshots, as on main.

Tested on .68: lint, unit -race, integration -short (88 passed / 17 skipped).

🤖 Generated with Claude Code

mattmattox and others added 4 commits October 24, 2026 18:07
A write decides to write a block in place from an inode it read without
the inode lock. A snapshot or clone on another node could mark the file
shared and count the new owner between that decision and the data
write, so the snapshot then showed data written after it was made.

Snapshots and clones now mark the file shared and count the owners
under its exclusive inode lock, holding the refcount lock first as lock
order requires. In-place writes take the shared inode lock and check
again that the block maps to the same place and has no other owner,
else they retry and copy it. Concurrent writers to one file still run
in parallel, since the check looks at the block, not MetaGeneration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Holding the refcount lock while waiting for the inode lock could take
longer than the lock manager lets a hold be reused, so Share hit the
lock-order check and the snapshot failed with EIO under write load.
Snapshots now count the owners first and then mark the file shared
under the exclusive inode lock, undoing the counts and retrying if a
write copied a block in between.

The stress test skips in -short mode and in Docker and VM until #29:
two nodes can both hold the refcount lock, and a lost count lets a
later write go in place into a block a snapshot shares.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Move the doc comments the new functions separated from their owners.
- Skip the two-node stress test in full mode too until #29.
- Test the release-and-retry path when a write, copy, truncate or append
  lands between counting the blocks and marking the file shared.
- A snapshot or clone that keeps losing that race fails with EAGAIN.
- The in-place re-check retries only when the block moved or is gone,
  and returns a corrupt tree as EIO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep main's refcount coverage check: shareSource refuses blocks past the
refcount region before it counts them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 07e6ee6 into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
…irmed

#42 skipped it in Go, and its Docker and VM versions, because two nodes could
both hold the refcount lock. The merge resolution re-enables the shell ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 13, 2026
#29)

* feat(lock): confirm claims after a settle wait and cache released holds

Taking a lock reads the record and then writes a claim, so two nodes that
read a free record at the same moment could both win. A node now waits
ClaimSettleDelay (20ms) after writing a claim or a renewal and re-reads the
record; it keeps the lock only if its claim is still there. A shared add on
a record that is already shared skips the wait: losing that race can only
over-count readers.

To keep the wait off hot paths, a node keeps a record for CacheGrace (250ms)
after its last local holder releases it, so the next local acquire needs no
disk I/O. A cached hold that is too weak for the request, lost, or older than
renewAfter is released and acquired afresh. FS.Close releases cached holds.
RetryMaxDelay drops to 100ms so waiters notice a released hold soon.

Writes of whole 4KB sectors to an allocated, written, unshared block no longer
take the per-block extent lock or read the sector first.

FUSE cancels a request's context whenever the caller gets a signal; Go
callers get SIGURG constantly, so lock waits now end only on a deadline or
after DefaultTimeout. Torn reads of a record another node is writing are
read again before the record is reported invalid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): document the lock settle wait, cache grace and I/O limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): benchmark an acquire that cannot reuse a cached hold

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): let a node take the locks its registry slot's previous owner held

With cached holds, a node that crashes leaves every record it used in the
last 250ms held, so its restarted mount waited 15s for its own earlier
incarnation's locks. A record held exclusively under this node's registry
slot by another UUID belongs to an earlier occupant of the slot, which the
registry hands out only after that occupant failed, so it is taken at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): yield records a busy node has held past the cache grace

A node that kept reusing a record rejoined its cached hold before the grace
ran out, so it never released it, and another node waited as long as the
first stayed busy: a restarted mount waited while a peer kept creating
files. A hold taken from disk more than CacheGrace ago is now released when
its last local key drops, and this node waits YieldGap before claiming it
again. RetryMaxDelay drops to 25ms so a waiter polls within that gap.

A block allocated by the current write stays invisible to other nodes until
the inode, whose lock the write holds, is committed, so writing it no
longer takes the per-block extent lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: un-skip the first-write test and check readers during allocating writes

Claim confirmation stops two nodes from both taking a free inode lock, so the
first-write test runs in short mode again. A new test has one node read a file
while another fills it with fresh blocks, which it writes without the extent
lock; the reader must see only holes, the old size or each block's own data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fs): check a reader on another node never sees uncommitted blocks

The mounted-FS version could not see the file grow on the second node: its
inode stays cached while the file is open when the cache broadcast is not
running. Two FS instances on one image, with their own lock managers on a
shared lock region and no inode cache, read the on-disk state every time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(lock): drop the slot-owner takeover now that #23 adopts a dead node's locks

#23 frees a crashed node's exclusive locks once it is proven dead, during the
journal replay a restarted node runs before it serves, so the restarted
mount no longer waits for its earlier incarnation's cached holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): keep shared claims that other readers join, drop slow claim writes

A node's first shared claim failed its settle check when another node joined
it during the wait, then re-joined on top, leaking one reader count until the
lock went stale. A shared claim now also holds when the record is still shared
at a newer sequence.

A claim whose write would land more than half the settle delay after its read
is dropped and retried, so a slow or queued write cannot overwrite a claim
another node has already confirmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(goclusterfs): name the extent-lock skip condition

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): check claim write deadlines after the shard wait, bound record retries

The 10ms check on a claim write ran before WriteAt took the device's I/O
shard mutex, so a wait there went unmeasured. storage.Device.WriteAtBefore
now checks the deadline once it holds the shard, right before the write. A
write that still finishes more than 10ms after its read is not held, and
the node leaves that record alone until it is stale, because the write may
have replaced a claim another node already confirmed.

Releases, renewals and dead-node takeovers no longer retry forever on
storage that keeps missing the deadline: they back off with jitter and give
up after 10 tries.

Adds the PM's repro tests: two nodes reading one directory, or the
refcounts, at once must not delay the next create, unlink, snapshot create,
copy-on-write or snapshot delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(ops): note late read claims beside a writer and the cost of a late lock write

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): close late read claims to other readers, free kept locks in parallel

A read claim whose write lands late can sit on top of a write claim another
node has just confirmed. Left as a read claim, it let further readers in
beside that writer. The node now rewrites it as its own write claim before
leaving the record to go stale.

ReleaseCached frees kept records eight at a time, so an unmount on slow
storage no longer pays each record's retries one after another.

The two CAS test hooks become one injectable record write, so tests can
model a write that passes its deadline check and then lands late.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): write the late read claim fence without a deadline

The fence that turns a late read claim into this node's write claim used the
normal 10ms write deadline, so on still-slow storage it was skipped and the
record stayed open to readers beside a writer. The fence can only make the
record exclusive to this node, so it now writes without a deadline. It
fences the record as it is now, so readers that joined before it are shut
out from then on too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(storage): check deferred close and remove errors like the rest of the file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): bound the late read claim fence to half the stale threshold

An unbounded fence write that stalled past StaleThreshold could land on a
claim another node made after taking the record over as stale. The fence
now skips a record that is already stale, and its write must start within
fenceWindow (StaleThreshold/2) of its read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): keep unit tests off the 10ms record write limit unless they test it

On the CI runner's disk a 4KB O_DIRECT lock record write took 25-27ms, so
every claim counted as late, froze its record for 15s, and 12 tests timed
out. Each Manager now carries its write window. The lock tests' TestMain
relaxes the default, and the tests of the window itself restore the real
10ms on the nodes whose writes they slow down. CI also runs unit test
images on tmpfs, checking first that tmpfs takes O_DIRECT writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run concurrent snapshot creates again now that lock claims are confirmed

#33 skipped them in Docker, VM and Go -short because two nodes could both
claim one free inode's lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): run concurrent file creation in -short mode

Its skip blamed a race in the distributed lock. Two nodes allocating inodes
and blocks at once go through the lock claims #29 now confirms, and the
test passed 23 of 23 runs on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lock): log who holds the record when an acquire gives up

Operators need to see which node holds a lock that made an operation fail.
The timeout now logs at WARN with the lock key and the record's holder,
mode and age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run snapshots during writes again now that lock claims are confirmed

#42 skipped it in Go, and its Docker and VM versions, because two nodes could
both hold the refcount lock. The merge resolution re-enables the shell ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(goclusterfs): retry a write up to 10 times when other nodes change the file

A node writing a file while another node snapshots it back to back loses
the copy-on-write check each time a snapshot lands. With the 20ms lock
claim wait those windows are wider, and 3 tries made TestSnapshotsDuringWritesAcrossNodes
fail 4 of 10 runs with EAGAIN. With 10 tries it passed 20 of 20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run integration test images on tmpfs and allow 8 minutes

Lock record writes on the runner's disk can miss their 10ms limit and stall
a lock for 15s, which pushed the -short suite past its 5 minute timeout.
Main's suite already takes about 4 minutes, and lock claims now wait 20ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lock): measure where the time of late lock record writes goes

Each skipped or late record write now logs its time split into waits inside
this process (the device's shard lock, the gap between the read and the write
call) and the read and write system calls. Each node counts them and logs the
totals at unmount. CI integration tests go back to the runner's own disk, so
these numbers show what users on ordinary disks would see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lock): log acquire counts per lock type and a write gap histogram

At unmount each node logs, per lock type, how its acquires were served
(held key, joined record, cached record, fresh claim with or without the
settle wait, blocked tries), and a histogram of the time from a record
CAS's read to its write finishing. These are the numbers ClaimSettleDelay
and CacheGrace are chosen from. Docker CI keeps each node's lock log lines
with its test results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): read claims skip the settle wait and are not kept after release

A node killed within 250ms of a read left its cached read hold on disk, and
other nodes' writers waited 15s for it to go stale (TestNetworkPartition:
3s on main, 17s here). Read claims now release as soon as their last local
reader does, and they no longer wait out ClaimSettleDelay. Only write claims
need that wait: every record write still finishes within half the settle
delay of its read, so a read claim that replaced a writer's claim lands
before the writer's confirm re-read, and the writer backs off. A read can
still run beside a writer that won the race, as on main; two writers cannot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): give up a claim write only if it finished after the settle delay

A claim write can replace a claim another node already confirmed only if it
lands more than ClaimSettleDelay after its own read: until then, the other
node's confirm re-read is still ahead. The 15s give-up now applies past the
full delay, not half of it, and the margin moves to the start: a write that
cannot start within a quarter of the delay is skipped and retried, which is
cheap. On CI, both late writes in run 42863343192/42863342671 (10ms, 16.5ms)
would have been held.

Lock record reads and writes also stop taking the device's shard lock, so a
node's own data and journal I/O cannot delay them: 2 of the 6 missed writes
on CI waited about 10ms for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): name readers with per-node bits and confirm read claims

A read lock's holder count was updated by read-modify-write from several
nodes, so racing claims and releases lost updates. A lost release left the
count too high, and writers waited 15s for the record to go stale
(TestMultiNodeConcurrentDirectoryOps, ...CreatesDoNotCollide on 9af90ae).

Each read-locked record now has a bit per registry slot in its reserved
bytes. A read claim sets this node's bit and waits out ClaimSettleDelay
like a write claim, then keeps the lock only if its bit is still set. A
release clears the bit and checks again in the background that no racing
update put it back. Read holds are cached again. AdoptOwner, given the dead
node's registry slot, clears its reader bits, so a reader killed while it
kept a hold blocks writers only until the node is proven dead. Every claim
writes the current time, since readers' bits now clear on their own.

Late claims (a claim write that finished after the settle delay) are logged
with their lock key and latency and counted at unmount.

Lock records change meaning in the reserved bytes (bits 64:72), which goes
with on-disk format v3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: say 20ms in the unit job's tmpfs comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): stop waiting on lock hint counters and retry reads past attr timeout

The lock hint cache is gone, so its counters stay 0. A restarted node can
cache the size of a peer's file it looked up mid-write for -attr-timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): tie the verifyFiles retry to the node's -attr-timeout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): keep idle holds 10ms and cap busy holds with a separate 250ms turn

An idle cached hold kept other nodes waiting up to 250ms. With the root
directory and the block bitmap both hot, two nodes waited on each other's
idle holds and concurrent creates ran 2.7x slower than main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: add the idle cached hold test to the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): address the #29 review

- Whole-sector writes to existing blocks take the extent lock again. A peer's
  read-modify-write of the same sector could otherwise write back a stale copy
  over an acknowledged write.
- Once a hold's turn is over, new local requests wait for the node's holders
  to drain, so a record whose holds keep overlapping is still yielded. Only a
  goroutine that holds other locks stops waiting after another turn.
- A record adopted in this node's own slot keeps its adopted write claim
  through the replay's own acquires and releases until ReleaseAdopted.
- Adopted claims are flagged, and every claimant defers to them, so a waiter
  in a lower slot cannot keep the replaying node out.
- ReleaseAdopted leaves a slot that a new node has written since.
- A slot that stays torn for StaleThreshold stops blocking its record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(goclusterfs): run the whole-sector lost-update test with a full lock region

With the 1MB test region the two writes' other locks share one of 4 records,
which serializes them and hides the lost update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: make the lost-update and overlapping-hold tests catch their bugs

The whole-sector test raced two writers through a gate, and on fast storage
the partial write finished before the other woke, so it passed with the bug.
A hook after the partial write's sector read now runs the whole-sector write
while the old sector is held. The overlapping-hold test gets a case with
holds longer than a turn, which only an unbounded drain wait passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): expire a torn slot only while it stays torn with the same bytes

A node kept the time it first read a slot torn until it read that slot whole
again. Long after it stopped reading the record, a torn read of a live node's
renewal in flight looked expired, and both nodes held the record. The wait now
restarts whenever the torn bytes change. They are compared by FNV-1a, since
CRC32C gives every slot torn the same way one value.

OPERATIONS.md no longer says whole-sector writes skip the block lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(goclusterfs): take the inode lock exclusive when the block's lock shares its record

A write into an existing block holds the inode lock shared and then takes the
block's extent lock exclusive. When both keys hash to one lock record, the lock
manager sees a shared-to-exclusive upgrade and the write fails with EIO. Main
hits this only on partial-sector writes; #29 makes every write to an existing
block take the extent lock, so the Docker "Snapshot During Writes" test failed
when inode 130 and block 163 both mapped to record 36.

The Docker test now prints dd's error and the node's write errors when the
writer fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant