Skip to content

fix(test): replace fixed deadlines and sleeps with polling in snapshot, worker, broadcast and journal tests - #14

Merged
mattmattox merged 9 commits into
mainfrom
claude/test-timing-bounds
Oct 24, 2026
Merged

mattmattox merged 9 commits into
mainfrom
claude/test-timing-bounds

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

TestCloneBasic asserted clone creation under 500ms and took 589ms on main's CI (92f024b) and on #9. It and the two 100ms snapshot latency checks now log the time instead (2.8–18ms on .68).

Fixed sleeps followed by asserts now poll and fail on timeout:

  • Snapshot integration tests poll user.snapshots for active, or for removal after a delete. TestSnapshotDelete used to check nothing when Getxattr succeeded.
  • TestWorkerQueueDeleteJob and 5 other worker tests poll for the job's last step.
  • The two remaining broadcast tests poll for content. The journal crash tests wait for the killed mount to exit.

Fix: QueueCreateJob/QueueDeleteJob selected on the request context, so a canceled context sometimes dropped the job while setxattr succeeded. About 1 in 60 snapshots stayed creating. They now ignore the caller's cancellation, and a new unit test covers it (it fails on main).

Not fixed: deleteSnapshot decrements the live file's blocks. After an overwrite the delete fails with reference count underflow, so TestSnapshotDeleteAndReuse keeps its fixed wait. This goes to the snapshot P0 work.

Left alone: multinode_test.go (lock PR), helpers.go (its Unmount leaves a stale mount on EBUSY, so TestSnapshotPersistence keeps its sleep before unmount), clone tests (nothing to poll), recovery heartbeat waits (-short skips them).

On .68: snapshot and clone tests passed 50/50 on tmpfs and 52/52 under ZFS with 4 parallel workers, except TestSnapshotPersistence at 51/52, the same rate as unchanged main (the Unmount issue above). The worker tests, including the new one, passed 50/50 with -race. The broadcast and journal tests passed 50/50 earlier and are unchanged since.

🤖 Generated with Claude Code

mattmattox and others added 9 commits October 24, 2026 12:10
TestCloneBasic asserted clone creation under 500ms and took 589ms on CI.
TestSnapshotCreation and TestSnapshotLargeFile asserted under 100ms the
same way. They now log the time.

Snapshot tests slept 500ms-1s for the background worker before checking
the snapshot list. They now poll user.snapshots until the snapshot is
active, or gone after a delete. TestSnapshotMiddleDeletion could fail
when the worker had not removed snap-2 within 500ms, and
TestSnapshotDelete never checked the list when Getxattr succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestBroadcastMultipleUpdates and TestBroadcastBidirectional slept 300ms
and then read once from the other node. They now poll for the expected
content for up to 15s and log the time.

The journal crash tests slept 500ms after killing the mount process.
They now wait for it to exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polling for removal in TestSnapshotDeleteAndReuse fails every time:
the delete job decrements the live file's blocks instead of the
snapshot's, hits a refcount underflow, and leaves the snapshot in the
deleting state. The test never checked removal before, so keep its
original wait and leave the bug to a product fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
About 1 in 60 snapshot creations on .68 stays "creating" forever:
QueueCreateJob selects between the job queue and the request context,
so a canceled context can drop the job while setxattr still succeeds.
Asserting "active" turned that product bug into test failures. The
waits now log instead, and the tests keep their original checks that
the snapshot is listed. Waiting for a deleted snapshot to disappear
still fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ence

Unmounting right after the snapshot became active raised busy-unmount
failures under load from 1 to 7 in 52 runs on .68. FUSE releases
closed files asynchronously, and the Unmount helper force-kills the
mount when fusermount reports busy, which leaves a stale mount that the
remount then fails on. Keep the original 500ms wait until the helper
handles busy unmounts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestWorkerQueueDeleteJob and five other worker tests slept 300-500ms
and then asserted the job's effects. They now poll until the create
job marks the snapshot active, or the delete job removes it (both are
the jobs' last step), for up to 10s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker drops delete jobs queued with a canceled request context,
just like create jobs, which left TestSnapshotDelete waiting on a
snapshot stuck in "deleting" (1 in 50 runs on .68). The test now fails
only if the snapshot is still active after the delete, and logs if it
is never removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QueueCreateJob and QueueDeleteJob selected on the job queue and on the
caller's context. With a canceled FUSE request context Go picks a ready
case at random, so the job was sometimes dropped while setxattr still
succeeded. The snapshot then stayed "creating" or "deleting" until a
remount (about 1 in 60 creations on .68). SetRefCountJobID could also
give up on a contended lock with the canceled context.

The descriptor already exists when these run, so they now ignore the
caller's cancellation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With create and delete jobs no longer dropped, the snapshot tests can
require the state they wait for. They fail if a snapshot is not active
within 15s, and TestSnapshotDelete fails if the deleted snapshot is not
removed. TestSnapshotDeleteAndReuse keeps its fixed wait after the
delete: deleteSnapshot still decrements the live file's blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox force-pushed the claude/test-timing-bounds branch from 1641d07 to 5b8252d Compare October 24, 2026 17:31
@mattmattox
mattmattox merged commit c5d9df4 into main Oct 24, 2026
6 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* fix(refcount): count the existing owner when a block is first shared

New blocks start at a count of 0, so a snapshot raised them to 1 and
copy-on-write, which needs more than 1, never fired. Share treats 0 and
1 alike as one owner and adds the new one, and Release drops an owner
and reports the blocks nobody holds any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extent): free shared blocks only after their last owner lets go

Freed blocks now drop one owner in the refcount table and go back to
the bitmap only when none is left. Shared blocks used to be skipped and
leaked. A shared block with a count of 0 comes from a clone made by an
older version, so it stays allocated. A copy-on-write remap bumps
DataGeneration so readers drop cached block mappings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): keep each snapshot's blocks in a hidden data inode

A snapshot stored no block list, so deleting it used the file's current
blocks and a clone of it copied the live file. Each snapshot now owns a
hidden inode that shares the blocks it captured; the descriptor names it
in bytes 28-31 under the new FlagHasData. Creation shares the blocks
before marking the file shared, so later writes copy them. Delete and
crash recovery give back the data inode's blocks. Jobs no longer drop
when the request context is canceled. fsck accepts data inodes and
treats a refcount of 0 as one owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): check snapshot data, clone writes and delete in all suites

Adds Go, Docker and VM tests that read a snapshot through a clone after
in-place overwrite, append and truncate, write into a clone and its
source with dd conv=notrunc, and delete a snapshot after an overwrite.
The Docker image and VMs install attr for setfattr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): free a snapshot's data inode after its descriptor is gone

If a crash hit after the data inode was freed but before the descriptor
was removed, another snapshot could reuse the slot and recovery would
empty it. Freeing last leaks at most an empty inode, which fsck reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): wait for deleted snapshots to disappear

TestSnapshotDeleteAndReuse kept a fixed wait because the delete hit a
refcount underflow and never finished; it now polls for removal like the
other snapshot tests. The manifest names #14's canceled-context test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extent): conflict-check copy-on-write and never free uncounted sharing

Two nodes could copy the same shared block of an inline file: neither
inode write was checked, so both released the old block and freed it
while a snapshot still mapped it. A copy-on-write now leaves a pending
change, so the second write fails with a tree conflict and retries.

New sharing sets FlagCounted. Blocks older versions marked shared
without it have unknown owner counts, so frees leave them allocated and
HasUncountedSharing lets callers refuse to share them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): order sharing safely and refuse files shared without refcounts

- Mark the source shared before its blocks gain owners, and sync the
  refcounts before the copy maps them, so a crash leaves at most a
  leak.
- Refuse to snapshot or clone a file an older version shared
  (ENOTSUP).
- Re-check that a snapshot is still active before a clone of it maps
  its blocks.
- Only a snapshot still being created can become active.
- Snapshot IDs come from the highest ID on disk, so two nodes no longer
  reuse one.
- Refcounts stay on with -snapshots=false, so shared blocks are still
  copied before writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): lock the snapshot registry across nodes

The registry got FS.lockMgr, which is nil, so two nodes creating
snapshots at once could take the same slot or ID: one snapshot vanished
or a delete by ID removed the other node's. InitSnapshots now takes the
mount's lock manager.

Adds a two-node registry test and Go, Docker and VM tests that snapshot
on two nodes at once. Adds tests where node A commits a copy of a shared
block between node B's copy and commit, and where an older version's
snapshot and clone of one file are written and deleted. The ENOTSUP for
files an older version shared now points to goclusterfs-fsck, and the
CHANGELOG says to run fsck before the first mount after upgrading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): never release uncounted blocks, and never roll back active snapshots

A copy-on-write of a block an older version shared without counting its
owners released it like a counted one. With two old snapshots and two
old clones, two copies dropped the count from 2 to 0 and freed a block
the other clone mapped. Such blocks are now copied but never released.

Mount recovery marked any snapshot it saw creating as deleting, even if
another node had activated it since. It now uses a guarded
creating-to-deleting transition, and the delete job refuses snapshots
that are not marked deleting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): keep snapshot IDs and descriptor sectors apart per node

In CI, two nodes creating snapshots at once still got the same ID even
under the registry lock, because the cross-node lock does not always
exclude them. A new ID is now a counter above the highest ID on disk,
times 64, plus the node slot. Each node fills descriptor sectors it
alone owns first, so two nodes neither reuse an ID nor overwrite each
other's sector. The registry test now runs with no lock at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): check IDs after older images and node slot reuse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): copy shared blocks before taking the inode lock

With #13's FS locks, a write into a shared block held the inode lock and
then needed the refcount and bitmap locks, which come first in lock
order, so it failed with EIO. The write now persists and drops the inode
lock, copies the block on its unlocked copy, then takes the lock and
keeps the copy only if no node wrote the inode meanwhile; otherwise it
frees the copy and retries. A conflict after an earlier mid-write commit
frees only the copies that commit did not map.

The in-process snapshot tests now run with a lock manager per FS, like
separate nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): failed final write after a mid-write copy commit

Covers #13's release of committed copies on a failed final inode write:
each old block loses the file's ownership once across the retry, and
uncounted blocks from an older version lose none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): log why a snapshot could not be added to the registry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(snapshot): skip the two-node concurrent snapshot test in short mode

Two nodes can still both claim one free inode until lock claims are
confirmed (delta-lease), and creating 20 snapshots on cold lock records
across two nodes hits that often. The Docker and VM versions still run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(inode): reject stale copies whose inline extents changed on disk

The stale-write check compared only extent trees. A copy of an inline
inode read before a copy-on-write committed, such as a slow chmod on
another node, could write the old mapping back after the old block was
released, leaving the snapshot's block undercounted. A plain write now
fails with ErrTreeConflict when the inline extents on disk differ from
the ones the copy was read with; callers already retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(snapshot): retry copy-on-write on a stale tree and finish interrupted snapshot calls

A copy-on-write on an unlocked inode copy can find the tree rewritten by
another node; it now retries instead of returning EIO. Snapshot and clone
xattrs span several writes, and go-fuse cancels the context when the
kernel interrupts a request, which made a contended registry lock fail
with EIO. They now run to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant