Skip to content

test(integration): stop the crash writer before unmounting the killed node - #184

Merged
mattmattox merged 1 commit into
mainfrom
claude/restart-test-kill-order
Nov 17, 2026
Merged

mattmattox merged 1 commit into
mainfrom
claude/restart-test-kill-order

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

TestMultiNodeRestartKeepsPeerJournal failed about 1 in 7: node A couldn't find the last file node B acknowledged before B was killed. The bug is in the test, not the product.

KillNode kills the mount process and then unmounts it. The writer goroutine was still running, so a write that started after the unmount went to the plain directory left at the mount point. It succeeded there and counted as acknowledged. Evidence from the 8 failing runs out of 30 with the image saved: the root directory on disk holds every acknowledged file except the last one, and no in-use inode is left without a name. That file never reached GoClusterFS. Replay timing wasn't the cause: in the failing runs, "[Recovery] Completed" was logged before the read failed.

The test now kills B, stops the writer, and only then unmounts. A dead mount fails every write until it's unmounted.

No other test has a writer running on the node it kills.

Results (.66, 16 GB memory cap)

  • main 580d18f: 3 of 20 runs failed.
  • This PR: 30 of 30 passed.
  • Lint: 0 issues.

🤖 Generated with Claude Code

… node

TestMultiNodeRestartKeepsPeerJournal killed node B with KillNode, which also
unmounts it, while the writer goroutine was still running. A write that started
after the unmount went to the plain directory left at the mount point, succeeded,
and counted as acknowledged, so node A could never find that file. The test now
kills B, stops the writer, and only then unmounts: the dead mount fails every
write until it is unmounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 7c31ff8 into main Nov 17, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant