Repository navigation
test(integration): stop the crash writer before unmounting the killed node - #184
Merged
Merged
Conversation
… node TestMultiNodeRestartKeepsPeerJournal killed node B with KillNode, which also unmounts it, while the writer goroutine was still running. A write that started after the unmount went to the plain directory left at the mount point, succeeded, and counted as acknowledged, so node A could never find that file. The test now kills B, stops the writer, and only then unmounts: the dead mount fails every write until it is unmounted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestMultiNodeRestartKeepsPeerJournalfailed about 1 in 7: node A couldn't find the last file node B acknowledged before B was killed. The bug is in the test, not the product.KillNodekills the mount process and then unmounts it. The writer goroutine was still running, so a write that started after the unmount went to the plain directory left at the mount point. It succeeded there and counted as acknowledged. Evidence from the 8 failing runs out of 30 with the image saved: the root directory on disk holds every acknowledged file except the last one, and no in-use inode is left without a name. That file never reached GoClusterFS. Replay timing wasn't the cause: in the failing runs, "[Recovery] Completed" was logged before the read failed.The test now kills B, stops the writer, and only then unmounts. A dead mount fails every write until it's unmounted.
No other test has a writer running on the node it kills.
Results (.66, 16 GB memory cap)
🤖 Generated with Claude Code