Repository navigation
fix(journal): stop replay at a stale record, and order completions after their writes - #123
Merged
Merged
Conversation
…s place From the crash-injection review (ed0b38e on test/crash-injection-review). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter their writes Commit wrote the record, then the superblock tail, and syncs both together, so a power cut could keep the tail and lose the record. Replay then read whatever record was there: one from an earlier mount, since every mount starts over at offset 0 and seq 1, or from the ring's previous lap. Version 2 journals number records in the order they are written: Commit takes the seq as it appends the record, and completions are headed records in the same sequence. Replay stops at the first record that is not the next seq of this node and generation. Version 1 journals left by older builds replay as before. CompleteIntent synced after writing the completion, so a power cut during that sync could keep the completion and lose the operation's writes. It now syncs first and leaves the completion to the next sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replay applied stale records. Commit writes the record, then the superblock tail, and one fsync covers both, so a power cut can keep the tail and lose the record. Replay then read the record left at that spot: one from an earlier mount, since every mount starts over at offset 0 and seq 1, or one from the ring's previous lap. vm-harness's tests from ed0b38e (
stale_record_test.go) reproduce both cases.Fix: version 2 journals
TxnTypeIntentComplete) in the same seq sequence, carrying the node UUID and generation. The old IntentCompletion had neither, so a stale completion from an earlier mount could mark the current mount's intent with the same seq as done.LastCheckpointSeq.Seqs still start at 1 each mount. The generation check already rejects every earlier mount's record, so keeping the seq across mounts would add nothing.
Flush points
Every sync is
fsyncon the device file. On a block device that sends a cache flush; on an image file it flushes the file and its filesystem. Nothing uses FUA or BLKFLSBUF.In both Sync and Batch mode, the completion used to be able to land before the operation's writes. A power cut then kept "done" and lost the writes, and replay skipped them. Replaying a finished operation is safe since #80 and #94, so a lost completion only means a replay.
The tail write needs no flush of its own: it shares the commit's fsync, and replay no longer trusts it to cover a record that landed.
Cost (.66: ext4 image on a spinning disk behind a PERC H310 in write-through mode; 1,000 operations; 3 alternating rounds; median)
There's no measurable cost; the rounds vary more than the medians do. A journaled operation still does two fsyncs, one in Commit and one in CompleteIntent, and the second now runs outside the journal lock.
fsync() from applications
File.Fsyncfsyncs the device, so data written before it is durable when it returns.Dirand the root don't implementFsync, so go-fuse returns ENOTSUP. Applications that fsync a directory to persist a create or unlink get an error. In Sync mode those operations are already durable when they return. Follow-up.Results at 03c4feb
TestRecoverIgnoresSameSizeRecordFromEarlierLapis there because vm-harness's lap test hits a stale two-sector record, which the length check already stops; without the seq check, the new test replays the previous lap's transaction.🤖 Generated with Claude Code