Skip to content

fix(journal): stop replay at a stale record, and order completions after their writes - #123

Merged
mattmattox merged 2 commits into
mainfrom
claude/journal-record-checks
Nov 13, 2026
Merged

mattmattox merged 2 commits into
mainfrom
claude/journal-record-checks

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Replay applied stale records. Commit writes the record, then the superblock tail, and one fsync covers both, so a power cut can keep the tail and lose the record. Replay then read the record left at that spot: one from an earlier mount, since every mount starts over at offset 0 and seq 1, or one from the ring's previous lap. vm-harness's tests from ed0b38e (stale_record_test.go) reproduce both cases.

Fix: version 2 journals

  • Commit takes the seq as it appends the record, so seqs follow the order records are written in. Before, Begin assigned it, and two transactions could commit in the other order.
  • Completions are now headed records (TxnTypeIntentComplete) in the same seq sequence, carrying the node UUID and generation. The old IntentCompletion had neither, so a stale completion from an earlier mount could mark the current mount's intent with the same seq as done.
  • Replay stops at the first record that isn't intact, from this node and generation, with exactly the next seq, starting above LastCheckpointSeq.
  • A version 1 journal, left dirty by an older build, still replays as before.

Seqs still start at 1 each mount. The generation check already rejects every earlier mount's record, so keeping the seq across mounts would add nothing.

Flush points
Every sync is fsync on the device file. On a block device that sends a cache flush; on an image file it flushes the file and its filesystem. Nothing uses FUA or BLKFLSBUF.

Mode Commit CompleteIntent Checkpoint
Sync (mount's default) writes record and tail, then fsyncs before returning, so the record is durable before the operation writes anything before: wrote the completion, then fsynced. Now: fsyncs, then writes the completion, which the next fsync makes durable fsyncs, then moves the head
Batch waits for a batch fsync before returning before: no sync at all. Now: waits for a batch fsync, then writes the completion same
Async (tests only) no fsync no fsync same

In both Sync and Batch mode, the completion used to be able to land before the operation's writes. A power cut then kept "done" and lost the writes, and replay skipped them. Replaying a finished operation is safe since #80 and #94, so a lost completion only means a replay.

The tail write needs no flush of its own: it shares the commit's fsync, and replay no longer trusts it to cover a record that landed.

Cost (.66: ext4 image on a spinning disk behind a PERC H310 in write-through mode; 1,000 operations; 3 alternating rounds; median)

creates/s allocating 4 KB writes/s 4 KB overwrites/s
main fe141a9 34 115 613
this PR 33 130 686

There's no measurable cost; the rounds vary more than the medians do. A journaled operation still does two fsyncs, one in Commit and one in CompleteIntent, and the second now runs outside the journal lock.

fsync() from applications

  • Files: File.Fsync fsyncs the device, so data written before it is durable when it returns.
  • Directories: Dir and the root don't implement Fsync, so go-fuse returns ENOTSUP. Applications that fsync a directory to persist a create or unlink get an error. In Sync mode those operations are already durable when they return. Follow-up.

Results at 03c4feb

  • Lint: 0 issues.
  • .66: journal package passes under race.
  • VM at 03c4feb: 59 passed, 0 failed on both .68 and .66, including Power Failure Simulation, Concurrent Writes Power Failure, fsck Repair After Power Failure, VM Kill Recovery, and the nine journal recovery and crash tests.
  • Full race unit tests and integration: running.
  • Mutation checks: dropping the owner/generation check, the seq check, or the version 1 path each makes a test fail. TestRecoverIgnoresSameSizeRecordFromEarlierLap is there because vm-harness's lap test hits a stale two-sector record, which the length check already stops; without the seq check, the new test replays the previous lap's transaction.

🤖 Generated with Claude Code

mattmattox and others added 2 commits November 13, 2026 15:59
…s place

From the crash-injection review (ed0b38e on test/crash-injection-review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter their writes

Commit wrote the record, then the superblock tail, and syncs both together, so
a power cut could keep the tail and lose the record. Replay then read whatever
record was there: one from an earlier mount, since every mount starts over at
offset 0 and seq 1, or from the ring's previous lap.

Version 2 journals number records in the order they are written: Commit takes
the seq as it appends the record, and completions are headed records in the
same sequence. Replay stops at the first record that is not the next seq of
this node and generation. Version 1 journals left by older builds replay as
before.

CompleteIntent synced after writing the completion, so a power cut during that
sync could keep the completion and lose the operation's writes. It now syncs
first and leaves the completion to the next sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 51c0081 into main Nov 13, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant