Skip to content

fix(journal): flush replayed writes before moving the head and recheck the lease - #20

Merged
mattmattox merged 1 commit into
mainfrom
claude/journal-replay-flush-lease
Oct 24, 2026
Merged

mattmattox merged 1 commit into
mainfrom
claude/journal-replay-flush-lease

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #15, from its review: two gaps in resumable journal replay.

  • Power loss: replay advanced the journal head without flushing. With a volatile device cache, the head could survive ahead of replayed writes that were lost, and a resumed replay would skip them. Replay now syncs the device before each head write.
  • Stalled recoverer: replay now checks the recovery lease before each head write, not only before each transaction. A lease renewal re-reads the slot after 100 ms, as a takeover does. A recoverer that stalled past the 60 s lease can no longer write its cached Recovering header over a journal that another node has since reset or initialized.

New test: TestRecoverCheckedLeavesHeaderAloneAfterLosingLease.

Tests on .66 with ext4 images:

  • Unit tests with -race: pass.
  • Integration -short: 73 pass.
  • Integration full: 84 of 85 pass. TestJournalCrashDuringWrite fails 5 of 5 runs on main before fix(journal): give each node registry slot its own journal region #15 (c312a28), on main now (a1957da) and on this branch alike, so the failure is not new. Mount-time replay waits about 15 s for the root directory lock still held by the crashed process (the lock-staleness rule), longer than the test's 10 s mount timeout. CI skips this test under -short. The fix would let a node free locks its own previous incarnation held once it takes back that slot, which is a lock-package change.
  • VM journal tests on .68 (Journal Recovery x4, Restart Keeps Peer Journal, Journal Crash Recovery x3, Journal Crash During Write): 9/9 pass.

🤖 Generated with Claude Code

…k the lease

Replay advanced the journal head without a flush, so after power loss
with a volatile device cache the head could persist ahead of unflushed
replayed writes and a resume would skip them. Replay now syncs the device
before each head write.

Replay also checks the recovery lease before each head write, and a lease
renewal re-reads the slot after 100ms like a takeover, so a recoverer
that stalled inside a transaction cannot write its cached header over a
journal another node has reset or initialized since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit a9d21ff into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
…ims are confirmed (#22)

Taking a lock is still a read then a write, so two nodes that claim a free
inode lock at the same moment can both hold it. The test does that for 20
files per run and loses one node's write in about 1 file in 150: 4 of 30
runs failed on main and 2 of 30 on #13 before #18 and #20 landed, so #18 is
not the cause. CI runs integration with -short; full runs still include it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant