Repository navigation
fix(journal): keep the head behind intents whose operation has not completed - #65
Merged
Merged
Conversation
Commit takes a transaction out of activeTxns before the operation runs, so a checkpoint can move the head past an intent that is committed but not yet applied, and a crash then loses the operation. These tests stop between Commit and CompleteIntent, checkpoint, and check the next mount replays the intent: in the journal alone, through a create on the filesystem followed by fsck, and across an unmount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpleted Commit took a transaction out of activeTxns before the operation it logs ran, so the 5 s checkpoint, the flush-journal endpoint, a full journal or unmount could move the head past a committed intent that was not yet applied. A crash in that window lost the half-done operation. The journal now keeps each committed transaction pending until CompleteIntent. Checkpoints stop at the oldest active or pending transaction and sync the device first, so every change they pass is durable. Unmount leaves the journal dirty while an intent is pending. CompleteIntent is idempotent, and each operation defers it right after its commit, so an operation that fails after committing still closes its intent and is never replayed, as before. A Commit whose sync fails drops its intent, since its caller never runs the operation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both create paths logged the mode FUSE passed, while the runtime writes S_IFREG with the permission bits. A replayed create therefore depended on the caller's mode for the inode type, and fsck reported "invalid file type" when it lacked S_IFREG. Mkdir already logs S_IFDIR this way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n stop TestJournalCleanOnStop and TestCheckpointAdvancesHeadNow committed without CompleteIntent and expected the journal to drop the transaction, which is the bug. They now complete the intent first, and the checkpoint test also checks that the head does not move before completion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Truncate journals the logical block where the freed range starts, and replay frees physical blocks by that number. A crash between a truncate's commit and its changes then makes replay free blocks that files still map. The crash hook now covers setattr too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Truncate and unlink log the logical file blocks they free, and replay freed physical blocks with those numbers. Replaying an interrupted truncate or unlink in a subdirectory therefore freed blocks files still map, starting at block 0, which the root directory uses. Replay now frees nothing for a BlockFree record: an interrupted free leaves its blocks allocated, which fsck reports, instead of freeing blocks in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records are sector-aligned and WriteAt accepted one that used the last free byte, so a ring filled while a pending intent pinned the head ended with head == tail, which recovery reads as empty. The next write then treated the whole ring as free and overwrote the pending intent. AvailableSpace now keeps one sector free, and Commit and CompleteIntent check the sector-aligned size WriteAt writes before deciding to checkpoint. Recovery also never finished when the tail sat at the end of the data area: the scan wrapped to 0 and kept looking for it. It now stops there. Mkdir in the root now journals only the permission bits with S_IFDIR, as mkdir in subdirectories does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 12, 2026
This was referenced Nov 12, 2026
mattmattox
added a commit
that referenced
this pull request
Nov 12, 2026
…links leave (#75) * test(fsck): free the blocks an interrupted truncate leaves A node crashes after a truncate writes the shorter inode but before it frees the cut blocks. Since #65, replay frees nothing for that record, so the blocks stay allocated. The test stalls the truncate's free on a refcount lock another node holds, crashes by closing the device, replays the journal, and checks that fsck -repair frees exactly the 2 cut blocks and that both files keep their data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fsck): cover an interrupted subdirectory unlink and either replay behavior The truncate test required replay to leave the cut blocks allocated. A shared helper now accepts whatever of the released blocks replay leaves: check must name exactly those, repair must free exactly those, and the allocated set must end as before minus the released blocks. The test keeps passing once replay frees these blocks itself. The new test crashes while unlinking a file in a subdirectory, after its entry is gone and before its blocks are freed. Replay deletes the inode and frees nothing for the BlockFree record, so the blocks leak until repair frees them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 12, 2026
… the inode still maps them (#78) * feat(journal): add an ExtentFree record that replay frees only while still mapped BlockFree records hold logical file blocks, so #65 made their replay free nothing, and an interrupted truncate or unlink leaks its blocks. ExtentFree records one freed extent: its file blocks and the disk blocks they map. Replay unmaps and frees a block only while the inode still maps that disk block at that file block. A block the inode no longer maps there was freed already, or holds a later write (another node may have taken the inode's lock after a crash), so replay leaves it alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fs): journal truncated and unlinked extents, and free blocks of failed writes and mkdirs Truncate and unlink in a subdirectory now journal an ExtentFree record for each extent they free, instead of a BlockFree with logical blocks, so replaying an interrupted truncate or unlink frees those blocks. At most 256 extents are journaled per operation; replay leaves any others allocated for fsck -repair. A write whose inode write fails now frees the blocks it allocated and its copy-on-write copies, and a mkdir whose inode write or directory setup fails frees the new directory's block. Both free only blocks the inode on disk does not map, since a failed inode write may have landed. The fsck leak tests from #72 now leak a block directly, as a failed write no longer does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: track the exact-free replay and failure tests in the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(journal): free logged extents only if the inode is unwritten since, through refcounts A peer can take a dead node's inode lock as stale after 15 s, before that node's journal is replayed, and change the file. With only the mapping check, a block the peer freed and the allocator later gave back at the same offset would be freed again by replay. ExtentFree now records the inode's MetaGeneration, which every inode write bumps, and replay frees nothing unless it still matches. One record per inode holds all its freed extents, so replay checks and writes the inode once. Replay frees through the extent manager's refcount-aware path, so a block shared with a snapshot or clone loses one reference instead of its bitmap bit. Mount now creates the refcount region before it replays its own journal so that path has refcounts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(journal): keep the extent free test inside the test device's blocks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(journal): check the freed file block reads as a hole Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fs): keep the errors import the subdirectory unlink test uses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fs): free an unlinked inode before its blocks and close a failed write's intent before freeing Replay of an unlink's ExtentFree passed both checks while the inode still existed after its blocks were freed, and a failed write freed blocks its open intent could map again. ExtentFree now also carries the size, so a replayed truncate cuts the file instead of leaving zeros past the cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fs): keep the reused-block test off the unlinked inode's slot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: track the crash-window and payload tests in the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The journal could checkpoint past an operation that was committed but not yet applied.
Commitremoved the transaction fromactiveTxnsbefore the operation ran. The 5 s checkpoint, the flush-journal endpoint from #57, a full journal and unmount (MarkClean) only checkedactiveTxns, so any of them could move the head past it. A crash in that window lost the half-done operation. #57's security review found it.Fix
CompleteIntent.Stopleaves the journal dirty while an intent is pending.CompleteIntentis idempotent. Each of the 10 commit sites defers it right after a successful commit. An operation that fails after committing therefore still closes its intent and is never replayed, which is what the 5 s checkpoint did before. ACommitwhose sync fails drops its intent, since its caller never runs the operation.S_IFREGwith the permission bits, as mkdir already journalsS_IFDIR. A replayed create had taken its type from the caller's mode, and fsck flagged it.BlockFreerecord now frees nothing. Truncate and subdirectory unlink log logical file blocks, and replay freed physical blocks with those numbers, starting at block 0, which the root directory uses.TestTruncateReplayKeepsMappedBlocksreproduced this: replay "freed blocks 1-2", which the truncated file still mapped. An interrupted free now leaves its blocks allocated, which fsck reports.Tests (written first; all three crash tests failed on main)
TestCreateReplaysAfterCheckpointAndCrashstops a create after its commit (through a test hook that is nil in production) and runs a checkpoint, then remounts with mount's replay handler. The file must exist and fsck must be clean.The journal tests check that a checkpoint keeps the incomplete intent, that completed intents are checkpointed away, and that unmount leaves an incomplete intent for replay.
There is no Docker or VM variant: a crash can't be timed into the millisecond window between commit and completion without the hook. The manifest records this.
The journal keeps one sector free (
AvailableSpace). Records are sector-aligned, so before this a ring that filled while an intent pinned the head ended exactly at the head. head == tail reads as empty, and the next write overwrote the pending intent.CommitandCompleteIntentnow check the sector-aligned sizeWriteAtwrites. Once the ring is full, a node getsErrJournalFull, and so EIO, until the pinning operation completes.Recovery stops when the tail sits exactly at the end of the data area. Before, the scan wrapped to 0 and never reached the tail, so a crash in that state hung the next mount. This predates fix(journal): keep the head behind intents whose operation has not completed #65.
Root mkdir journals only the permission bits with
S_IFDIR.What a failure after commit leaves behind
The deferred
CompleteIntentabandons these, as the 5 s checkpoint did before. fsck-repairfixes only refcounts and the superblock checksum. It reports everything below but repairs none of it.Unlink, rmdir and truncate also log and ignore some errors after their main step, then return success: orphan inodes, leaked blocks. Those intents complete either way.
Results
-race(journal and FS packages on cdb24a4; all packages on 00d6075 except two region tests updated for the free sector).-race(00d6075).-short: 127 pass, 0 fail (00d6075, .66).pkillpattern is longer than the kernel's 15-character process name. Fixed separately in fix(test): kill the exact mount process in crash tests and check it is gone #73.TestJournalCleanOnStopandTestCheckpointAdvancesHeadNowexpected a commit withoutCompleteIntentto be dropped, which is this bug. They now complete the intent first.🤖 Generated with Claude Code