Skip to content

fix(journal): keep the head behind intents whose operation has not completed - #65

Merged
mattmattox merged 10 commits into
mainfrom
claude/journal-checkpoint-intents
Nov 12, 2026
Merged

mattmattox merged 10 commits into
mainfrom
claude/journal-checkpoint-intents

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The journal could checkpoint past an operation that was committed but not yet applied. Commit removed the transaction from activeTxns before the operation ran. The 5 s checkpoint, the flush-journal endpoint from #57, a full journal and unmount (MarkClean) only checked activeTxns, so any of them could move the head past it. A crash in that window lost the half-done operation. #57's security review found it.

Fix

  • The journal keeps each committed transaction pending until CompleteIntent.
  • All checkpoints go through one function. It stops at the oldest active or pending transaction, measuring ring distance from the head so wraparound works, and syncs the device before moving the head.
  • Stop leaves the journal dirty while an intent is pending.
  • CompleteIntent is idempotent. Each of the 10 commit sites defers it right after a successful commit. An operation that fails after committing therefore still closes its intent and is never replayed, which is what the 5 s checkpoint did before. A Commit whose sync fails drops its intent, since its caller never runs the operation.
  • Both create paths now journal S_IFREG with the permission bits, as mkdir already journals S_IFDIR. A replayed create had taken its type from the caller's mode, and fsck flagged it.
  • Replay of a BlockFree record now frees nothing. Truncate and subdirectory unlink log logical file blocks, and replay freed physical blocks with those numbers, starting at block 0, which the root directory uses. TestTruncateReplayKeepsMappedBlocks reproduced this: replay "freed blocks 1-2", which the truncated file still mapped. An interrupted free now leaves its blocks allocated, which fsck reports.

Tests (written first; all three crash tests failed on main)

  • TestCreateReplaysAfterCheckpointAndCrash stops a create after its commit (through a test hook that is nil in production) and runs a checkpoint, then remounts with mount's replay handler. The file must exist and fsck must be clean.

  • The journal tests check that a checkpoint keeps the incomplete intent, that completed intents are checkpointed away, and that unmount leaves an incomplete intent for replay.

  • There is no Docker or VM variant: a crash can't be timed into the millisecond window between commit and completion without the hook. The manifest records this.

  • The journal keeps one sector free (AvailableSpace). Records are sector-aligned, so before this a ring that filled while an intent pinned the head ended exactly at the head. head == tail reads as empty, and the next write overwrote the pending intent. Commit and CompleteIntent now check the sector-aligned size WriteAt writes. Once the ring is full, a node gets ErrJournalFull, and so EIO, until the pinning operation completes.

  • Recovery stops when the tail sits exactly at the end of the data area. Before, the scan wrapped to 0 and never reached the tail, so a crash in that state hung the next mount. This predates fix(journal): keep the head behind intents whose operation has not completed #65.

  • Root mkdir journals only the permission bits with S_IFDIR.

What a failure after commit leaves behind

The deferred CompleteIntent abandons these, as the 5 s checkpoint did before. fsck -repair fixes only refcounts and the superblock checksum. It reports everything below but repairs none of it.

Operation Fails after commit when Left on disk Would replay fix it?
create (2 paths) directory add fails nothing n/a
inode claim or write fails Usually nothing, because the entry and inode are rolled back. If that rollback also fails, a dangling entry or half-written inode stays, which fsck reports. No: replay skips inodes and entries that already exist
mkdir (2 paths) as create, or inode write fails after the directory block is set up the new directory's 1 MB block leaks even when the rollback works No: replay creates no directory block
unlink, rmdir (3 paths) only the directory remove can fail nothing n/a
write inode write fails (not a tree conflict) The inode is unchanged; the blocks allocated for this write and its copy-on-write copies leak. Partly: replay would map the logged extents, completing a write the caller was told failed
truncate, setattr inode write fails Nothing: frees only run once the inode write lands, and new tree blocks are freed. No: inode-update replay is a no-op
SFB migration inode write fails the new block leaks No: replay skips SFB records
rename not implemented

Unlink, rmdir and truncate also log and ignore some errors after their main step, then return success: orphan inodes, leaked blocks. Those intents complete either way.

Results

  • Unit tests pass with -race (journal and FS packages on cdb24a4; all packages on 00d6075 except two region tests updated for the free sector).
  • The crash and ring tests pass 10 out of 10 with -race (00d6075).
  • Integration -short: 127 pass, 0 fail (00d6075, .66).
  • Lint is clean (cdb24a4).
  • VM, full suite at a load average of 285–306 on .68:

TestJournalCleanOnStop and TestCheckpointAdvancesHeadNow expected a commit without CompleteIntent to be dropped, which is this bug. They now complete the intent first.

🤖 Generated with Claude Code

mattmattox and others added 10 commits November 12, 2026 00:34
Commit takes a transaction out of activeTxns before the operation runs,
so a checkpoint can move the head past an intent that is committed but
not yet applied, and a crash then loses the operation. These tests stop
between Commit and CompleteIntent, checkpoint, and check the next mount
replays the intent: in the journal alone, through a create on the
filesystem followed by fsck, and across an unmount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpleted

Commit took a transaction out of activeTxns before the operation it logs
ran, so the 5 s checkpoint, the flush-journal endpoint, a full journal or
unmount could move the head past a committed intent that was not yet
applied. A crash in that window lost the half-done operation.

The journal now keeps each committed transaction pending until
CompleteIntent. Checkpoints stop at the oldest active or pending
transaction and sync the device first, so every change they pass is
durable. Unmount leaves the journal dirty while an intent is pending.

CompleteIntent is idempotent, and each operation defers it right after
its commit, so an operation that fails after committing still closes its
intent and is never replayed, as before. A Commit whose sync fails drops
its intent, since its caller never runs the operation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both create paths logged the mode FUSE passed, while the runtime writes
S_IFREG with the permission bits. A replayed create therefore depended on
the caller's mode for the inode type, and fsck reported "invalid file
type" when it lacked S_IFREG. Mkdir already logs S_IFDIR this way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n stop

TestJournalCleanOnStop and TestCheckpointAdvancesHeadNow committed
without CompleteIntent and expected the journal to drop the transaction,
which is the bug. They now complete the intent first, and the checkpoint
test also checks that the head does not move before completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Truncate journals the logical block where the freed range starts, and
replay frees physical blocks by that number. A crash between a
truncate's commit and its changes then makes replay free blocks that
files still map. The crash hook now covers setattr too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Truncate and unlink log the logical file blocks they free, and replay
freed physical blocks with those numbers. Replaying an interrupted
truncate or unlink in a subdirectory therefore freed blocks files still
map, starting at block 0, which the root directory uses. Replay now
frees nothing for a BlockFree record: an interrupted free leaves its
blocks allocated, which fsck reports, instead of freeing blocks in use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records are sector-aligned and WriteAt accepted one that used the last
free byte, so a ring filled while a pending intent pinned the head ended
with head == tail, which recovery reads as empty. The next write then
treated the whole ring as free and overwrote the pending intent.
AvailableSpace now keeps one sector free, and Commit and CompleteIntent
check the sector-aligned size WriteAt writes before deciding to
checkpoint.

Recovery also never finished when the tail sat at the end of the data
area: the scan wrapped to 0 and kept looking for it. It now stops there.
Mkdir in the root now journals only the permission bits with S_IFDIR,
as mkdir in subdirectories does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit fbf1ca8 into main Nov 12, 2026
11 checks passed
mattmattox added a commit that referenced this pull request Nov 12, 2026
…links leave (#75)

* test(fsck): free the blocks an interrupted truncate leaves

A node crashes after a truncate writes the shorter inode but before it
frees the cut blocks. Since #65, replay frees nothing for that record,
so the blocks stay allocated. The test stalls the truncate's free on a
refcount lock another node holds, crashes by closing the device,
replays the journal, and checks that fsck -repair frees exactly the 2
cut blocks and that both files keep their data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fsck): cover an interrupted subdirectory unlink and either replay behavior

The truncate test required replay to leave the cut blocks allocated. A
shared helper now accepts whatever of the released blocks replay leaves:
check must name exactly those, repair must free exactly those, and the
allocated set must end as before minus the released blocks. The test
keeps passing once replay frees these blocks itself.

The new test crashes while unlinking a file in a subdirectory, after its
entry is gone and before its blocks are freed. Replay deletes the inode
and frees nothing for the BlockFree record, so the blocks leak until
repair frees them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 12, 2026
… the inode still maps them (#78)

* feat(journal): add an ExtentFree record that replay frees only while still mapped

BlockFree records hold logical file blocks, so #65 made their replay
free nothing, and an interrupted truncate or unlink leaks its blocks.
ExtentFree records one freed extent: its file blocks and the disk blocks
they map. Replay unmaps and frees a block only while the inode still
maps that disk block at that file block. A block the inode no longer
maps there was freed already, or holds a later write (another node may
have taken the inode's lock after a crash), so replay leaves it alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): journal truncated and unlinked extents, and free blocks of failed writes and mkdirs

Truncate and unlink in a subdirectory now journal an ExtentFree record
for each extent they free, instead of a BlockFree with logical blocks,
so replaying an interrupted truncate or unlink frees those blocks. At
most 256 extents are journaled per operation; replay leaves any others
allocated for fsck -repair.

A write whose inode write fails now frees the blocks it allocated and
its copy-on-write copies, and a mkdir whose inode write or directory
setup fails frees the new directory's block. Both free only blocks the
inode on disk does not map, since a failed inode write may have landed.
The fsck leak tests from #72 now leak a block directly, as a failed
write no longer does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: track the exact-free replay and failure tests in the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(journal): free logged extents only if the inode is unwritten since, through refcounts

A peer can take a dead node's inode lock as stale after 15 s, before that
node's journal is replayed, and change the file. With only the mapping
check, a block the peer freed and the allocator later gave back at the
same offset would be freed again by replay. ExtentFree now records the
inode's MetaGeneration, which every inode write bumps, and replay frees
nothing unless it still matches. One record per inode holds all its
freed extents, so replay checks and writes the inode once.

Replay frees through the extent manager's refcount-aware path, so a block
shared with a snapshot or clone loses one reference instead of its bitmap
bit. Mount now creates the refcount region before it replays its own
journal so that path has refcounts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(journal): keep the extent free test inside the test device's blocks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(journal): check the freed file block reads as a hole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fs): keep the errors import the subdirectory unlink test uses

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): free an unlinked inode before its blocks and close a failed write's intent before freeing

Replay of an unlink's ExtentFree passed both checks while the inode still
existed after its blocks were freed, and a failed write freed blocks its
open intent could map again. ExtentFree now also carries the size, so a
replayed truncate cuts the file instead of leaving zeros past the cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fs): keep the reused-block test off the unlinked inode's slot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: track the crash-window and payload tests in the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant