Repository navigation
feat(fsck): free leaked blocks in -repair - #72
Merged
Merged
Conversation
Failed writes and mkdirs, frees that give up on the 30s lock wait, interrupted truncates and unlinks, and SFB migrations leave blocks allocated that no inode maps. fsck only reported them. -repair now frees every allocated block that no inode maps as data or extent-tree nodes and no SFB extent points into. Empty SFB containers count as leaked, since SFB allocation is disabled. Phase 5 first sets a leaked block's refcount to 0 and syncs, then phase 8 clears its bitmap bit, one write per bitmap sector, so a crash in between leaves a leak the next run frees. Repair keeps every leaked block, and says why, when the bitmap, inode or refcount phase did not finish, an inode or extent tree could not be read, or a node's journal still needs replay, even with -force. It keeps those in bitmap sectors with a bad checksum. It still refuses to run while a node has the filesystem mounted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The harness copies goclusterfs-fsck to the VMs when it is built, and the VM make targets build it. The new test kills a VM while it writes files, unmounts every node, runs fsck -repair inside a VM, and checks that the check after it reports no leaked blocks and that a file written before the failure is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n reclaim is blocked Review of #72 found repair freeing blocks a file still maps. Phase 3 skipped extents without FlagAllocated and whole extents that run past the device end, but the runtime maps every extent with a length. It now counts the in-range blocks of each. Phase 5 cleared leaked blocks' refcounts even when phase 8 then kept the blocks, and with -skip-phase 2 or 3 its rebuild saw no references and could zero the counts of shared blocks. It now clears leak counts only when reclaim will run, rebuilds only when the bitmap and inode phases finished, and reports at most 10 leaks by name. Leaks in bitmap sectors with a bad checksum are reported apart, so the listed leaks are marked repaired when repair frees them. The VM test also checks the victim's files survive repair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two runs found no leaked blocks after the power failure, so the test shows repair is safe after one rather than that it reclaims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 12, 2026
… the inode still maps them (#78) * feat(journal): add an ExtentFree record that replay frees only while still mapped BlockFree records hold logical file blocks, so #65 made their replay free nothing, and an interrupted truncate or unlink leaks its blocks. ExtentFree records one freed extent: its file blocks and the disk blocks they map. Replay unmaps and frees a block only while the inode still maps that disk block at that file block. A block the inode no longer maps there was freed already, or holds a later write (another node may have taken the inode's lock after a crash), so replay leaves it alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fs): journal truncated and unlinked extents, and free blocks of failed writes and mkdirs Truncate and unlink in a subdirectory now journal an ExtentFree record for each extent they free, instead of a BlockFree with logical blocks, so replaying an interrupted truncate or unlink frees those blocks. At most 256 extents are journaled per operation; replay leaves any others allocated for fsck -repair. A write whose inode write fails now frees the blocks it allocated and its copy-on-write copies, and a mkdir whose inode write or directory setup fails frees the new directory's block. Both free only blocks the inode on disk does not map, since a failed inode write may have landed. The fsck leak tests from #72 now leak a block directly, as a failed write no longer does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: track the exact-free replay and failure tests in the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(journal): free logged extents only if the inode is unwritten since, through refcounts A peer can take a dead node's inode lock as stale after 15 s, before that node's journal is replayed, and change the file. With only the mapping check, a block the peer freed and the allocator later gave back at the same offset would be freed again by replay. ExtentFree now records the inode's MetaGeneration, which every inode write bumps, and replay frees nothing unless it still matches. One record per inode holds all its freed extents, so replay checks and writes the inode once. Replay frees through the extent manager's refcount-aware path, so a block shared with a snapshot or clone loses one reference instead of its bitmap bit. Mount now creates the refcount region before it replays its own journal so that path has refcounts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(journal): keep the extent free test inside the test device's blocks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(journal): check the freed file block reads as a hole Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fs): keep the errors import the subdirectory unlink test uses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fs): free an unlinked inode before its blocks and close a failed write's intent before freeing Replay of an unlink's ExtentFree passed both checks while the inode still existed after its blocks were freed, and a failed write freed blocks its open intent could map again. ExtentFree now also carries the size, so a replayed truncate cuts the file instead of leaving zeros past the cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fs): keep the reused-block test off the unlinked inode's slot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: track the crash-window and payload tests in the manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
mattmattox
added a commit
that referenced
this pull request
Nov 13, 2026
* fix(broadcast): count multicast failures in a row and retry after a pause The failure counter never reset, so after 3 failed multicast sends in total over a mount, a node sent invalidations only to unicast peers until it unmounted. Now a successful send resets the count, and after MaxMulticastFailures failures in a row the node sends only to unicast peers for 30 seconds before it tries multicast again. The total stays in TransportStats; /metrics does not export broadcast counters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(journal): remove the unused FullThreshold setting Nothing read it. A checkpoint already runs every 5 seconds and is forced when a transaction does not fit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(mount): log that SFB is disabled and stop creating its unused manager -sfb defaulted to true and logged "SFB: enabled", but nothing used the SFB manager it created: writes never allocate slots, and reading, migrating and freeing old slots does not go through it. Mount now logs what is in effect and still accepts -sfb. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: raise the integration test timeout and fix stale help text make test-integration ran the full suite with a 5 minute timeout; it takes about 15 minutes, so allow 30 and say so in make help. goclusterfs-fsck -h said repair never frees blocks or deletes inodes, which changed with #72 and #86, and the api package doc called the API read-only, which changed with #57. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
-repairnow frees blocks the bitmap marks allocated that no inode maps, as data or extent-tree nodes, and no SFB extent points into. Several paths leave these behind by design: failed writes, frees that hit #54's 30s lock wait, interrupted truncates and unlinks after #65, and SFB migrations. With SFB disabled, an empty container now counts as leaked.Safety
-skip-phase), an inode or extent tree is unreadable, or a node's journal needs replay. That holds even with-force. Blocks in bitmap sectors with a bad checksum are also kept.-skip-phaseskipped the bitmap or inode phase. On main, that rebuild sees no references and can zero the counts of shared blocks.Tests (unit tests on .66,
-race)-force. Removing any one guard fails a test.FlagAllocated, and one past the device end, keep their blocks.VM (.68, d779f90): Power Failure Simulation, Concurrent Writes Power Failure, and the new
test_fsck_repair_after_power_failurerun twice, all pass (4/4, 254s). Neither power failure left a leaked block, so in VMs the test shows repair is safe after a real power failure: exit 0, a clean check afterwards, and every file unchanged on both nodes. The unit tests cover the reclaim itself.Pending
🤖 Generated with Claude Code