Skip to content

feat(fsck): free leaked blocks in -repair - #72

Merged
mattmattox merged 5 commits into
mainfrom
claude/fsck-reclaim-leaks
Nov 12, 2026
Merged

mattmattox merged 5 commits into
mainfrom
claude/fsck-reclaim-leaks

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

-repair now frees blocks the bitmap marks allocated that no inode maps, as data or extent-tree nodes, and no SFB extent points into. Several paths leave these behind by design: failed writes, frees that hit #54's 30s lock wait, interrupted truncates and unlinks after #65, and SFB migrations. With SFB disabled, an empty container now counts as leaked.

Safety

  • A leaked block's refcount is set to 0 and synced before its bitmap bit is cleared, so a crash in between only leaves a leak for the next run.
  • The reference map now counts every extent with a length, whatever its flags, and the in-range part of extents that run past the device end, because the runtime maps all of them.
  • Repair keeps every leaked block and its refcount, and says why, when the bitmap, inode or refcount phase didn't finish (including -skip-phase), an inode or extent tree is unreadable, or a node's journal needs replay. That holds even with -force. Blocks in bitmap sectors with a bad checksum are also kept.
  • Phase 5 no longer rebuilds refcounts when -skip-phase skipped the bitmap or inode phase. On main, that rebuild sees no references and can zero the counts of shared blocks.
  • Repair still refuses while a node is mounted, and it re-reads the registry before writing.

Tests (unit tests on .66, -race)

  • Leaks from a failed write, timed-out frees, and a cloned pair whose block keeps a refcount of 2, alongside a snapshot and clone. Repair frees exactly those blocks, and a second repair changes nothing. Against main's fsck, repair leaves 4 leaked blocks.
  • Each of these keeps the leaks, and leaves the bitmap and the refcount region unchanged: an unreadable inode, a damaged extent tree, a skipped bitmap or inode phase, a bad bitmap sector, a mounted node, and a dirty journal under -force. Removing any one guard fails a test.
  • An extent without FlagAllocated, and one past the device end, keep their blocks.
  • A container emptied by an SFB migration is freed, while one still in use is kept.

VM (.68, d779f90): Power Failure Simulation, Concurrent Writes Power Failure, and the new test_fsck_repair_after_power_failure run twice, all pass (4/4, 254s). Neither power failure left a leaked block, so in VMs the test shows repair is safe after a real power failure: exit 0, a clean check afterwards, and every file unchanged on both nodes. The unit tests cover the reclaim itself.

Pending

🤖 Generated with Claude Code

mattmattox and others added 5 commits November 12, 2026 02:07
Failed writes and mkdirs, frees that give up on the 30s lock wait,
interrupted truncates and unlinks, and SFB migrations leave blocks
allocated that no inode maps. fsck only reported them.

-repair now frees every allocated block that no inode maps as data or
extent-tree nodes and no SFB extent points into. Empty SFB containers
count as leaked, since SFB allocation is disabled. Phase 5 first sets a
leaked block's refcount to 0 and syncs, then phase 8 clears its bitmap
bit, one write per bitmap sector, so a crash in between leaves a leak
the next run frees.

Repair keeps every leaked block, and says why, when the bitmap, inode
or refcount phase did not finish, an inode or extent tree could not be
read, or a node's journal still needs replay, even with -force. It
keeps those in bitmap sectors with a bad checksum. It still refuses to
run while a node has the filesystem mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The harness copies goclusterfs-fsck to the VMs when it is built, and
the VM make targets build it. The new test kills a VM while it writes
files, unmounts every node, runs fsck -repair inside a VM, and checks
that the check after it reports no leaked blocks and that a file
written before the failure is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n reclaim is blocked

Review of #72 found repair freeing blocks a file still maps. Phase 3
skipped extents without FlagAllocated and whole extents that run past
the device end, but the runtime maps every extent with a length. It now
counts the in-range blocks of each.

Phase 5 cleared leaked blocks' refcounts even when phase 8 then kept
the blocks, and with -skip-phase 2 or 3 its rebuild saw no references
and could zero the counts of shared blocks. It now clears leak counts
only when reclaim will run, rebuilds only when the bitmap and inode
phases finished, and reports at most 10 leaks by name.

Leaks in bitmap sectors with a bad checksum are reported apart, so the
listed leaks are marked repaired when repair frees them. The VM test
also checks the victim's files survive repair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two runs found no leaked blocks after the power failure, so the test
shows repair is safe after one rather than that it reclaims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 05153f4 into main Nov 12, 2026
11 checks passed
mattmattox added a commit that referenced this pull request Nov 12, 2026
… the inode still maps them (#78)

* feat(journal): add an ExtentFree record that replay frees only while still mapped

BlockFree records hold logical file blocks, so #65 made their replay
free nothing, and an interrupted truncate or unlink leaks its blocks.
ExtentFree records one freed extent: its file blocks and the disk blocks
they map. Replay unmaps and frees a block only while the inode still
maps that disk block at that file block. A block the inode no longer
maps there was freed already, or holds a later write (another node may
have taken the inode's lock after a crash), so replay leaves it alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): journal truncated and unlinked extents, and free blocks of failed writes and mkdirs

Truncate and unlink in a subdirectory now journal an ExtentFree record
for each extent they free, instead of a BlockFree with logical blocks,
so replaying an interrupted truncate or unlink frees those blocks. At
most 256 extents are journaled per operation; replay leaves any others
allocated for fsck -repair.

A write whose inode write fails now frees the blocks it allocated and
its copy-on-write copies, and a mkdir whose inode write or directory
setup fails frees the new directory's block. Both free only blocks the
inode on disk does not map, since a failed inode write may have landed.
The fsck leak tests from #72 now leak a block directly, as a failed
write no longer does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: track the exact-free replay and failure tests in the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(journal): free logged extents only if the inode is unwritten since, through refcounts

A peer can take a dead node's inode lock as stale after 15 s, before that
node's journal is replayed, and change the file. With only the mapping
check, a block the peer freed and the allocator later gave back at the
same offset would be freed again by replay. ExtentFree now records the
inode's MetaGeneration, which every inode write bumps, and replay frees
nothing unless it still matches. One record per inode holds all its
freed extents, so replay checks and writes the inode once.

Replay frees through the extent manager's refcount-aware path, so a block
shared with a snapshot or clone loses one reference instead of its bitmap
bit. Mount now creates the refcount region before it replays its own
journal so that path has refcounts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(journal): keep the extent free test inside the test device's blocks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(journal): check the freed file block reads as a hole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fs): keep the errors import the subdirectory unlink test uses

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): free an unlinked inode before its blocks and close a failed write's intent before freeing

Replay of an unlink's ExtentFree passed both checks while the inode still
existed after its blocks were freed, and a failed write freed blocks its
open intent could map again. ExtentFree now also carries the size, so a
replayed truncate cuts the file instead of leaving zeros past the cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fs): keep the reused-block test off the unlinked inode's slot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: track the crash-window and payload tests in the manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 13, 2026
* fix(broadcast): count multicast failures in a row and retry after a pause

The failure counter never reset, so after 3 failed multicast sends in
total over a mount, a node sent invalidations only to unicast peers
until it unmounted. Now a successful send resets the count, and after
MaxMulticastFailures failures in a row the node sends only to unicast
peers for 30 seconds before it tries multicast again. The total stays
in TransportStats; /metrics does not export broadcast counters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(journal): remove the unused FullThreshold setting

Nothing read it. A checkpoint already runs every 5 seconds and is
forced when a transaction does not fit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): log that SFB is disabled and stop creating its unused manager

-sfb defaulted to true and logged "SFB: enabled", but nothing used the
SFB manager it created: writes never allocate slots, and reading,
migrating and freeing old slots does not go through it. Mount now logs
what is in effect and still accepts -sfb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: raise the integration test timeout and fix stale help text

make test-integration ran the full suite with a 5 minute timeout; it
takes about 15 minutes, so allow 30 and say so in make help.

goclusterfs-fsck -h said repair never frees blocks or deletes inodes,
which changed with #72 and #86, and the api package doc called the API
read-only, which changed with #57.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant