Skip to content

fix(refcount): never read or write past the refcount region - #39

Merged
mattmattox merged 2 commits into
mainfrom
claude/refcount-region-bounds
Oct 24, 2026
Merged

mattmattox merged 2 commits into
mainfrom
claude/refcount-region-bounds

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

On volumes over about 128 GiB, refcount updates for high blocks wrote past the 1 MB refcount region. A snapshot, a clone or a plain delete of a file with data above block 130816 silently rewrote snapshot registry bytes, because registry sectors use the same CRC32C format. Above about 255.5 GiB the writes land in data blocks.

Proven on main with a 200 GiB sparse image:

  • Increment(130816) succeeded and wrote over the registry's first sector.
  • Release changed a registry descriptor without returning an error.

The new test fails nine ways on main.

Now blocks past the region read as single-owner. Snapshots and clones of them fail with ErrBlockPastRegion, and deletes keep working. The real fix, sizing the region at format time, is a layout change that will be folded into the v3 layout work.

Refused snapshots and clones return ENOTSUP and leave the source file unmarked.

Volumes used with older builds: on a volume over about 128 GiB where snapshots or clones were taken before this change, counts for blocks 130816 and up went into the snapshot registry. Those blocks now read as having one owner, so deleting one copy of such a file can free a block the other copy still maps. fsck can't see this yet; the follow-up PR adds that check.

🤖 Generated with Claude Code

mattmattox and others added 2 commits October 24, 2026 17:56
The refcount region is 1 MB, which holds entries for 130816 blocks
(127.75 GiB). Region bounded block numbers only by the volume's block
count, so on larger volumes the sectors of higher blocks fell in the
snapshot registry and, past about 255.5 GiB, in data blocks. Registry
sectors use the same CRC32C format, so a snapshot or clone of a high
block, or a plain delete, rewrote registry bytes without an error.

NewRegion now takes the region size. Blocks past it read as having a
single owner and are never written; sharing them fails with
ErrBlockPastRegion. Scan and Initialize stay inside the region, and
mount warns when the volume has more blocks than the region covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shareInto marked the source shared before Share refused blocks past the
refcount region, so after a refusal every write to the file took the
copy-on-write check. Check that the region has entries for all source
blocks first, and report the refusal as ENOTSUP like uncounted sharing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 0077dcc into main Oct 24, 2026
8 checks passed
mattmattox added a commit that referenced this pull request Nov 10, 2026
…he device (#47)

* feat(superblock)!: size the bitmap, refcount and inode regions from the device

Layout v2 gave the block bitmap, refcount table and inode table 1 MB each.
That holds 4096 inodes, and refcounts for blocks past 128 GB landed in
the snapshot registry and then in data blocks; the bitmap overflows into
the inode table past 8 TB. Layout v3 puts the fixed-size regions first and
sizes the other three at format time: the bitmap and refcount table cover
every block the device can hold, and the inode table gets one inode per
-bytes-per-inode bytes (default 1 MB, one per data block), between 4096
and 16M inodes. Mount checks the region order and that each region is big
enough for the stored block and inode counts, so a grown device still
mounts. Mount and fsck refuse v2 volumes and say to copy the data off and
reformat.

The inode table is now up to 4 GB, so FindFreeSlot and FreeCount (statfs)
read it 1 MB at a time instead of one sector per inode. fsck reports a
refcount table too small for the volume's blocks.

BREAKING CHANGE: v2 volumes no longer mount; reformat with -format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(integration): cover layout v3 with 4200 files and a 200 GB volume

TestMoreThan4096Files formats with -bytes-per-inode 65536 and keeps 4200
files across a remount. TestLargeVolumeSnapshotsHighBlocks fallocates
past block 130816, where v2's refcount table ended, then snapshots,
copies on write and clones data there on a 200 GB sparse image. Both end
with a clean fsck. TestMountRefusesLayoutV2 checks that mount and fsck
say to reformat a v2 volume. FormatAndMount now passes ExtraArgs too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: create 4200 files from every node, also in Docker and VM

A create costs about 130 ms, so TestLargeVolume has four nodes fill their
own directories at once, on the 200 GB image, and skips in -short mode;
it replaces TestMoreThan4096Files and TestLargeVolumeSnapshotsHighBlocks.
Docker and VM clusters now format with -bytes-per-inode 65536 (12288 and
16384 inodes), and test_more_than_4096_files creates and removes 4200
files there the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(refcount): test the region bound without the v2 layout

Layout v3 sizes the refcount table for every block, so mount can no longer
see a table smaller than the volume; drop the warning #39 added for that.
Its test now builds an undersized region directly instead of formatting a
200 GB v2 volume. The superblock package computes sector capacities itself
and a test checks them against bitmap and refcount, because a refcount
test imports superblock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(superblock): check the grown device's Close error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fsck): report a refcount table smaller than the volume once, as an error

#39 already reports a table without entries for every block, as a
warning, since v2 volumes had one. Drop the second check added with v3
and make #39's an error: v3 mount refuses such a volume. Its test now
shrinks the table in the superblock, since v3 never formats one that
small.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(test): wait for other nodes to see the directories in the 4200-file test

Docker run on 4455dfb: node 2 could not cd into a directory node 1 had
just made, because its attribute cache still held the old listing, so
1100 files were never created. Each node now polls up to 60 s for its
directories before creating files, and the final count polls up to 60 s
instead of sleeping 1.5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(test): point the refcount bound scenario at its renamed test

CI's parity check failed: the manifest still named
TestBlocksPastRegionLeaveTheSnapshotRegistryAlone. Also describe both
refcount-past-region scenarios the way their tests now set them up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(docker): run the 4200-file test in its own group

It took 13m38s in CI, which pushed group 1 to 19m56s against the
orchestrator's 20-minute wait. Alone in group 3 it runs next to the
other two groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(superblock): bound region offsets and recognize a destroyed v3 superblock

Review of #47 found two ways the v3 checks fail open:
- CheckLayout added region sizes without bounding them, so a hand-made
  superblock with a bitmap size near 2^64 wrapped the next offset back to
  1 MB and put the refcount table on the node registry. Each region must
  now end inside the device, and InodeCount must fill the inode table
  exactly. The fuzzer runs CheckLayout and checks that accepted regions
  are disjoint and inside the device.
- A v3 device with an overwritten superblock was reported as unformatted,
  telling the operator to run -format. The old probe looks for an inode
  table at 2 MB, where v3 keeps lock records. Node registry slots at 1 MB
  now identify a v3 device, and mount says not to format it.

Format refuses a -bytes-per-inode that leaves no data blocks instead of
writing a volume the next mount refuses, and the v1 and v2 version
errors now explain each version separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fsck): report layouts that mount refuses

fsck accepted superblocks mount refuses, such as a swapped bitmap and
refcount table, an inode count below the table, or a gap before the data.
Phase 1 now runs the same CheckLayout as mount and reports a failure as
an error, then keeps checking. The fsck unit tests build real v3
superblocks instead of a hand-made 16-node layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(inode): count free inodes without the table lock

Every statfs, not only df -i, scans the whole inode table: 50 MB at 200 GB.
Holding t.mu for that stalled inode writes on this node, and other nodes
change the table without it anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: poll for other nodes' files in TestLargeVolume

It slept 1.5 s before counting files the other nodes made; the Docker
and VM versions already poll. Also shorten the comments this PR added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fsck): refuse to repair a layout mount refuses

With the bitmap and refcount offsets swapped, phase 5 reads the real
bitmap as refcount sectors, which pass the same checksum, and repair
rewrites them as refcounts. Repair now refuses (exit 8) after phase 1
reports a layout error, and a dry run says it would refuse.

The destroyed-superblock message now says -format is fine when the
device was wiped on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant