Skip to content

feat(superblock)!: size the bitmap, refcount and inode regions from the device - #47

Merged
mattmattox merged 15 commits into
mainfrom
claude/layout-v3
Nov 10, 2026
Merged

mattmattox merged 15 commits into
mainfrom
claude/layout-v3

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Format v3 sizes the block bitmap, refcount table and inode table from the device. In v2 each was 1 MB, which meant at most 4096 files. Refcounts for blocks past 128 GB also overwrote the snapshot registry and then data, and above 8 TB the bitmap overwrote the inode table. Mount and fsck now refuse v2 images and say to copy the data off and reformat.

Offset Size Region
0 1 MB Superblock
1 MB 1 MB Node registry
2 MB 1 MB Lock records
3 MB 1 MB Snapshot registry
4 MB 256 MB Journals, 64 × 4 MB
260 MB 1 MB per 8 TB Block bitmap
next about 8 MB per TB Refcounts
next 256 B per inode Inode table
next rest Data

Decisions to review:

  • Fixed-size regions come first, so their offsets never depend on the device size.
  • -bytes-per-inode defaults to 1 MB, not 64 KB. Every file with data takes at least one 1 MB block, so extra inodes only help empty files. statfs also reads the whole inode table: 50 MB at 200 GB with the default, 800 MB with 64 KB. The count is clamped to 4096–16M inodes (a 4 GB table at most).
  • Mount checks the layout structurally instead of recomputing it: regions must be in v3 order and big enough for the stored block and inode counts. A device that grew after format still mounts.
  • FindFreeSlot and statfs now read the inode table 1 MB at a time instead of 4 KB per inode.
  • fix(refcount): never read or write past the refcount region #39's mount warning can't fire on v3, so I removed it. Its bound check stays, and its test now uses an undersized region directly.
  • Docker and VM clusters format with -bytes-per-inode 65536.

A create costs about 130 ms on main too (about 840 reads per create), so 4200 creates take 9 minutes on one node. TestLargeVolume (200 GB, 4 nodes, high-block snapshot, remount, fsck) and the Docker/VM tests split the creates across nodes. TestLargeVolume skips in -short.

After review (a2ef830): CheckLayout bounds every region inside the device and requires InodeCount to fill the table. fsck runs the same check and reports an error. A destroyed v3 superblock is recognized by its node registry slots, and mount says not to format. Format refuses a -bytes-per-inode that leaves no data blocks. FreeCount scans without the table lock.

Results:

  • a2ef830 on .66: unit -race 18/18; FuzzDeserialize and FuzzCheck 60 s each; integration -short 112/0; layout and fsck integration tests 6/0, including TestLargeVolume. Lint is clean.
  • 3e1e220: all 10 CI checks pass, and the Docker groups take 5m31s, 4m36s and 12m43s. Docker on .66 was 51/0 and VM on .68 was 56/0 (both on b68f038).

The Docker and VM 4200-file tests now poll up to 60 s for other nodes to see the new directories and files, instead of sleeping 1.5 s. Without that, a Docker run lost one node's 1100 files to the 1 s attribute cache. In CI the test takes 13m38s, so it runs alone as Docker group 3.

The branch merges main instead of rebasing, so the review history stays intact.

🤖 Generated with Claude Code

mattmattox and others added 10 commits October 24, 2026 18:57
…he device

Layout v2 gave the block bitmap, refcount table and inode table 1 MB each.
That holds 4096 inodes, and refcounts for blocks past 128 GB landed in
the snapshot registry and then in data blocks; the bitmap overflows into
the inode table past 8 TB. Layout v3 puts the fixed-size regions first and
sizes the other three at format time: the bitmap and refcount table cover
every block the device can hold, and the inode table gets one inode per
-bytes-per-inode bytes (default 1 MB, one per data block), between 4096
and 16M inodes. Mount checks the region order and that each region is big
enough for the stored block and inode counts, so a grown device still
mounts. Mount and fsck refuse v2 volumes and say to copy the data off and
reformat.

The inode table is now up to 4 GB, so FindFreeSlot and FreeCount (statfs)
read it 1 MB at a time instead of one sector per inode. fsck reports a
refcount table too small for the volume's blocks.

BREAKING CHANGE: v2 volumes no longer mount; reformat with -format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestMoreThan4096Files formats with -bytes-per-inode 65536 and keeps 4200
files across a remount. TestLargeVolumeSnapshotsHighBlocks fallocates
past block 130816, where v2's refcount table ended, then snapshots,
copies on write and clones data there on a 200 GB sparse image. Both end
with a clean fsck. TestMountRefusesLayoutV2 checks that mount and fsck
say to reformat a v2 volume. FormatAndMount now passes ExtraArgs too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A create costs about 130 ms, so TestLargeVolume has four nodes fill their
own directories at once, on the 200 GB image, and skips in -short mode;
it replaces TestMoreThan4096Files and TestLargeVolumeSnapshotsHighBlocks.
Docker and VM clusters now format with -bytes-per-inode 65536 (12288 and
16384 inodes), and test_more_than_4096_files creates and removes 4200
files there the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Layout v3 sizes the refcount table for every block, so mount can no longer
see a table smaller than the volume; drop the warning #39 added for that.
Its test now builds an undersized region directly instead of formatting a
200 GB v2 volume. The superblock package computes sector capacities itself
and a test checks them against bitmap and refcount, because a refcount
test imports superblock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n error

#39 already reports a table without entries for every block, as a
warning, since v2 volumes had one. Drop the second check added with v3
and make #39's an error: v3 mount refuses such a volume. Its test now
shrinks the table in the superblock, since v3 never formats one that
small.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le test

Docker run on 4455dfb: node 2 could not cd into a directory node 1 had
just made, because its attribute cache still held the old listing, so
1100 files were never created. Each node now polls up to 60 s for its
directories before creating files, and the final count polls up to 60 s
instead of sleeping 1.5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's parity check failed: the manifest still named
TestBlocksPastRegionLeaveTheSnapshotRegistryAlone. Also describe both
refcount-past-region scenarios the way their tests now set them up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It took 13m38s in CI, which pushed group 1 to 19m56s against the
orchestrator's 20-minute wait. Alone in group 3 it runs next to the
other two groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox and others added 4 commits November 10, 2026 12:28
…perblock

Review of #47 found two ways the v3 checks fail open:
- CheckLayout added region sizes without bounding them, so a hand-made
  superblock with a bitmap size near 2^64 wrapped the next offset back to
  1 MB and put the refcount table on the node registry. Each region must
  now end inside the device, and InodeCount must fill the inode table
  exactly. The fuzzer runs CheckLayout and checks that accepted regions
  are disjoint and inside the device.
- A v3 device with an overwritten superblock was reported as unformatted,
  telling the operator to run -format. The old probe looks for an inode
  table at 2 MB, where v3 keeps lock records. Node registry slots at 1 MB
  now identify a v3 device, and mount says not to format it.

Format refuses a -bytes-per-inode that leaves no data blocks instead of
writing a volume the next mount refuses, and the v1 and v2 version
errors now explain each version separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fsck accepted superblocks mount refuses, such as a swapped bitmap and
refcount table, an inode count below the table, or a gap before the data.
Phase 1 now runs the same CheckLayout as mount and reports a failure as
an error, then keeps checking. The fsck unit tests build real v3
superblocks instead of a hand-made 16-node layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every statfs, not only df -i, scans the whole inode table: 50 MB at 200 GB.
Holding t.mu for that stalled inode writes on this node, and other nodes
change the table without it anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It slept 1.5 s before counting files the other nodes made; the Docker
and VM versions already poll. Also shorten the comments this PR added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the bitmap and refcount offsets swapped, phase 5 reads the real
bitmap as refcount sectors, which pass the same checksum, and repair
rewrites them as refcounts. Repair now refuses (exit 8) after phase 1
reports a layout error, and a dry run says it would refuse.

The destroyed-superblock message now says -format is fine when the
device was wiped on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit df997d6 into main Nov 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant