Skip to content

fix: data-safety bugs in fallocate, block 0 reads, and SFB - #12

Merged
mattmattox merged 2 commits into
mainfrom
claude/fix-data-safety-bugs
Oct 24, 2026
Merged

mattmattox merged 2 commits into
mainfrom
claude/fix-data-safety-bugs

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Fixes four data-safety bugs. Each has a regression test that fails on cd52194 and passes here.

  • SFB is now disabled. New writes never allocate SFB slots, so every file uses extents and -sfb has no effect. SFB could not allocate in any build: the container header buffer is stack-allocated, so it isn't aligned for O_DIRECT. Files that older builds put in slots can still be read, move to a full block on their next write or truncate, and give their slot back when deleted. "Make SFB work end to end or remove it" is tracked separately.
  • Fallocate: the first partial write to a fallocated block now zero-fills the rest of that block. The other blocks in the range stay unwritten and read as zeros.
  • Test TODOs: the Docker and VM tests for fallocate are TODO. They would need to reuse a deleted file's blocks before TRIM clears them (5s by default), so they would be flaky. The block 0 and SFB cases can't be reached from FUSE, so those tests run in Go only.

Not fixed, found while working on this:

  • A partial write to a newly allocated block that wasn't fallocated can still expose old disk data, for example a sparse write in the middle of a block.
  • allocate() re-reads the inode after taking the inode lock, which drops the extents it just added. This only happens when lockMgr is set.
  • sfb.Manager.writeContainerHeader zeroes the rest of the header sector, which holds the first 4032 bytes of slot 0.

🤖 Generated with Claude Code

mattmattox and others added 2 commits October 24, 2026 11:04
- The first write to a fallocated block cleared the Unwritten flag on
  the whole extent, so its other blocks returned old disk contents. It
  now splits off only the written block and zero-fills it before a
  partial write, so the rest of that block reads as zeros too.
- LookupPhysical returned 0 for holes, so File.Read treated data in
  physical block 0 as a hole. Holes are now extent.Hole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SFB could not allocate a slot in any build: the container header buffer
is not aligned for O_DIRECT. Had it worked, headers would land 11 MB
too low and File.Read would treat the packed container/slot value as a
block number. Writes no longer try to allocate slots, so every file
uses extents and -sfb has no effect.

For SFB inodes written by older builds:
- File.Read reads the slot with aligned I/O, or returns EIO if the
  size does not fit the slot.
- Any write or truncate first moves the data to a full block with
  aligned I/O, and fails instead of mapping a block without the data.
- Unlink and migration free the slot after the inode stops using it,
  with header I/O at the correct offset. A failure leaks the slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit c312a28 into main Oct 24, 2026
6 checks passed
mattmattox added a commit that referenced this pull request Oct 24, 2026
Since #12, the first write to an unwritten block clears only that block, which
can split one extent into three and allocate extent-tree blocks. Under the
live inode lock that allocation failed the lock order check, so a write into
the middle of a fallocated range returned EIO. The flag is now cleared on an
unlocked copy; the write keeps it only if the inode is unchanged once the lock
is held, and retries otherwise. The lock stays held from that check through
the zero-fill, the data write and the inode commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Oct 24, 2026
* fix(lock): stop hash collisions and same-node races from creating double holders

The lock region has 256 records in the production layout, and tryAcquire
treated a record held for a different resource as free. Any node could take
over a held lock by touching a resource that hashed to the same record.

Other double-holder windows fixed here:
- two goroutines on one node acquiring the same key at once left the
  on-disk record cleared while both believed they held it
- releasing a shared lock cleared the record for all readers when the
  releasing node was the last one to join
- an exclusive request piggybacked on a same-node shared hold
- reclaiming a stale shared record kept the dead readers' count
- a release whose CAS lost a race returned nil without releasing

Records now act as lock stripes: a held record blocks every resource that
hashes to it, a node can hold several keys on one record, and a per-record
gate serializes this node's on-disk acquire and release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): document lock stall hazard and safe operating limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): wire the lock manager into mounted filesystems

cmd/mount built the FS with NewWithComponents, which had no lock manager
parameter, so every inode, extent and SFB lock branch in internal/goclusterfs
was dead in production. Two nodes writing a new file's first block at the same
time lost one node's data in 11 of 20 files.

The dead branches had never run and lost data once enabled: after allocating a
block without the inode lock, writeInternal and Allocate re-read the inode
under the lock and dropped the new extent. They now map the new block in the
fresh inode, or free it and retry if another node mapped that block first.
Changes made under the lock are written before the lock is dropped for the
next allocation, and the final size update builds on the inode read under the
lock.

Readers joining a shared lock no longer refresh its age. A reader count leaked
by a lost CAS update now goes stale after 15s instead of blocking writers for
as long as readers keep arriving; Docker hung on rmdir without this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: verify first-write lost updates from a restarted reader

The VM runner deletes every file before each test, so new files reuse inode
numbers and a node that read earlier files serves stale cached extents. That
made the VM check fail even though no write was lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): age records through a test helper now that writeRecord is gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): stop same-node writers waiting on their own readers; renew busy records

An exclusive request waited for this node's shared holders of the record.
inode.Table holds its mutex across Acquire, and a directory listing holds the
directory's shared lock while it reads the table, so creating a file while the
directory was listed hung the whole mount. Exclusive requests now join the
node's existing hold, as they did before this branch. The record on disk stays
shared, so other nodes can read but not write meanwhile.

Same-node joins never touched disk, so a record kept busy by overlapping keys
kept its first AcquireTime and was taken over as stale after 15s while still
held. A key joining a hold older than renewAfter now rewrites AcquireTime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): drop the inode lock before freeing blocks and releasing tree reservations

The bitmap lock comes before the inode lock, so frees made while holding it
fail the lock order check. Tree reservations released under the inode lock
parked their blocks in an in-memory list that unmount drops: appends merging
into a full inline extent leaked one block each. writeFailed freed the write's
blocks the same way. Both now run after the inode lock is released.

writeInternal released the inode lock it took for the size update, then
released it again in its deferred cleanup. If another goroutine on the node
had joined that lock in between, the second release dropped its hold. All
releases now go through one helper that clears lockHeld.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): stop hash collisions and same-node races from creating double holders

The lock region has 256 records in the production layout, and tryAcquire
treated a record held for a different resource as free. Any node could take
over a held lock by touching a resource that hashed to the same record.

Other double-holder windows fixed here:
- two goroutines on one node acquiring the same key at once left the
  on-disk record cleared while both believed they held it
- releasing a shared lock cleared the record for all readers when the
  releasing node was the last one to join
- an exclusive request piggybacked on a same-node shared hold
- reclaiming a stale shared record kept the dead readers' count
- a release whose CAS lost a race returned nil without releasing

Records now act as lock stripes: a held record blocks every resource that
hashes to it, a node can hold several keys on one record, and a per-record
gate serializes this node's on-disk acquire and release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): document lock stall hazard and safe operating limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): wire the lock manager into mounted filesystems

cmd/mount built the FS with NewWithComponents, which had no lock manager
parameter, so every inode, extent and SFB lock branch in internal/goclusterfs
was dead in production. Two nodes writing a new file's first block at the same
time lost one node's data in 11 of 20 files.

The dead branches had never run and lost data once enabled: after allocating a
block without the inode lock, writeInternal and Allocate re-read the inode
under the lock and dropped the new extent. They now map the new block in the
fresh inode, or free it and retry if another node mapped that block first.
Changes made under the lock are written before the lock is dropped for the
next allocation, and the final size update builds on the inode read under the
lock.

Readers joining a shared lock no longer refresh its age. A reader count leaked
by a lost CAS update now goes stale after 15s instead of blocking writers for
as long as readers keep arriving; Docker hung on rmdir without this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: verify first-write lost updates from a restarted reader

The VM runner deletes every file before each test, so new files reuse inode
numbers and a node that read earlier files serves stale cached extents. That
made the VM check fail even though no write was lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lock): age records through a test helper now that writeRecord is gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): stop same-node writers waiting on their own readers; renew busy records

An exclusive request waited for this node's shared holders of the record.
inode.Table holds its mutex across Acquire, and a directory listing holds the
directory's shared lock while it reads the table, so creating a file while the
directory was listed hung the whole mount. Exclusive requests now join the
node's existing hold, as they did before this branch. The record on disk stays
shared, so other nodes can read but not write meanwhile.

Same-node joins never touched disk, so a record kept busy by overlapping keys
kept its first AcquireTime and was taken over as stale after 15s while still
held. A key joining a hold older than renewAfter now rewrites AcquireTime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): drop the inode lock before freeing blocks and releasing tree reservations

The bitmap lock comes before the inode lock, so frees made while holding it
fail the lock order check. Tree reservations released under the inode lock
parked their blocks in an in-memory list that unmount drops: appends merging
into a full inline extent leaked one block each. writeFailed freed the write's
blocks the same way. Both now run after the inode lock is released.

writeInternal released the inode lock it took for the size update, then
released it again in its deferred cleanup. If another goroutine on the node
had joined that lock in between, the second release dropped its hold. All
releases now go through one helper that clears lockHeld.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): clear unwritten flags outside the inode lock

Since #12, the first write to an unwritten block clears only that block, which
can split one extent into three and allocate extent-tree blocks. Under the
live inode lock that allocation failed the lock order check, so a write into
the middle of a fallocated range returned EIO. The flag is now cleared on an
unlocked copy; the write keeps it only if the inode is unchanged once the lock
is held, and retries otherwise. The lock stays held from that check through
the zero-fill, the data write and the inode commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lock): wait for same-node readers again; never hold a local mutex across Acquire

Letting an exclusive request join this node's shared hold kept the record
shared on disk, so two nodes that both read a directory could each take it
exclusively and lose each other's entries. Exclusive requests wait for this
node's readers again.

The hang that change avoided came from local mutexes held across Acquire: a
writer held inode.Table's mutex while waiting for a reader that needed it.
inode.Table, bitmap, refcount and the snapshot registry now take the
distributed lock before their mutex.

Renewal now covers exclusive holds only, so a dead node's reader count no
longer stays fresh. A join that finds its record taken over fails with
ErrLockLost, and later joins to that hold fail too. Pending tree changes are
discarded after the inode lock is dropped on every error path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(operations): note that read locks are not renewed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fs): keep blocks a mid-write persist committed when the final inode write fails

Before allocating its next block, a write persists the inode, which maps the
blocks allocated so far. If the final inode write then failed, writeFailed
freed every block of the write, including the ones the inode on disk still
mapped, so the retry handed them out again. It now frees only allocations and
copy-on-write copies made after the last persist, and drops the refcounts of
copies that persist already committed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant