Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
33fac6c
fix(refcount): count the existing owner when a block is first shared
mattmattox Oct 24, 2026
e91eab4
fix(extent): free shared blocks only after their last owner lets go
mattmattox Oct 24, 2026
2a50bf0
fix(snapshot): keep each snapshot's blocks in a hidden data inode
mattmattox Oct 24, 2026
8364b7d
test(snapshot): check snapshot data, clone writes and delete in all s…
mattmattox Oct 24, 2026
b003b70
fix(snapshot): free a snapshot's data inode after its descriptor is gone
mattmattox Oct 24, 2026
4b95080
test(snapshot): wait for deleted snapshots to disappear
mattmattox Oct 24, 2026
b76afaa
fix(extent): conflict-check copy-on-write and never free uncounted sh…
mattmattox Oct 24, 2026
9d2ba62
fix(snapshot): order sharing safely and refuse files shared without r…
mattmattox Oct 24, 2026
fe98dbd
fix(snapshot): lock the snapshot registry across nodes
mattmattox Oct 24, 2026
ec6b52b
fix(snapshot): never release uncounted blocks, and never roll back ac…
mattmattox Oct 24, 2026
e0d77d9
fix(snapshot): keep snapshot IDs and descriptor sectors apart per node
mattmattox Oct 24, 2026
20941cd
test(snapshot): check IDs after older images and node slot reuse
mattmattox Oct 24, 2026
6f7faff
fix(snapshot): copy shared blocks before taking the inode lock
mattmattox Oct 24, 2026
6ccaacd
test(snapshot): failed final write after a mid-write copy commit
mattmattox Oct 24, 2026
d70af72
fix(snapshot): log why a snapshot could not be added to the registry
mattmattox Oct 24, 2026
9b7105b
test(snapshot): skip the two-node concurrent snapshot test in short mode
mattmattox Oct 24, 2026
096987f
fix(inode): reject stale copies whose inline extents changed on disk
mattmattox Oct 24, 2026
1a6db24
fix(snapshot): retry copy-on-write on a stale tree and finish interru…
mattmattox Oct 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- Upgrade all nodes together: unmount on every node, upgrade, then mount again. Don't mount one shared device with a mix of old and new binaries.
- Older binaries reset an inode's generation counters to 0 when they free it, so other nodes can serve stale data after the inode number is reused.
- Older binaries zero the extent-tree generation (inode bytes 188–195) when they rewrite an inode. Newer binaries then cannot read that file if it has more than two extents, because such files use an extent tree.
- Run `goclusterfs-fsck` before the first mount after upgrading an image that has snapshots or clones. Phase 5 reports blocks shared by more files than their refcount allows.
- Older versions marked a file's blocks shared, on snapshot or clone, without counting the owners. This version never frees such blocks, so deleting those files leaks their space. It also refuses to snapshot or clone them (`ENOTSUP`), and in-place writes to an old clone still change its source. To get files this version can snapshot and free, copy each affected file with `cp` and delete the original.
- Snapshots made by older versions captured no data. They still list and delete, but cloning one fails with `ENOTSUP`.

### Added

Expand All @@ -26,7 +29,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- UDP multicast cache invalidation between nodes, with optional HMAC authentication (`-broadcast-secret`).
- Batched TRIM of freed blocks (`-trim`).
- SFB containers for files under 64 KB, disabled: every file uses normal extents and `-sfb` has no effect. A file that an older build stored in an SFB slot stays readable, moves to a full block on its next write or truncate, and frees its slot when deleted.
- Experimental: file snapshots and writable clones through `user.snapshot*` and `user.clone` extended attributes. Copy-on-write does not trigger yet, so a snapshot does not preserve data, and an in-place write to a clone changes the source file.
- Experimental: file snapshots and writable clones through `user.snapshot*` and `user.clone` extended attributes. Read a snapshot by cloning it with `user.snapshot.clone`.
- Experimental: `goclusterfs-fsck` consistency checker with a repair mode. It reports errors on healthy volumes: the superblock free counters are not updated after format, and it expects refcount and snapshot sector checksums the runtime does not write.
- Benchmark suite (`tests/benchmark`) and an `internal/metrics` package.
- Go, Docker, and QEMU VM test suites, with `tests/test-manifest.yaml` and `make test-parity` to track coverage across them.
Expand All @@ -53,6 +56,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- All nodes wrote the same journal region. A restarting node replayed and then reset the journal of nodes still writing, and no node could replay a crashed node's journal. Each node registry slot now owns a 4 MB journal, and a crashed node's journal is replayed through its slot.
- Nodes never sent registry heartbeats, so a node mounting more than 10 seconds after another took the live node's registry slot.
- After a partial write into a fallocated range, reads of the other fallocated blocks, and of the rest of the written block, returned old disk contents. They now return zeros.
- Snapshots and clones never copied on write: block refcounts stopped at 1, so a snapshot kept nothing and a write to a clone changed its source. Each snapshot now keeps the blocks it captured in a hidden inode, a write to a shared block copies it first, and a block is freed only when its last owner lets go.
- Deleting a snapshot after its file changed failed with a refcount underflow and left the snapshot deleting forever, because the delete used the file's current blocks.
- Writes failed with `EAGAIN` once a file needed a third extent, so most files over 2 MB could not be written. Extents past the first two now go in an on-disk extent tree, and adjacent extents merge.
- `goclusterfs-fsck` phase 4 never checked file extents, because it read inodes into unaligned buffers that `O_DIRECT` rejects.
- A node could serve a stale cached inode after another node freed and reused its inode number. Inode generations now survive free and reuse, and opening a file reloads its inode from disk.
Expand Down
11 changes: 4 additions & 7 deletions cmd/mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -266,12 +266,9 @@ func main() {
log.Println("SFB: disabled (all files use full 1MB blocks)")
}

// Create refcount region for snapshot CoW support
var refcountReg *refcount.Region
if *snapshotsEnabled {
refcountReg = refcount.NewRegion(dev, refcountOffset, totalBlocks, lockMgr)
log.Printf("Refcount: region at %dMB (%d blocks)", refcountOffset/(1024*1024), totalBlocks)
}
// Refcounts stay on without snapshots: blocks shared earlier still need copy-on-write.
refcountReg := refcount.NewRegion(dev, refcountOffset, totalBlocks, lockMgr)
log.Printf("Refcount: region at %dMB (%d blocks)", refcountOffset/(1024*1024), totalBlocks)

// Create filesystem with journal, TRIM, SFB, and refcount for crash consistency
gfs := goclusterfs.NewWithComponents(dev, bm, inodeTbl, extentMgr, dirMgr, cacheMgr, j, trimMgr, sfbMgr, refcountReg, lockMgr, nodeUUID, dataOffset)
Expand Down Expand Up @@ -307,7 +304,7 @@ func main() {

// Initialize snapshot subsystem after verifying filesystem is valid
if *snapshotsEnabled {
if err := gfs.InitSnapshots(ctx, snapshotOffset, snapshotSize); err != nil {
if err := gfs.InitSnapshots(ctx, snapshotOffset, snapshotSize, lockMgr, slotIdx); err != nil {
log.Fatalf("Failed to initialize snapshots: %v", err)
}
log.Printf("Snapshots: enabled (registry at %dMB, %dMB capacity)",
Expand Down
73 changes: 56 additions & 17 deletions internal/extent/extent.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ const (
FlagUnwritten = 1 << 1 // Allocated but lazy-zero (not yet written)
FlagShared = 1 << 2 // Shared with snapshot (requires CoW)
FlagSFB = 1 << 3 // Stored in SFB container (not full block)
// FlagCounted marks sharing that refcounts track. Older versions set
// FlagShared alone and never counted the owners.
FlagCounted = 1 << 6
)

// Hole is what LookupPhysical returns for a logical block with no data on disk.
Expand All @@ -66,6 +69,9 @@ type CoWResult struct {
OldPhysBlock uint64 // Original physical block (for refcount decrement)
NewPhysBlock uint64 // New physical block after CoW
WasShared bool // True if CoW was actually performed
// Uncounted means an older version shared the old block without counting
// its owners, so it must never be released.
Uncounted bool
}

// Extent represents a contiguous range of blocks.
Expand All @@ -86,10 +92,18 @@ type Manager struct {
leafCap int // extents per leaf before it splits
indexCap int // leaves per tree

refs *refcount.Region // nil when snapshots are disabled

deferredMu sync.Mutex
deferred []uint64 // frees postponed by lock order
}

// SetRefcounts makes frees respect block sharing: a block shared by a
// snapshot or clone goes back to the bitmap only after its last owner frees it.
func (m *Manager) SetRefcounts(refs *refcount.Region) {
m.refs = refs
}

// NewManager creates a new extent manager. dataOffset is the device offset of
// data block 0; extent tree nodes are stored in data blocks.
func NewManager(dev *storage.Device, bm *bitmap.Region, lockMgr *lock.Manager, dataOffset uint64) *Manager {
Expand Down Expand Up @@ -315,34 +329,51 @@ func (m *Manager) unmap(ctx context.Context, ino *inode.Inode, from, to uint64)

// release updates block accounting for removed extents and queues their
// blocks to be freed when the inode write commits: if that write loses a
// conflict, the inode on disk still maps them. Shared blocks stay allocated for
// the refcount system, and SFB extents point into containers rather than at
// bitmap blocks.
// conflict, the inode on disk still maps them. SFB extents point into
// containers rather than at bitmap blocks.
func (m *Manager) release(ctx context.Context, ino *inode.Inode, removed []Extent) {
var blocks []uint64
for _, x := range removed {
ino.AllocatedBlocks -= min(ino.AllocatedBlocks, x.Length)
if x.Flags&FlagUnwritten == 0 {
ino.WrittenBlocks -= min(ino.WrittenBlocks, x.Length)
}
if x.Flags&(FlagShared|FlagSFB) != 0 {
continue
}
for j := uint64(0); j < x.Length; j++ {
blocks = append(blocks, x.PhysicalBlock+j)
if x.Flags&FlagSFB == 0 {
blocks = append(blocks, m.ownedBlocks(x)...)
}
}
if len(blocks) == 0 {
return
}
p := pendingOf(ino)
if p == nil {
p = &pendingTree{base: committedTree(ino)}
ino.PendingTree = p
}
p := ensurePending(ino)
p.release = append(p.release, blocks...)
}

// ownedBlocks returns the blocks of x that frees may give back. Nobody knows
// how many inodes share an uncounted block, so it stays allocated.
func (m *Manager) ownedBlocks(x Extent) []uint64 {
if x.Flags&FlagShared != 0 && (x.Flags&FlagCounted == 0 || m.refs == nil) {
return nil
}
blocks := make([]uint64, x.Length)
for j := range blocks {
blocks[j] = x.PhysicalBlock + uint64(j)
}
return blocks
}

// HasUncountedSharing reports whether ino shares blocks the way older versions
// did, without refcounts. Such blocks cannot be shared again safely.
func (m *Manager) HasUncountedSharing(ino *inode.Inode) (bool, error) {
uncounted := false
err := m.forEach(ino, func(x Extent) {
if x.Flags&FlagShared != 0 && x.Flags&FlagCounted == 0 {
uncounted = true
}
})
return uncounted, err
}

// Delete unmaps every block of an inode that is being freed and frees the
// blocks now, since no inode write will follow to commit the change.
func (m *Manager) Delete(ctx context.Context, ino *inode.Inode) error {
Expand Down Expand Up @@ -529,7 +560,10 @@ func (m *Manager) CopyOnWrite(ctx context.Context, ino *inode.Inode, ext *Extent
if err != nil {
return nil, err
}
result := &CoWResult{OldPhysBlock: cur.PhysicalBlock + offset}
result := &CoWResult{
OldPhysBlock: cur.PhysicalBlock + offset,
Uncounted: cur.Flags&FlagShared != 0 && cur.Flags&FlagCounted == 0,
}
block := Extent{logicalBlock, result.OldPhysBlock, 1, cur.Flags}

needsCoW, err := m.NeedsCoW(ctx, &block, refcountReg)
Expand Down Expand Up @@ -564,12 +598,17 @@ func (m *Manager) CopyOnWrite(ctx context.Context, ino *inode.Inode, ext *Extent
return nil, fmt.Errorf("CoW write failed: %w", err)
}

private := Extent{logicalBlock, newPhysBlock, 1, cur.Flags &^ FlagShared}
private := Extent{logicalBlock, newPhysBlock, 1, cur.Flags &^ (FlagShared | FlagCounted)}
if err := m.remap(ctx, ino, private); err != nil {
m.bitmap.Free(ctx, newPhysBlock)
return nil, err
}
// The caller releases the old block after the inode write, so that write
// must fail if another node copied the block first.
ensurePending(ino)
*ext = private
// Readers keep cached block mappings until DataGeneration changes.
ino.DataGeneration++

result.NewPhysBlock = newPhysBlock
result.WasShared = true
Expand Down Expand Up @@ -599,7 +638,7 @@ func (m *Manager) MarkAllShared(ino *inode.Inode) error {
if err != nil {
return err
}
if err := e.updateAll(func(x *Extent) { x.Flags |= FlagShared }); err != nil {
if err := e.updateAll(func(x *Extent) { x.Flags |= FlagShared | FlagCounted }); err != nil {
return err
}
return e.commit()
Expand All @@ -615,7 +654,7 @@ func (m *Manager) CopyExtentsForClone(ctx context.Context, src *inode.Inode, dst
}
all := []Extent{}
if err := m.forEach(src, func(x Extent) {
x.Flags |= FlagShared
x.Flags |= FlagShared | FlagCounted
all = append(all, x)
}); err != nil {
return err
Expand Down
165 changes: 165 additions & 0 deletions internal/extent/extent_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package extent

import (
"context"
"errors"
"os"
"path/filepath"
"testing"
Expand Down Expand Up @@ -1507,3 +1508,167 @@ func listExtents(tb testing.TB, mgr *Manager, ino *inode.Inode) []*Extent {
}
return extents
}

func TestFreeKeepsSharedBlocks(t *testing.T) {
env := newTreeEnv(t, 0)
refs := refcount.NewRegion(env.dev, treeTestInodeOffset+512<<10, treeTestBlocks, nil)
env.mgr.SetRefcounts(refs)
initial := env.bm.FreeCount()

src := env.newInode()
if _, err := env.mgr.AllocateBlocks(env.ctx, src, 0, 2); err != nil {
t.Fatal(err)
}
if err := env.mgr.MarkAllShared(src); err != nil {
t.Fatal(err)
}
env.write(src)
clone := env.newInode()
if err := env.mgr.CopyExtentsForClone(env.ctx, src, clone); err != nil {
t.Fatal(err)
}
env.write(clone)
blocks, err := env.mgr.DataBlocks(src)
if err != nil {
t.Fatal(err)
}
if err := refs.Share(env.ctx, blocks); err != nil {
t.Fatal(err)
}

truncate := func(ino *inode.Inode) {
t.Helper()
if err := env.mgr.Truncate(env.ctx, ino, 0); err != nil {
t.Fatal(err)
}
env.write(ino)
}
truncate(src)
if got := env.bm.FreeCount(); got != initial-2 {
t.Fatalf("free blocks after truncating the source = %d, want %d: the clone still maps them", got, initial-2)
}
truncate(clone)
if got := env.bm.FreeCount(); got != initial {
t.Fatalf("free blocks after truncating both = %d, want %d", got, initial)
}

}

// legacyShared maps a new block into each inode the way older versions shared
// blocks: FlagShared without FlagCounted, and a refcount of count.
func legacyShared(t *testing.T, env *treeEnv, refs *refcount.Region, count uint32, inodes ...*inode.Inode) uint64 {
t.Helper()
blocks, err := env.bm.Alloc(env.ctx, 1)
if err != nil {
t.Fatal(err)
}
if err := refs.Set(env.ctx, blocks[0], count, refcount.FlagNone); err != nil {
t.Fatal(err)
}
for _, ino := range inodes {
env.insert(ino, Extent{LogicalBlock: 0, PhysicalBlock: blocks[0], Length: 1, Flags: FlagAllocated | FlagShared})
env.write(ino)
}
return blocks[0]
}

func TestUncountedSharingIsNeverFreed(t *testing.T) {
env := newTreeEnv(t, 0)
refs := refcount.NewRegion(env.dev, treeTestInodeOffset+512<<10, treeTestBlocks, nil)
env.mgr.SetRefcounts(refs)
truncate := func(ino *inode.Inode) {
t.Helper()
if err := env.mgr.Truncate(env.ctx, ino, 0); err != nil {
t.Fatal(err)
}
env.write(ino)
}

// An old clone left a count of 0; an old snapshot and then an old clone left 1.
for _, count := range []uint32{0, 1} {
src, clone := env.newInode(), env.newInode()
block := legacyShared(t, env, refs, count, src, clone)
for _, ino := range []*inode.Inode{src, clone} {
uncounted, err := env.mgr.HasUncountedSharing(ino)
if err != nil || !uncounted {
t.Fatalf("count %d: HasUncountedSharing = %v, %v; want true", count, uncounted, err)
}
}
truncate(clone)
if ok, _ := env.bm.IsAllocated(block); !ok {
t.Fatalf("count %d: truncating the old clone freed block %d that its source maps", count, block)
}
}

fresh := env.newInode()
if _, err := env.mgr.AllocateBlocks(env.ctx, fresh, 0, 1); err != nil {
t.Fatal(err)
}
if err := env.mgr.MarkAllShared(fresh); err != nil {
t.Fatal(err)
}
if uncounted, err := env.mgr.HasUncountedSharing(fresh); err != nil || uncounted {
t.Fatalf("sharing marked by this version reported as uncounted: %v, %v", uncounted, err)
}
}

// Two nodes copy the same shared block of an inline file: only the first
// inode write may land, so the old block is released once.
func TestConcurrentInlineCopyOnWrite(t *testing.T) {
env := newTreeEnv(t, 0)
refs := refcount.NewRegion(env.dev, treeTestInodeOffset+512<<10, treeTestBlocks, nil)
env.mgr.SetRefcounts(refs)

src := env.newInode()
ext, err := env.mgr.AllocateBlocks(env.ctx, src, 0, 1)
if err != nil {
t.Fatal(err)
}
old := ext.PhysicalBlock
if err := env.mgr.MarkAllShared(src); err != nil {
t.Fatal(err)
}
env.write(src)
if err := refs.Share(env.ctx, []uint64{old}); err != nil {
t.Fatal(err)
}
snap := env.newInode()
if err := env.mgr.CopyExtentsForClone(env.ctx, src, snap); err != nil {
t.Fatal(err)
}
env.write(snap)

nodeA, nodeB := env.reread(src), env.reread(src)
var copies []uint64
for _, ino := range []*inode.Inode{nodeA, nodeB} {
x, _, err := env.mgr.Lookup(ino, 0)
if err != nil {
t.Fatal(err)
}
res, err := env.mgr.CopyOnWrite(env.ctx, ino, x, 0, treeTestDataOffset, refs)
if err != nil || !res.WasShared {
t.Fatalf("copy-on-write: %+v, %v", res, err)
}
copies = append(copies, res.NewPhysBlock)
}

env.write(nodeA)
env.mgr.ReleaseBlocks(env.ctx, []uint64{old})
if err := env.tbl.Write(env.ctx, nodeB); !errors.Is(err, ErrTreeConflict) {
t.Fatalf("second copy's inode write: got %v, want ErrTreeConflict", err)
}
env.mgr.DiscardPending(env.ctx, nodeB)
if err := env.bm.Free(env.ctx, copies[1]); err != nil {
t.Fatal(err)
}

if ok, _ := env.bm.IsAllocated(old); !ok {
t.Fatalf("block %d freed while the snapshot still maps it", old)
}
if e, err := refs.Get(env.ctx, old); err != nil || e.RefCount != 1 {
t.Fatalf("old block refcount = %+v, %v; want 1 (the snapshot)", e, err)
}
if got, _, err := env.mgr.Lookup(env.reread(src), 0); err != nil || got.PhysicalBlock != copies[0] {
t.Fatalf("file maps %+v, %v; want the first copy %d", got, err, copies[0])
}
}
Loading
Loading