Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- Inode cache with generation-based coherency checks, and an extent cache.
- UDP multicast cache invalidation between nodes, with optional HMAC authentication (`-broadcast-secret`).
- Batched TRIM of freed blocks (`-trim`).
- Experimental: SFB containers for files under 64 KB (`-sfb`). Container I/O uses unaligned buffers that `O_DIRECT` rejects, so files fall back to normal extents.
- SFB containers for files under 64 KB, disabled: every file uses normal extents and `-sfb` has no effect. A file that an older build stored in an SFB slot stays readable, moves to a full block on its next write or truncate, and frees its slot when deleted.
- Experimental: file snapshots and writable clones through `user.snapshot*` and `user.clone` extended attributes. Copy-on-write does not trigger yet, so a snapshot does not preserve data, and an in-place write to a clone changes the source file.
- Experimental: `goclusterfs-fsck` consistency checker with a repair mode. It fails on newly formatted volumes because `-format` does not write a superblock.
- Benchmark suite (`tests/benchmark`) and an `internal/metrics` package.
Expand All @@ -45,6 +45,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Fixed

- After a partial write into a fallocated range, reads of the other fallocated blocks, and of the rest of the written block, returned old disk contents. They now return zeros.
- Writes failed with `EAGAIN` once a file needed a third extent, so most files over 2 MB could not be written. Extents past the first two now go in an on-disk extent tree, and adjacent extents merge.
- `goclusterfs-fsck` phase 4 never checked file extents, because it read inodes into unaligned buffers that `O_DIRECT` rejects.
- A node could serve a stale cached inode after another node freed and reused its inode number. Inode generations now survive free and reuse, and opening a file reloads its inode from disk.
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ fusermount3 -u mnt # or Ctrl+C the mo
| `internal/cache/broadcast` | UDP multicast/unicast cache invalidation, optional HMAC |
| `internal/refcount` | Per-block reference counts for copy-on-write |
| `internal/snapshot` | Snapshot registry and background refcount worker |
| `internal/sfb` | Small-file containers: 4/8/16/64 KB slots inside a 1 MB block |
| `internal/sfb` | Small-file containers (4/8/16/64 KB slots in a 1 MB block). Disabled: nothing allocates slots, `goclusterfs` only reads and migrates files older builds stored there |
| `internal/trim` | Batched `BLKDISCARD` for freed blocks |
| `internal/fsck` | The fsck phases |
| `internal/metrics` | Atomic counters, not used by `cmd/mount` yet |
Expand Down Expand Up @@ -91,7 +91,7 @@ Clone paths are absolute from the filesystem root, and clones can only go in the
- Every node uses the same 4 MB journal region at 5 MB. `cmd/mount` does not give each node its own journal slot.
- Each file holds at most 2 extents (inline in the inode). The overflow extent tree is being built.
- The inode count is fixed at 4096.
- SFB containers fail on real devices: `internal/sfb` writes buffers that are not 4 KB aligned, so `storage.WriteAt` rejects them and small files fall back to full 1 MB blocks.
- SFB is disabled: writes never allocate SFB slots, so every file uses full 1 MB blocks. `internal/sfb` container I/O is broken (unaligned header buffers, no data offset) and stays until it is fixed or removed.
- No rename, symlinks, or hard links.

## Testing
Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod
- **Network Cache Invalidation**: UDP multicast for near-instant cross-node cache updates
- **Copy-on-Write Snapshots**: O(1) file snapshots with refcount-based block sharing
- **Thin Provisioning**: Allocate-on-write with TRIM/BLKDISCARD support
- **Small File Optimization**: SFB containers pack files under 64KB efficiently
- **8-Phase fsck**: Comprehensive filesystem consistency checker with repair mode

## Requirements
Expand Down Expand Up @@ -111,7 +110,7 @@ These are the `goclusterfs-mount` flags. Run `./goclusterfs-mount -h` for the fu

| Flag | Default | Description |
|------|---------|-------------|
| `-sfb` | `true` | Enable Small File Block containers |
| `-sfb` | `true` | Ignored. SFB containers are disabled and every file uses full blocks |
| `-snapshots` | `true` | Enable O(1) file snapshots |

## Multi-Node Setup
Expand Down
12 changes: 2 additions & 10 deletions docs/API-CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ These flags are passed when starting `goclusterfs`:

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `-sfb` | bool | `true` | Enable Small File Block containers |
| `-sfb` | bool | `true` | Ignored. SFB containers are disabled |
| `-snapshots` | bool | `true` | Enable O(1) file snapshots |

---
Expand Down Expand Up @@ -550,15 +550,7 @@ Optimize for consistency:

### Small Files (< 64KB)

Enable SFB optimization:

```bash
./goclusterfs -device /dev/device -mount /mnt/fs \
-sfb \
-journal
```

SFB (Small File Block) containers pack multiple small files into single blocks, reducing fragmentation and improving locality.
SFB containers are disabled, so small files use full 1 MB blocks. `-sfb` is accepted and ignored.

### Large File Streaming

Expand Down
4 changes: 1 addition & 3 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,8 +368,7 @@ Good for general workloads:
./goclusterfs -device /dev/device -mount /mnt/fs \
-journal \
-cache-ttl 5s \
-trim \
-sfb
-trim
```

### High Consistency
Expand Down Expand Up @@ -405,7 +404,6 @@ Disable advanced features for debugging:
-journal=false \
-broadcast=false \
-trim=false \
-sfb=false \
-snapshots=false \
-debug
```
Expand Down
2 changes: 1 addition & 1 deletion internal/cache/extent_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ func (m *Manager) LookupPhysicalCached(extMgr *extent.Manager, ino *inode.Inode,
return 0, err
}

// Cache the result (including holes which have phys=0)
// Cache the result (including holes, which map to extent.Hole)
m.CacheExtent(ino.InodeNum, logicalBlock, phys, ino.DataGeneration)

return phys, nil
Expand Down
42 changes: 25 additions & 17 deletions internal/extent/extent.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ const (
FlagSFB = 1 << 3 // Stored in SFB container (not full block)
)

// Hole is what LookupPhysical returns for a logical block with no data on disk.
// Physical block 0 is a valid data block.
const Hole = math.MaxUint64

// Errors
var (
ErrExtentNotFound = errors.New("extent not found")
Expand Down Expand Up @@ -125,12 +129,12 @@ func (m *Manager) Lookup(ino *inode.Inode, logicalBlock uint64) (*Extent, uint64
return &ext, logicalBlock - ext.LogicalBlock, nil
}

// LookupPhysical returns the physical block number for a logical block.
// Returns 0 if the block is a hole (not allocated).
// LookupPhysical returns the physical block number for a logical block, or Hole
// if the block is not allocated.
func (m *Manager) LookupPhysical(ino *inode.Inode, logicalBlock uint64) (uint64, error) {
ext, offset, err := m.Lookup(ino, logicalBlock)
if err == ErrExtentNotFound {
return 0, nil // Hole - return 0
return Hole, nil
}
if err != nil {
return 0, err
Expand All @@ -139,7 +143,7 @@ func (m *Manager) LookupPhysical(ino *inode.Inode, logicalBlock uint64) (uint64,
// Unwritten extents return zeros like holes for read operations.
// The extent exists (space is allocated) but data hasn't been written yet.
if ext.Flags&FlagUnwritten != 0 {
return 0, nil // Lazy-zero: treat as hole for reads
return Hole, nil
}

return ext.PhysicalBlock + offset, nil
Expand Down Expand Up @@ -226,30 +230,34 @@ func contiguousRuns(logicalBlock uint64, blocks []uint64, flags uint32) []Extent
return runs
}

// ClearUnwritten clears the Unwritten flag for an extent and increments WrittenBlocks.
// Called on first write to an unwritten block.
// Returns true if the flag was cleared (first write), false if already written.
func (m *Manager) ClearUnwritten(ino *inode.Inode, ext *Extent) (bool, error) {
// ClearUnwritten marks one block of an unwritten extent as written, splitting
// the extent so its other blocks still read as zeros. Called on first write to
// the block. Returns true if the block was unwritten, false if already written.
func (m *Manager) ClearUnwritten(ctx context.Context, ino *inode.Inode, logicalBlock uint64) (bool, error) {
ext, _, err := m.Lookup(ino, logicalBlock)
if err != nil {
return false, err
}
if ext.Flags&FlagUnwritten == 0 {
return false, nil
}
e, err := m.edit(context.Background(), ino)
e, err := m.edit(ctx, ino)
if err != nil {
return false, err
}
var length uint64
found, err := e.update(ext.LogicalBlock, func(x *Extent) {
x.Flags &^= FlagUnwritten
length = x.Length
})
if err != nil || !found {
removed, err := e.removeRange(logicalBlock, logicalBlock+1)
if err != nil {
return false, err
}
block := removed[0]
block.Flags &^= FlagUnwritten
if err := e.insert(block); err != nil {
return false, err
}
if err := e.commit(); err != nil {
return false, err
}
ext.Flags &^= FlagUnwritten
ino.WrittenBlocks += length
ino.WrittenBlocks++
return true, nil
}

Expand Down
82 changes: 37 additions & 45 deletions internal/extent/extent_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -425,13 +425,13 @@ func TestLookupPhysical(t *testing.T) {
t.Errorf("Physical = %d, want %d", phys, expected)
}

// Lookup hole (not allocated) should return 0
// Lookup hole (not allocated) should return Hole
phys, err = mgr.LookupPhysical(ino, 5)
if err != nil {
t.Fatalf("LookupPhysical hole failed: %v", err)
}
if phys != 0 {
t.Errorf("Physical for hole = %d, want 0", phys)
if phys != Hole {
t.Errorf("Physical for hole = %d, want Hole", phys)
}
}

Expand Down Expand Up @@ -928,9 +928,9 @@ func TestAllocateUnwritten_DoesNotIncrementWrittenBlocks(t *testing.T) {
}
}

// TestClearUnwritten_ClearsFlagAndIncrementsWritten verifies that ClearUnwritten
// properly clears the flag and increments WrittenBlocks.
func TestClearUnwritten_ClearsFlagAndIncrementsWritten(t *testing.T) {
// TestClearUnwritten_ClearsOnlyThatBlock verifies that writing one block of a
// multi-block unwritten extent leaves the other blocks reading as holes.
func TestClearUnwritten_ClearsOnlyThatBlock(t *testing.T) {
dev, cleanup := createTestDevice(t, 10*1024*1024)
defer cleanup()

Expand All @@ -939,46 +939,39 @@ func TestClearUnwritten_ClearsFlagAndIncrementsWritten(t *testing.T) {
mgr := NewManager(dev, bm, nil, 0)
ctx := context.Background()

ino := &inode.Inode{
Magic: inode.InodeMagic,
InodeNum: 100,
AllocatedBlocks: 0,
WrittenBlocks: 0,
}

// Allocate unwritten extent
ext, err := mgr.AllocateUnwritten(ctx, ino, 0, 1)
ino := &inode.Inode{Magic: inode.InodeMagic, InodeNum: 100}
ext, err := mgr.AllocateUnwritten(ctx, ino, 0, 3)
if err != nil {
t.Fatalf("AllocateUnwritten failed: %v", err)
}

// Verify starting state
if ino.WrittenBlocks != 0 {
t.Fatalf("Expected WrittenBlocks=0 before clear, got %d", ino.WrittenBlocks)
if ext.Length != 3 {
t.Fatalf("want one 3-block extent, got length %d", ext.Length)
}

// Clear unwritten flag
cleared, err := mgr.ClearUnwritten(ino, ext)
if err != nil {
t.Fatalf("ClearUnwritten failed: %v", err)
cleared, err := mgr.ClearUnwritten(ctx, ino, 1)
if err != nil || !cleared {
t.Fatalf("ClearUnwritten(1) = %v, %v; want true, nil", cleared, err)
}
if !cleared {
t.Error("ClearUnwritten should return true for first clear")
for lb, want := range []uint64{Hole, ext.PhysicalBlock + 1, Hole} {
if got, err := mgr.LookupPhysical(ino, uint64(lb)); err != nil || got != want {
t.Errorf("LookupPhysical(%d) = %d, %v; want %d", lb, got, err, want)
}
}

// Verify FlagUnwritten is now cleared
if ext.Flags&FlagUnwritten != 0 {
t.Errorf("Expected FlagUnwritten to be cleared, got flags=%d", ext.Flags)
if ino.WrittenBlocks != 1 || ino.AllocatedBlocks != 3 {
t.Errorf("WrittenBlocks=%d AllocatedBlocks=%d, want 1 and 3", ino.WrittenBlocks, ino.AllocatedBlocks)
}

// FlagAllocated should still be set
if ext.Flags&FlagAllocated == 0 {
t.Errorf("Expected FlagAllocated to remain set, got flags=%d", ext.Flags)
for _, lb := range []uint64{0, 2} {
if cleared, err := mgr.ClearUnwritten(ctx, ino, lb); err != nil || !cleared {
t.Fatalf("ClearUnwritten(%d) = %v, %v; want true, nil", lb, cleared, err)
}
}

// Verify WrittenBlocks was incremented
if ino.WrittenBlocks != 1 {
t.Errorf("Expected WrittenBlocks=1 after clear, got %d", ino.WrittenBlocks)
exts, err := mgr.ListExtents(ino)
if err != nil {
t.Fatal(err)
}
if len(exts) != 1 || exts[0].Length != 3 || exts[0].Flags != FlagAllocated {
t.Errorf("written blocks should merge back into one 3-block extent, got %d extents", len(exts))
}
}

Expand Down Expand Up @@ -1014,7 +1007,7 @@ func TestClearUnwritten_NoOpIfAlreadyWritten(t *testing.T) {
initialWrittenBlocks := ino.WrittenBlocks

// ClearUnwritten should be no-op
cleared, err := mgr.ClearUnwritten(ino, ext)
cleared, err := mgr.ClearUnwritten(ctx, ino, 0)
if err != nil {
t.Fatalf("ClearUnwritten failed: %v", err)
}
Expand All @@ -1029,9 +1022,9 @@ func TestClearUnwritten_NoOpIfAlreadyWritten(t *testing.T) {
}
}

// TestLookupPhysical_ReturnsZeroForUnwritten verifies that LookupPhysical
// returns 0 (hole) for unwritten extents.
func TestLookupPhysical_ReturnsZeroForUnwritten(t *testing.T) {
// TestLookupPhysical_ReturnsHoleForUnwritten verifies that LookupPhysical
// returns Hole for unwritten extents.
func TestLookupPhysical_ReturnsHoleForUnwritten(t *testing.T) {
dev, cleanup := createTestDevice(t, 10*1024*1024)
defer cleanup()

Expand All @@ -1051,13 +1044,13 @@ func TestLookupPhysical_ReturnsZeroForUnwritten(t *testing.T) {
t.Fatalf("AllocateUnwritten failed: %v", err)
}

// LookupPhysical should return 0 (treat as hole for reads)
// LookupPhysical should return Hole (treat as hole for reads)
physBlock, err := mgr.LookupPhysical(ino, 0)
if err != nil {
t.Fatalf("LookupPhysical failed: %v", err)
}
if physBlock != 0 {
t.Errorf("Expected physBlock=0 for unwritten extent, got %d", physBlock)
if physBlock != Hole {
t.Errorf("Expected Hole for unwritten extent, got %d", physBlock)
}

// But Lookup should find the extent
Expand All @@ -1070,7 +1063,7 @@ func TestLookupPhysical_ReturnsZeroForUnwritten(t *testing.T) {
}

// Now clear unwritten and LookupPhysical should return the real block
cleared, err := mgr.ClearUnwritten(ino, foundExt)
cleared, err := mgr.ClearUnwritten(ctx, ino, 0)
if err != nil {
t.Fatalf("ClearUnwritten failed: %v", err)
}
Expand All @@ -1082,7 +1075,6 @@ func TestLookupPhysical_ReturnsZeroForUnwritten(t *testing.T) {
if err != nil {
t.Fatalf("LookupPhysical after clear failed: %v", err)
}
// Note: ext.PhysicalBlock could be 0 (valid block number), so we compare directly
if physBlock != ext.PhysicalBlock {
t.Errorf("Expected physBlock=%d after clearing unwritten, got %d", ext.PhysicalBlock, physBlock)
}
Expand Down
24 changes: 0 additions & 24 deletions internal/extent/tree_edit.go
Original file line number Diff line number Diff line change
Expand Up @@ -292,16 +292,6 @@ func (e *editor) set(i int, exts []Extent) {
e.changed = true
}

func (e *editor) find(lb uint64) (leaf, idx int, ok bool, err error) {
leaf = route(e.leaves, lb)
exts, err := e.extentsAt(leaf)
if err != nil {
return 0, 0, false, err
}
idx, ok = find(exts, lb)
return leaf, idx, ok, nil
}

type piece struct {
leaf int
ext Extent
Expand Down Expand Up @@ -361,20 +351,6 @@ func (e *editor) removeRange(from, to uint64) ([]Extent, error) {
return removed, nil
}

// update applies fn to the extent containing lb and reports whether one exists.
func (e *editor) update(lb uint64, fn func(*Extent)) (bool, error) {
leaf, idx, ok, err := e.find(lb)
if err != nil || !ok {
return false, err
}
exts, err := e.mutable(leaf)
if err != nil {
return false, err
}
fn(&exts[idx])
return true, nil
}

func (e *editor) updateAll(fn func(*Extent)) error {
for i := range e.leaves {
exts, err := e.mutable(i)
Expand Down
Loading
Loading