Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod
- **Linux only.** GoClusterFS needs FUSE and O_DIRECT.
- **Every file with data uses at least one full 1 MB block.** The block size is 1 MB, and small-file packing (SFB) is disabled since [#12](https://github.com/SupportTools/GoClusterFS/pull/12), so `-sfb` has no effect.
- **Name lookups are slow.** Each lookup reads the whole directory from disk and scans it entry by entry. In one test on a single node, a lookup took about 25 ms, and `ls -l` on a 1,000-file directory took about 47 seconds.
- **Other nodes' changes can show up late.** The kernel caches file attributes and directory entries for `-attr-timeout` (default 1s). For that long, `stat` and `ls` can show another node's file with its old size and mtime, and a read can stop at the old size. A file that stays open can use the old size and block mappings for up to `-cache-ttl` (default 5s). Opening the file after `-attr-timeout` has passed reads the current data. See [Cache Flags](docs/API-CONFIG.md#cache-flags).
- **On-disk format version 2** since [#15](https://github.com/SupportTools/GoClusterFS/pull/15). Mount refuses images formatted by earlier builds. Copy the data off with the old build, reformat, and copy it back.
- **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md).
- **Snapshots and clones are experimental.** Before you mount an image that has snapshots or clones with a newer build, read the upgrade notes in [CHANGELOG.md](CHANGELOG.md). They say whether to run `goclusterfs-fsck` first.
Expand Down
8 changes: 3 additions & 5 deletions cmd/mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ func main() {
// Cache configuration
cacheStrict = flag.Bool("cache-strict", false, "Enable strict cache mode (always validate)")
cacheTTL = flag.Duration("cache-ttl", 5*time.Second, "Cache TTL before validation")
attrTimeout = flag.Duration("attr-timeout", time.Second, "How long the kernel caches file attributes and directory entries")

// Journal configuration
journalEnabled = flag.Bool("journal", true, "Enable write-ahead journal for crash consistency")
Expand Down Expand Up @@ -321,9 +322,6 @@ func main() {

// Configure go-fuse mount options
// AttrTimeout and EntryTimeout control kernel-side caching
attrTimeout := 1 * time.Second
entryTimeout := 1 * time.Second

mountOpts := &fs.Options{
MountOptions: fuse.MountOptions{
FsName: "goclusterfs",
Expand All @@ -333,8 +331,8 @@ func main() {
DisableXAttrs: false,
// Note: AllowOther requires 'user_allow_other' in /etc/fuse.conf
},
AttrTimeout: &attrTimeout,
EntryTimeout: &entryTimeout,
AttrTimeout: attrTimeout,
EntryTimeout: attrTimeout,
}

// Mount filesystem using go-fuse v2 API
Expand Down
7 changes: 6 additions & 1 deletion docs/API-CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ These flags are passed when starting `goclusterfs`:
|------|------|---------|-------------|
| `-cache-strict` | bool | `false` | Enable strict cache mode (always validate against disk) |
| `-cache-ttl` | duration | `5s` | Cache entry TTL before revalidation required |
| `-attr-timeout` | duration | `1s` | How long the kernel caches file attributes and directory entries |

The kernel caches file attributes and directory entries for `-attr-timeout`. For that long after another node changes a file, `stat` and path lookups on this node can return the old size, and a read can stop at the old size even right after `open`. After that, opening a file reads its inode from disk, so the reader sees the other node's completed writes. Reads through a file that is already open use the inode cache and can use the old size and block mappings for up to `-cache-ttl`, including a block that another node has since freed and reused for other data. With broadcast enabled, writes and directory changes invalidate peers' caches sooner; truncate, fallocate and unlink do not.

`-attr-timeout=0` makes `stat` and path lookups always current, but each `stat` and path step then reads the directory from disk under a cluster lock. In one test on a single mount, 2000 `stat` calls on one file took 13.9s at 0 and 0.01s at 1s, and `ls -l` of 300 files took 11.8s and 3.7s.

### Journal Flags

Expand Down Expand Up @@ -143,7 +148,7 @@ Time after which a cached inode must be revalidated against disk.
| Memory | ~3.2 MB (for defaults) |
| Recommended | 50,000 - 500,000 |

The maximum number of extent mappings to cache. Extent cache maps (inode, logical block) → physical block.
The maximum number of extent mappings to cache. Extent cache maps (inode, logical block) → physical block. A cached mapping is only used with the inode version it was looked up with, so any inode write by any node makes it stale.

**Sizing guidance**:
- Large files benefit more from extent caching
Expand Down
12 changes: 11 additions & 1 deletion internal/cache/cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,15 @@ func (c *CachedInode) IsStale(diskDataGen, diskMetaGen uint64) bool {
return c.DataGeneration != diskDataGen || c.MetaGeneration != diskMetaGen
}

// newerThan reports whether c holds a later version of the inode than in.
// DataGeneration only changes when the inode number is reused, so it orders first.
func (c *CachedInode) newerThan(in *inode.Inode) bool {
if c.DataGeneration != in.DataGeneration {
return c.DataGeneration > in.DataGeneration
}
return c.MetaGeneration > in.MetaGeneration
}

// NeedsValidation returns true if the cache entry needs revalidation against disk.
func (c *CachedInode) NeedsValidation(ttl time.Duration, strictMode bool) bool {
if strictMode {
Expand Down Expand Up @@ -254,7 +263,8 @@ func (m *Manager) BroadcastEnabled() bool {
}

// BroadcastInvalidateInode invalidates an inode locally AND broadcasts
// the invalidation to peer nodes. Call this after modifying an inode.
// the invalidation to peer nodes. Call this after modifying an inode, and
// cache the new inode after this call, since it drops the local entry.
//
// Parameters:
// - inodeNum: The inode number being invalidated
Expand Down
29 changes: 15 additions & 14 deletions internal/cache/cache_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -381,16 +381,17 @@ func TestCachedExtent(t *testing.T) {
cached := &CachedExtent{
PhysicalBlock: 100,
InodeDataGen: 5,
InodeMetaGen: 7,
}

// Same generation - not stale
if cached.IsStale(5) {
t.Error("IsStale() should return false when generation matches")
if cached.IsStale(5, 7) {
t.Error("IsStale() should return false when generations match")
}

// Different generation - stale
if !cached.IsStale(6) {
t.Error("IsStale() should return true when generation differs")
if !cached.IsStale(6, 7) {
t.Error("IsStale() should return true when DataGeneration differs")
}
if !cached.IsStale(5, 8) {
t.Error("IsStale() should return true when MetaGeneration differs")
}
})
}
Expand Down Expand Up @@ -491,14 +492,14 @@ func TestExtentCache(t *testing.T) {
t.Fatal("Get() should find cached extent")
}

// Same generation - not stale
if got.IsStale(10) {
t.Error("Extent should not be stale with matching generation")
if got.IsStale(10, 0) {
t.Error("Extent should not be stale with matching generations")
}

// Different generation - stale
if !got.IsStale(11) {
t.Error("Extent should be stale with different generation")
if !got.IsStale(11, 0) {
t.Error("Extent should be stale with different DataGeneration")
}
if !got.IsStale(10, 1) {
t.Error("Extent should be stale with different MetaGeneration")
}
})

Expand Down
106 changes: 37 additions & 69 deletions internal/cache/extent_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,122 +16,92 @@ type ExtentCacheKey struct {

// CachedExtent holds a cached extent lookup result.
type CachedExtent struct {
// PhysicalBlock is the physical disk block number.
// A value of 0 means "hole" (sparse file - unallocated region).
// PhysicalBlock is the physical disk block number, or extent.Hole.
PhysicalBlock uint64

// InodeDataGen is the inode's DataGeneration at cache time.
// When the inode's DataGeneration changes, all extent cache
// entries for that inode become stale.
// InodeDataGen and InodeMetaGen are the inode's generations at cache time.
// Every extent mapping change is committed by an inode write, which bumps
// MetaGeneration, so a mapping is only valid for the inode version it was
// looked up with.
InodeDataGen uint64
InodeMetaGen uint64

// CachedAt is when this entry was cached.
CachedAt time.Time
}

// IsStale returns true if the cached extent is stale relative to the inode.
func (c *CachedExtent) IsStale(currentDataGen uint64) bool {
return c.InodeDataGen != currentDataGen
// IsStale returns true if the cached extent was looked up with a different inode version.
func (c *CachedExtent) IsStale(dataGen, metaGen uint64) bool {
return c.InodeDataGen != dataGen || c.InodeMetaGen != metaGen
}

// GetExtent retrieves a cached extent mapping.
//
// The retrieval flow is:
// 1. Check extent cache for (inodeNum, logicalBlock)
// 2. If found, validate against inode's DataGeneration
// 3. If valid, return cached physical block
// 4. If stale or not found, return cache miss
// GetExtent retrieves a cached extent mapping for this version of ino.
//
// This function does NOT perform disk I/O - caller must fall back to
// extent.Manager.LookupPhysical() on cache miss.
//
// Returns:
// - physicalBlock: The physical block number (0 = hole)
// - physicalBlock: The physical block number or extent.Hole
// - found: true if cache hit and valid
func (m *Manager) GetExtent(inodeNum uint32, logicalBlock uint64, inodeDataGen uint64) (physicalBlock uint64, found bool) {
func (m *Manager) GetExtent(ino *inode.Inode, logicalBlock uint64) (physicalBlock uint64, found bool) {
key := ExtentCacheKey{
InodeNum: inodeNum,
InodeNum: ino.InodeNum,
LogicalBlock: logicalBlock,
}

cached, ok := m.extents.Get(key)
cached, ok := m.extents.GetIf(key, func(c *CachedExtent) bool {
return !c.IsStale(ino.DataGeneration, ino.MetaGeneration)
})
if !ok {
return 0, false
}

// Validate against inode's current DataGeneration
if cached.IsStale(inodeDataGen) {
// Stale entry - remove it
m.extents.Delete(key)
return 0, false
}

return cached.PhysicalBlock, true
}

// CacheExtent stores an extent mapping in the cache.
// CacheExtent stores an extent mapping looked up with this version of ino.
//
// Call this after a successful extent.Manager.LookupPhysical() to cache
// the result for future reads.
//
// Parameters:
// - inodeNum: The inode number
// - logicalBlock: The logical file block being looked up
// - physicalBlock: The physical disk block (0 for hole)
// - inodeDataGen: The inode's current DataGeneration
func (m *Manager) CacheExtent(inodeNum uint32, logicalBlock uint64, physicalBlock uint64, inodeDataGen uint64) {
// the result for future reads. physicalBlock is extent.Hole for a hole.
func (m *Manager) CacheExtent(ino *inode.Inode, logicalBlock uint64, physicalBlock uint64) {
if m.config.ExtentCacheSize == 0 {
return // Extent cache disabled
}

key := ExtentCacheKey{
InodeNum: inodeNum,
InodeNum: ino.InodeNum,
LogicalBlock: logicalBlock,
}

cached := &CachedExtent{
PhysicalBlock: physicalBlock,
InodeDataGen: inodeDataGen,
InodeDataGen: ino.DataGeneration,
InodeMetaGen: ino.MetaGeneration,
CachedAt: time.Now(),
}

m.extents.Put(key, cached)
}

// InvalidateExtents removes all cached extents for an inode.
// InvalidateExtents makes cached extents for an inode stale.
//
// Call this when:
// - File is truncated (all extents may change)
// - File data is modified in a way that changes extent mapping
// - Inode's DataGeneration has changed
//
// Note: Individual extent invalidation is handled lazily via generation
// check in GetExtent. This function is for bulk invalidation.
// Call this when another node changed the inode's extent mapping (truncate,
// hole fill, CoW). It drops the cached inode, so the next lookup reads the
// current inode version from disk and every mapping cached for an older
// version misses.
func (m *Manager) InvalidateExtents(inodeNum uint32) {
// Unfortunately, our LRU doesn't support prefix/range deletion.
// We rely on lazy invalidation via DataGeneration checks.
// For bulk invalidation, we'd need to iterate all entries.
//
// For now, extents are invalidated lazily when GetExtent detects
// a DataGeneration mismatch. This is acceptable because:
// 1. Stale entries don't return incorrect data (generation check)
// 2. They get evicted naturally by LRU or replaced on next lookup
// 3. InvalidateInodes() already handles the common case
m.inodes.Delete(inodeNum)
}

// InvalidateExtentsForInodes removes cached extents when inode generation changes.
// This is called from OnLockAcquire for extent-level locks.
// InvalidateExtentsForInodes makes cached extents for several inodes stale.
func (m *Manager) InvalidateExtentsForInodes(inodeNums ...uint32) {
// Lazy invalidation via generation check handles this.
// See InvalidateExtents() comment.
for _, inodeNum := range inodeNums {
m.InvalidateExtents(inodeNum)
}
}

// InvalidateExtentsForInode removes cached extents for a single inode.
// InvalidateExtentsForInode makes cached extents for a single inode stale.
// This satisfies the broadcast.InvalidationHandler interface.
func (m *Manager) InvalidateExtentsForInode(inodeNum uint32) {
// Lazy invalidation via DataGeneration check handles this.
// When the next GetExtent() call is made, the generation mismatch
// will cause the stale entry to be evicted.
m.InvalidateExtents(inodeNum)
}

Expand All @@ -142,26 +112,24 @@ func (m *Manager) InvalidateExtentsForInode(inodeNum uint32) {
//
// Parameters:
// - extMgr: The extent manager for disk fallback
// - ino: The inode (must have current DataGeneration)
// - ino: A committed inode version (no uncommitted extent edits)
// - logicalBlock: The logical file block to look up
//
// Returns:
// - physicalBlock: The physical disk block (0 = hole)
// - physicalBlock: The physical disk block or extent.Hole
// - err: Any error from disk lookup
func (m *Manager) LookupPhysicalCached(extMgr *extent.Manager, ino *inode.Inode, logicalBlock uint64) (uint64, error) {
// Try cache first
if phys, found := m.GetExtent(ino.InodeNum, logicalBlock, ino.DataGeneration); found {
if phys, found := m.GetExtent(ino, logicalBlock); found {
return phys, nil
}

// Cache miss - look up from disk
phys, err := extMgr.LookupPhysical(ino, logicalBlock)
if err != nil {
return 0, err
}

// Cache the result (including holes, which map to extent.Hole)
m.CacheExtent(ino.InodeNum, logicalBlock, phys, ino.DataGeneration)
m.CacheExtent(ino, logicalBlock, phys)

return phys, nil
}
Expand Down
55 changes: 55 additions & 0 deletions internal/cache/extent_cache_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package cache

import (
"testing"

"github.com/supporttools/GoClusterFS/internal/inode"
)

func TestGetExtentMissesAfterInodeWrite(t *testing.T) {
mgr, err := NewManager(DefaultConfig(), nil, nil)
if err != nil {
t.Fatal(err)
}

v1 := &inode.Inode{InodeNum: 3, DataGeneration: 1, MetaGeneration: 4}
mgr.CacheExtent(v1, 0, 100)
if phys, ok := mgr.GetExtent(v1, 0); !ok || phys != 100 {
t.Fatalf("GetExtent with the caching version = (%d, %v), want (100, true)", phys, ok)
}

v2 := &inode.Inode{InodeNum: 3, DataGeneration: 1, MetaGeneration: 5}
if phys, ok := mgr.GetExtent(v2, 0); ok {
t.Errorf("GetExtent after an inode write hit with block %d, want a miss", phys)
}
}

func TestInvalidateExtentsForInodeDropsCachedInode(t *testing.T) {
mgr, err := NewManager(DefaultConfig(), nil, nil)
if err != nil {
t.Fatal(err)
}

mgr.CacheInode(&inode.Inode{InodeNum: 3, MetaGeneration: 1})
mgr.InvalidateExtentsForInode(3)
if mgr.IsCached(3) {
t.Error("inode 3 still cached after InvalidateExtentsForInode")
}
}

func TestGetExtentCountsStaleEntryAsMiss(t *testing.T) {
mgr, err := NewManager(DefaultConfig(), nil, nil)
if err != nil {
t.Fatal(err)
}

mgr.CacheExtent(&inode.Inode{InodeNum: 3, MetaGeneration: 4}, 0, 100)
mgr.GetExtent(&inode.Inode{InodeNum: 3, MetaGeneration: 5}, 0)

if _, hits, misses, _ := mgr.ExtentCacheStats(); hits != 0 || misses != 1 {
t.Errorf("stale lookup counted hits=%d misses=%d, want 0 and 1", hits, misses)
}
if size, _, _, _ := mgr.ExtentCacheStats(); size != 0 {
t.Errorf("stale entry still cached, size=%d", size)
}
}
Loading
Loading