Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 92 additions & 35 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,48 +4,95 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Project Overview

GoClusterFS is a FUSE-based clustered filesystem written in Go. It mounts a block device and exposes it as a filesystem, designed for shared storage scenarios across cluster nodes.
GoClusterFS is a clustered FUSE filesystem written in Go. Several Linux nodes mount the same shared block device at the same time. Nodes coordinate through structures on the device itself: a node registry, CAS locks, and a write-ahead journal. All device I/O uses O_DIRECT, so no node trusts its own page cache. UDP multicast messages make cache invalidation faster, but correctness does not depend on them.

## Build and Run Commands
FUSE binding: `github.com/hanwen/go-fuse/v2`. Linux only.

```bash
# Build the binary
go build -o GoClusterFS .

# Run (requires root or fuse group membership)
./GoClusterFS <mountpoint> <block_device>

# Example with a file-backed block device
dd if=/dev/zero of=VirtBlock.img bs=1M count=3000 # Create 3GB virtual block
./GoClusterFS /mnt/goclusterfs ./VirtBlock.img
## Build and Run

# Unmount
fusermount -u /mnt/goclusterfs
```bash
make build # ./goclusterfs-mount and ./goclusterfs-fsck
go build ./... && go vet ./...
```

## Architecture

### Storage Layout
- **Block size**: 1MB fixed blocks
- **Block 0**: Reserved for metadata (file table serialized with `gob`)
- **Blocks 1+**: File data (one block per file currently)
Run against a file-backed image. The image must sit on a filesystem that supports O_DIRECT (tmpfs only on Linux 6.6+).

### Core Types
- `FS`: Root filesystem implementing `fs.FS` and `Statfs`
- `Dir`: Root directory implementing `Attr`, `ReadDirAll`, `Lookup`, `Create`
- `File`: File node implementing `Attr`, `ReadAll`, `Write`, `Flush`, `Setxattr`
- `FileMetadata`: Persisted file info (inode, filename, size, block index)

### File Locking
File locks are managed via extended attributes for cluster coordination:
- Set lock: `setfattr -n user.lock -v <owner_id> <file>`
- Release lock: `setfattr -n user.unlock -v <owner_id> <file>`
```bash
truncate -s 1G disk.img
mkdir -p mnt
./goclusterfs-mount -device ./disk.img -mount ./mnt -format # first time only, wipes the volume
./goclusterfs-mount -device ./disk.img -mount ./mnt # later mounts
fusermount3 -u mnt # or Ctrl+C the mount process
./goclusterfs-fsck ./disk.img # device is a positional argument
```

### Current Limitations
- Single block per file (max 1MB file size)
- No directory hierarchy (flat namespace)
- No delete operation implemented
- Metadata not checksummed
`goclusterfs-mount` stays in the foreground until the filesystem is unmounted. `-format` formats and then mounts. It logs a lot of `[DEBUG]` lines even without `-debug`, which only turns on go-fuse request tracing. Flags and defaults: `./goclusterfs-mount -h`, `./goclusterfs-fsck -h`, [docs/API-CONFIG.md](docs/API-CONFIG.md).

## Code Map

| Path | What it does |
|------|--------------|
| `cmd/mount` | `goclusterfs-mount`: fixes the on-disk layout, runs journal recovery, wires all managers, mounts with go-fuse |
| `cmd/fsck` | `goclusterfs-fsck`: 8-phase checker with `-repair` and `-force` modes |
| `cmd/crctest` | Prints CRC32C test values |
| `internal/goclusterfs` | FUSE nodes (`FS` root, `Dir`, `File`) and every VFS operation |
| `internal/storage` | O_DIRECT device, 4 KB aligned buffers, CRC32C |
| `internal/superblock` | Superblock format (see limitations: `cmd/mount` does not write it) |
| `internal/bitmap` | Block allocation bitmap, CRC32C per 4 KB sector |
| `internal/inode` | Inode table, 256-byte inodes, CRC32C per sector |
| `internal/extent` | File block mapping, up to 2 extents inline in the inode |
| `internal/dir` | Directory entries |
| `internal/lock` | Distributed CAS locks stored on the device |
| `internal/node` | Node registry, heartbeats, failed-node detection |
| `internal/journal` | Write-ahead journal, local and cross-node recovery |
| `internal/cache` | Inode and extent caches, checked against on-disk generation counters |
| `internal/cache/broadcast` | UDP multicast/unicast cache invalidation, optional HMAC |
| `internal/refcount` | Per-block reference counts for copy-on-write |
| `internal/snapshot` | Snapshot registry and background refcount worker |
| `internal/sfb` | Small-file containers: 4/8/16/64 KB slots inside a 1 MB block |
| `internal/trim` | Batched `BLKDISCARD` for freed blocks |
| `internal/fsck` | The fsck phases |
| `internal/metrics` | Atomic counters, not used by `cmd/mount` yet |

## On-Disk Layout

`cmd/mount/main.go` hardcodes this layout. Data blocks are 1 MB. Metadata I/O uses 4 KB sectors.

| Offset | Size | Region |
|--------|------|--------|
| 0 | 1 MB | Reserved for the superblock (not written today) |
| 1 MB | 1 MB | Block bitmap |
| 2 MB | 1 MB | Inode table: 4096 inodes of 256 bytes; root is inode 2 |
| 3 MB | 1 MB | Node registry |
| 4 MB | 1 MB | Lock records |
| 5 MB | 4 MB | Journal |
| 9 MB | 1 MB | Block refcounts |
| 10 MB | 1 MB | Snapshot registry |
| 11 MB | rest | Data blocks |

The smallest device is 12 MB.

## Snapshots and Clones

Snapshots and clones work through extended attributes on a file (`internal/goclusterfs/fs.go`, `File.Setxattr`):

- `setfattr -n user.snapshot -v <name> <file>` creates a snapshot
- `getfattr -n user.snapshots <file>` lists snapshots
- `setfattr -n user.snapshot.delete -v <name> <file>` or `setfattr -x user.snapshot.<name> <file>` deletes one
- `setfattr -n user.clone -v /<new-name> <file>` clones the file
- `setfattr -n user.snapshot.clone -v <snapshot>:/<new-name> <file>` clones from a snapshot

Clone paths are absolute from the filesystem root, and clones can only go in the root directory.

## Current Limitations

- Linux only. Needs FUSE and a backing store that supports O_DIRECT.
- `cmd/mount` never writes a superblock, and `internal/superblock` describes a different layout from the one above. `goclusterfs-fsck` checks the superblock first, so it stops at phase 1 with `invalid superblock magic` on any volume made with `-format`.
- Every node uses the same 4 MB journal region at 5 MB. `cmd/mount` does not give each node its own journal slot.
- Each file holds at most 2 extents (inline in the inode). The overflow extent tree is being built.
- The inode count is fixed at 4096.
- SFB containers fail on real devices: `internal/sfb` writes buffers that are not 4 KB aligned, so `storage.WriteAt` rejects them and small files fall back to full 1 MB blocks.
- No rename, symlinks, or hard links.

## Testing

Expand Down Expand Up @@ -109,3 +156,13 @@ For tests involving concurrent writes from multiple nodes, use the **pre-extensi
4. Verify data at each offset after writes complete

This avoids TOCTOU races in the distributed locking system.

## Commands Config

| Setting | Value |
|---------|-------|
| TaskForge Project ID | 88 |
| Build command | `make build` |
| Test command | `go test ./... -v` |
| Lint command | `gofmt -l .` |
| Tech stack | Go (FUSE filesystem) |
80 changes: 41 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod

## Requirements

- Go 1.21 or later
- Go 1.26 or later
- Linux with FUSE support (`fusermount3`)
- O_DIRECT-capable storage (not tmpfs)
- O_DIRECT-capable storage (tmpfs supports O_DIRECT only on Linux 6.6 and later)
- Root or `fuse` group membership
- For multi-node: Shared block device (SAN, iSCSI, NVMe-oF, etc.)

Expand All @@ -30,54 +30,50 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod
```bash
git clone https://github.com/supporttools/GoClusterFS.git
cd GoClusterFS
go build -o goclusterfs ./cmd/mount
make build
```

### Create and Format a Block Device
This builds `goclusterfs-mount` and `goclusterfs-fsck` in the repo root.

```bash
# Create a file-backed block device (3GB example)
dd if=/dev/zero of=VirtBlock.img bs=1M count=3000

# Format the filesystem
./goclusterfs -format -device ./VirtBlock.img
```
### Format and Mount

### Mount
Use a file as the block device. Keep it on a filesystem that supports O_DIRECT.

```bash
# Create mount point
mkdir -p /mnt/goclusterfs
truncate -s 1G disk.img
mkdir -p mnt
./goclusterfs-mount -device ./disk.img -mount ./mnt -format > mount.log 2>&1 &
sleep 2
```

# Mount with default settings
./goclusterfs -device ./VirtBlock.img -mount /mnt/goclusterfs
`-format` wipes the device and then mounts it. The mount process keeps running until you unmount, and logs to `mount.log`.

# Mount with journaling and network broadcast (multi-node)
./goclusterfs -device ./VirtBlock.img -mount /mnt/goclusterfs \
-journal -broadcast -mcast-group 239.192.74.75
### Use It

# Unmount
fusermount -u /mnt/goclusterfs
```bash
echo "Hello, Cluster!" > mnt/hello.txt
cat mnt/hello.txt
df -h mnt
```

### Verify
### Unmount

```bash
# Create a file
echo "Hello, Cluster!" > /mnt/goclusterfs/test.txt

# Check filesystem stats
df -h /mnt/goclusterfs
fusermount3 -u mnt
```

The mount process exits after the unmount. To mount the same image again, run the mount command without `-format`.

## Command-Line Options

These are the `goclusterfs-mount` flags. Run `./goclusterfs-mount -h` for the full list.

| Flag | Default | Description |
|------|---------|-------------|
| `-device` | (required) | Path to block device |
| `-mount` | (required) | Mount point directory |
| `-format` | `false` | Format the device (destroys data) |
| `-debug` | `false` | Enable debug logging |
| `-device` | (required) | Path to block device or image file |
| `-mount` | (required) | Mount point directory (must exist) |
| `-format` | `false` | Format the device, then mount it (destroys data) |
| `-debug` | `false` | Log every FUSE request |

### Cache Options

Expand Down Expand Up @@ -126,13 +122,15 @@ For clustered deployments:
2. **Network**: Nodes should be on the same subnet for multicast invalidation
3. **Mount Command**: Use identical mount options across all nodes

Format the device once, from one node, with `-format`. Then mount it on every node without `-format`:

```bash
# Node 1
./goclusterfs -device /dev/shared/lun0 -mount /mnt/cluster \
./goclusterfs-mount -device /dev/shared/lun0 -mount /mnt/cluster \
-journal -broadcast -mcast-group 239.192.74.75

# Node 2 (same block device, same options)
./goclusterfs -device /dev/shared/lun0 -mount /mnt/cluster \
./goclusterfs-mount -device /dev/shared/lun0 -mount /mnt/cluster \
-journal -broadcast -mcast-group 239.192.74.75
```

Expand All @@ -146,27 +144,31 @@ For clustered deployments:

## Filesystem Check

Run fsck to verify or repair the filesystem:
Run fsck on an unmounted device. The device is a positional argument, and flags go before it:

```bash
# Check only (read-only)
./goclusterfs-fsck -device ./VirtBlock.img
./goclusterfs-fsck ./disk.img

# Check and repair
./goclusterfs-fsck -device ./VirtBlock.img -repair
./goclusterfs-fsck -repair ./disk.img

# Force check (ignore clean flag)
./goclusterfs-fsck -device ./VirtBlock.img -force
# Keep going past fatal errors (data recovery)
./goclusterfs-fsck -force ./disk.img
```

Exit codes: `0` clean, `1` errors found, `2` usage error or fatal failure.

**Known issue:** fsck does not work yet on volumes made with `goclusterfs-mount -format`. The mount tool does not write a superblock, so fsck stops at phase 1 with `invalid superblock magic` and exits with `2`.

## Architecture

GoClusterFS uses a layered architecture:

```
┌─────────────────────────────────────────────────────┐
│ FUSE Layer │
│ (bazil.org/fuse) │
│ (hanwen/go-fuse v2) │
├─────────────────────────────────────────────────────┤
│ VFS Operations │
│ (Dir, File, Attr, Read, Write, Lookup) │
Expand Down
18 changes: 9 additions & 9 deletions docs/development/multi-task-execution-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ Follow implementation guidelines from [Task Execution Workflow](./task-execution
### 4.3 Build Verification (BLOCKER)

```bash
go build -o GoClusterFS .
make build
```

**MUST PASS** - Zero errors. If fails, fix and retry.
Expand All @@ -258,20 +258,20 @@ When task affects filesystem operations:

```bash
# Create test environment
dd if=/dev/zero of=/tmp/test.img bs=1M count=100
mkdir -p /tmp/testmount
truncate -s 100M test.img
mkdir -p testmount

# Mount and test
./GoClusterFS /tmp/testmount /tmp/test.img &
./goclusterfs-mount -device ./test.img -mount ./testmount -format > mount.log 2>&1 &
sleep 2

# Verify operations
ls -la /tmp/testmount
echo "test" > /tmp/testmount/testfile
cat /tmp/testmount/testfile
ls -la testmount
echo "test" > testmount/testfile
cat testmount/testfile

# Unmount
fusermount -u /tmp/testmount
fusermount3 -u testmount
```

### 4.7 Task Completion
Expand Down Expand Up @@ -397,7 +397,7 @@ Clear todo list for next batch

| Phase | Gate | Command | Blocker |
|-------|------|---------|---------|
| Build | Binary compiles | `go build -o GoClusterFS .` | YES |
| Build | Binary compiles | `make build` | YES |
| Format | Code formatted | `gofmt -l .` | YES |
| Test | All tests pass | `go test ./... -v` | YES |
| FUSE | Mount/unmount works | Manual test | Conditional |
Expand Down
Loading
Loading