Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- Directory entries now fill all 4 KB chunks of a directory block. When an older binary removes the first entry of a later chunk, it grows the last entry of the chunk before it across the boundary. This version then stops listing that entry, and fsck reports the directory block.
- Run `goclusterfs-fsck -repair` before the first mount after upgrading an image that has snapshots or clones. Unmount the device on every node first.
- Older versions marked a file's blocks shared, on snapshot or clone, without counting the owners. Until `goclusterfs-fsck -repair` counts them, this version never frees such blocks, so deleting those files leaks their space, and it refuses to snapshot or clone them (`ENOTSUP`). In-place writes to an old clone still change its source.
- Snapshots made by older versions captured no data. They still list and delete, but cloning one fails with `ENOTSUP`.
- Snapshots now record the birth generation of the file they were taken of, so a later file that reuses the inode number cannot list, clone or delete them. Snapshots older versions made do not record it, and some captured no data, so after the upgrade they are orphans: `getfattr -n user.snapshots` no longer lists them, `GET /api/v1/snapshots` lists them with `orphan: true`, and `DELETE /api/v1/snapshots/<name>` without `path` deletes them. Older binaries keep the birth generation in inode bytes 196–203 only while they do not rewrite the inode, so upgrade all nodes together.
- `-skip-recovery` now takes the slot number from the recovery error, as `-skip-recovery=N`, and discards a journal only when the mount takes slot N. A bare `-skip-recovery` is a usage error.

### Added
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Snapshots and clones work through extended attributes on a file (`internal/goclu
- `setfattr -n user.clone -v /<new-name> <file>` clones the file
- `setfattr -n user.snapshot.clone -v <snapshot>:/<new-name> <file>` clones from a snapshot

Clone paths are absolute from the filesystem root, and clones can only go in the root directory.
Clone paths are absolute from the filesystem root, and clones can only go in the root directory. A snapshot records its file's inode number and birth generation, so a later file in the same inode slot does not see it; once its file is deleted it is an orphan, listed and deleted only through the API without a path.

## Current Limitations

Expand Down
6 changes: 5 additions & 1 deletion cmd/goclusterfs-mount/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -511,7 +511,11 @@ func main() {
broadcastOn: cacheMgr.BroadcastEnabled(),
broadcastProblem: cacheMgr.BroadcastProblem,
})
apiServer.SetFileService(fileops.New(*mountPoint, gfs.SnapshotRegistry()))
var snaps fileops.Snapshots
if gfs.SnapshotRegistry() != nil {
snaps = gfs
}
apiServer.SetFileService(fileops.New(*mountPoint, snaps))
apiServer.SetAdminService(adminOps{journal: j, trim: trimMgr, cache: cacheMgr})
if err := apiServer.Start(ctx); err != nil {
log.Printf("Warning: Failed to start the API server: %v; the mount continues without it. Choose a free -api-addr, or pass -api=false", err)
Expand Down
5 changes: 3 additions & 2 deletions docs/API-CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,14 @@ Each mount on a host needs its own `-api-addr`. When the port is taken, mount lo

These endpoints change the filesystem or this node's state, and listing snapshots reads filesystem metadata. They only work with a token, also on loopback: the Host check stops web pages, not other local processes. Without a token they answer 403, and mount logs `API: snapshot, clone and admin endpoints are off`. The API refuses cross-origin writes: a CORS preflight for any method but `GET` or `HEAD` fails, and a write whose `Origin` header names another site gets 403. Same-origin pages and non-browser clients can call them.

Snapshots belong to single files. A request names the file by its path in the mount, such as `/dir/file`. The path must be absolute and clean, and may not go through a symlink or out of the mount. Snapshot names are unique in the filesystem. The API creates and clones only names without `:`, `/` or NUL, because `user.snapshot.clone` splits at the first `:`. It deletes any name; escape `/` as `%2F` in the URL.
Snapshots belong to single files. A snapshot stays with the file it was taken of: once that file is deleted, the snapshot is an orphan, which only the full listing shows and only a delete without `path` removes. A later file that reuses the inode number does not see it. Snapshots written by versions before this record cannot prove their file, so they are orphans too. A request names the file by its path in the mount, such as `/dir/file`. The path must be absolute and clean, and may not go through a symlink or out of the mount. Snapshot names are unique in the filesystem. The API creates and clones only names without `:`, `/` or NUL, because `user.snapshot.clone` splits at the first `:`. It deletes any name; escape `/` as `%2F` in the URL.

| Method and path | Body | Does the same as | Success |
|-----------------|------|------------------|---------|
| `GET /api/v1/snapshots[?path=/f]` | | Reads the snapshot registry: every snapshot, or the file's, as `getfattr -n user.snapshots` lists them | 200, `{snapshots: [{id, name, inode, state, size_bytes, blocks, created_at}], total}` |
| `GET /api/v1/snapshots[?path=/f]` | | Reads the snapshot registry: every snapshot, with `orphan: true` on those whose file is gone, or the file's, as `getfattr -n user.snapshots` lists them | 200, `{snapshots: [{id, name, inode, state, size_bytes, blocks, created_at, orphan}], total}` |
| `POST /api/v1/snapshots` | `{"path": "/f", "name": "s1"}` | `setfattr -n user.snapshot -v s1 f` | 201 |
| `DELETE /api/v1/snapshots/s1?path=/f` | | `setfattr -n user.snapshot.delete -v s1 f` | 200 |
| `DELETE /api/v1/snapshots/s1` | | Deletes an orphan: a snapshot whose file is gone | 200; 409 if its file exists |
| `POST /api/v1/snapshots/s1/clone` | `{"path": "/f", "dest": "/copy"}` | `setfattr -n user.snapshot.clone -v s1:/copy f` | 201 |
| `POST /api/v1/clones` | `{"path": "/f", "dest": "/copy"}` | `setfattr -n user.clone -v /copy f` | 201 |
| `POST /api/v1/admin/flush-journal` | | Checkpoints the journal now | 200 |
Expand Down
2 changes: 2 additions & 0 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,8 @@ setfattr -n user.clone -v /data.db.copy /mnt/fs/data.db
setfattr -x user.snapshot.snap1 /mnt/fs/data.db
```

A snapshot stays with the file it was taken of. When you delete the file, its snapshots become orphans: `getfattr` on any file no longer lists them, and `GET /api/v1/snapshots` lists them with `orphan: true`. Delete an orphan with `DELETE /api/v1/snapshots/<name>` and no `path`. Snapshots made by older versions are orphans after an upgrade, because they do not record which file they were taken of. `goclusterfs-fsck` warns about each orphan.

Snapshots and clones share blocks with the file and copy a block on its first write. A block is freed when no file, snapshot, or clone uses it. Before you mount an image with snapshots or clones using a newer build, read the upgrade notes in [CHANGELOG.md](../CHANGELOG.md).

---
Expand Down
2 changes: 2 additions & 0 deletions internal/api/writes.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ type SnapshotInfo struct {
SizeBytes uint64 `json:"size_bytes"`
Blocks uint64 `json:"blocks"`
CreatedAt string `json:"created_at,omitempty"`
// Orphan marks a snapshot whose file is gone, or that cannot prove which file it was taken of.
Orphan bool `json:"orphan,omitempty"`
}

// SnapshotListResponse is returned by GET /api/v1/snapshots.
Expand Down
62 changes: 42 additions & 20 deletions internal/fileops/fileops.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,28 @@ import (

// Ops implements api.FileService for the filesystem mounted at a directory.
type Ops struct {
root string
registry *snapshot.Registry
root string
snaps Snapshots
// checkRoot refuses a root that is not the mounted filesystem.
checkRoot func(fd int) error
}

// Snapshots is the filesystem's snapshot registry, which knows the file each snapshot was taken of.
type Snapshots interface {
AllSnapshots(ctx context.Context) ([]snapshot.Entry, error)
FileSnapshots(ctx context.Context, inodeNum uint32) ([]*snapshot.Descriptor, error)
DeleteOrphanSnapshot(ctx context.Context, name string) error
}

// New returns Ops for the mount at root, which must already be mounted.
// registry is nil when snapshots are off.
func New(root string, registry *snapshot.Registry) *Ops {
// snaps is nil when snapshots are off.
func New(root string, snaps Snapshots) *Ops {
var st unix.Stat_t
dev := uint64(0)
if unix.Stat(root, &st) == nil {
dev = uint64(st.Dev)
}
return &Ops{root: root, registry: registry, checkRoot: func(fd int) error { return isOurMount(fd, dev) }}
return &Ops{root: root, snaps: snaps, checkRoot: func(fd int) error { return isOurMount(fd, dev) }}
}

// isOurMount refuses a root that is no longer this FUSE mount, for example a
Expand Down Expand Up @@ -64,33 +71,41 @@ func sameDevice(fd int, dev uint64) error {

var _ api.FileService = (*Ops)(nil)

// ListSnapshots returns every snapshot, or only those of the file at p.
// ListSnapshots returns every snapshot, marking those whose file is gone, or only those of the file at p.
func (o *Ops) ListSnapshots(ctx context.Context, p string) ([]api.SnapshotInfo, error) {
if o.registry == nil {
if o.snaps == nil {
return nil, fmt.Errorf("snapshots are %w (-snapshots=false)", api.ErrDisabled)
}
var descs []*snapshot.Descriptor
var err error
var entries []snapshot.Entry
if p == "" {
descs, err = o.registry.List(ctx)
var err error
if entries, err = o.snaps.AllSnapshots(ctx); err != nil {
return nil, err
}
} else {
var st unix.Stat_t
if st, err = o.stat(p); err == nil {
descs, err = o.registry.ListForInode(ctx, uint32(st.Ino))
st, err := o.stat(p)
if err != nil {
return nil, err
}
descs, err := o.snaps.FileSnapshots(ctx, uint32(st.Ino))
if err != nil {
return nil, err
}
for _, d := range descs {
entries = append(entries, snapshot.Entry{Descriptor: d})
}
}
if err != nil {
return nil, err
}
out := make([]api.SnapshotInfo, 0, len(descs))
for _, d := range descs {
out := make([]api.SnapshotInfo, 0, len(entries))
for _, e := range entries {
d := e.Descriptor
info := api.SnapshotInfo{
ID: d.SnapshotID,
Name: d.GetName(),
Inode: d.TargetInode,
State: d.State.String(),
SizeBytes: d.TotalSize,
Blocks: d.TotalBlocks,
Orphan: e.Orphan,
}
if d.CreateTime > 0 {
info.CreatedAt = time.Unix(d.CreateTime, 0).UTC().Format(time.RFC3339)
Expand All @@ -108,11 +123,18 @@ func (o *Ops) CreateSnapshot(_ context.Context, p, name string) error {
return o.setxattr(p, "user.snapshot", name)
}

// DeleteSnapshot does what setfattr -n user.snapshot.delete -v name does.
func (o *Ops) DeleteSnapshot(_ context.Context, p, name string) error {
// DeleteSnapshot does what setfattr -n user.snapshot.delete -v name does. Without p it deletes a snapshot whose file
// is gone.
func (o *Ops) DeleteSnapshot(ctx context.Context, p, name string) error {
if name == "" || strings.ContainsRune(name, '\x00') {
return fmt.Errorf("%w: snapshot name %q", api.ErrInvalidPath, name)
}
if p == "" {
if o.snaps == nil {
return fmt.Errorf("snapshots are %w (-snapshots=false)", api.ErrDisabled)
}
return o.snaps.DeleteOrphanSnapshot(ctx, name)
}
return o.setxattr(p, "user.snapshot.delete", name)
}

Expand Down
2 changes: 2 additions & 0 deletions internal/fsck/fsck.go
Original file line number Diff line number Diff line change
Expand Up @@ -390,6 +390,7 @@ type Checker struct {
badBitmap map[uint64]bool // bitmap sectors with a bad checksum
inodeBlocks map[uint64]int // block -> reference count from inodes
inodeUsed map[uint32]bool // inode numbers that are in use
inodeBirth map[uint32]uint64
dirEntryRefs map[uint32]int // inode -> names in directories, not counting "." and ".."
dirNames map[uint32]int // directory inode -> names it holds
namesIncomplete []string // why dirEntryRefs may miss names
Expand Down Expand Up @@ -475,6 +476,7 @@ func New(dev *storage.Device, opts Options) *Checker {
finished: make(map[Phase]bool),
inodeBlocks: make(map[uint64]int),
inodeUsed: make(map[uint32]bool),
inodeBirth: make(map[uint32]uint64),
dirEntryRefs: make(map[uint32]int),
dirNames: make(map[uint32]int),
snapshotInodes: make(map[uint32]bool),
Expand Down
4 changes: 4 additions & 0 deletions internal/fsck/phase3_inodes.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ type InodeData struct {
ExtentDepth uint32
ExtentTreeGen uint64

BirthGeneration uint64

// Checksum
Checksum uint32
}
Expand Down Expand Up @@ -160,6 +162,7 @@ func (c *Checker) checkInodes() error {

usedInodes++
c.inodeUsed[inodeNum] = true
c.inodeBirth[inodeNum] = inode.BirthGeneration

// Validate inode number matches position
if inode.InodeNum != inodeNum {
Expand Down Expand Up @@ -272,6 +275,7 @@ func (c *Checker) parseInode(buf []byte) *InodeData {
inode.ExtentCount = binary.LittleEndian.Uint64(buf[176:184])
inode.ExtentDepth = binary.LittleEndian.Uint32(buf[184:188])
inode.ExtentTreeGen = binary.LittleEndian.Uint64(buf[188:196])
inode.BirthGeneration = binary.LittleEndian.Uint64(buf[196:204])

// Checksum (last 4 bytes)
inode.Checksum = binary.LittleEndian.Uint32(buf[252:256])
Expand Down
27 changes: 22 additions & 5 deletions internal/fsck/phase7_snapshots.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ const (

// SnapshotFlagHasData marks descriptors whose DataInode holds the captured extents.
SnapshotFlagHasData uint8 = 1 << 4
// SnapshotFlagHasTargetBirth marks descriptors that record the birth generation of the file they were taken of.
SnapshotFlagHasTargetBirth uint8 = 1 << 5

// MaxSnapshotNameLen is the maximum length of a snapshot name.
MaxSnapshotNameLen = 48
Expand Down Expand Up @@ -53,6 +55,7 @@ type SnapshotDescriptor struct {
CreateTime int64
DeleteTime int64
RefCountJobID uint64
TargetBirth uint64
Name string
Checksum uint32
}
Expand Down Expand Up @@ -161,10 +164,9 @@ func (c *Checker) checkSnapshots() error {
seenNames[desc.Name] = desc.SnapshotID
}

// Validate target inode
if desc.TargetInode > 0 && !c.inodeUsed[desc.TargetInode] {
c.addProblem(PhaseSnapshots, Error, fmt.Sprintf("snapshot %d", desc.SnapshotID),
fmt.Sprintf("references non-existent inode %d", desc.TargetInode),
if why := c.snapshotOrphaned(desc); why != "" {
c.addProblem(PhaseSnapshots, Warning, fmt.Sprintf("snapshot %d", desc.SnapshotID),
fmt.Sprintf("snapshot %q is an orphan: %s; only the API's full listing shows it, and it can be deleted there", desc.Name, why),
false)
}

Expand Down Expand Up @@ -222,6 +224,19 @@ func (c *Checker) checkSnapshots() error {
return nil
}

// snapshotOrphaned says why desc belongs to no existing file, or returns "".
func (c *Checker) snapshotOrphaned(desc *SnapshotDescriptor) string {
switch {
case !c.inodeUsed[desc.TargetInode]:
return fmt.Sprintf("inode %d, the file it was taken of, is free", desc.TargetInode)
case desc.Flags&SnapshotFlagHasTargetBirth == 0:
return "an older version wrote it without the birth generation that proves which file it was taken of"
case desc.TargetBirth != c.inodeBirth[desc.TargetInode]&(1<<48-1):
return fmt.Sprintf("inode %d now holds a later file than the one it was taken of", desc.TargetInode)
}
return ""
}

// parseSnapshotDescriptor extracts a snapshot descriptor from raw bytes.
func parseSnapshotDescriptor(buf []byte) *SnapshotDescriptor {
desc := &SnapshotDescriptor{}
Expand All @@ -232,8 +247,10 @@ func parseSnapshotDescriptor(buf []byte) *SnapshotDescriptor {
desc.TargetInode = binary.LittleEndian.Uint32(buf[20:24])
desc.State = buf[24]
desc.Flags = buf[25]
// Bytes 26-27 reserved
desc.DataInode = binary.LittleEndian.Uint32(buf[28:32])
if desc.Flags&SnapshotFlagHasTargetBirth != 0 {
desc.TargetBirth = uint64(binary.LittleEndian.Uint16(buf[26:28]))<<32 | uint64(binary.LittleEndian.Uint32(buf[32:36]))
}
desc.TotalBlocks = binary.LittleEndian.Uint64(buf[36:44])
desc.TotalSize = binary.LittleEndian.Uint64(buf[44:52])
desc.CreateTime = int64(binary.LittleEndian.Uint64(buf[52:60]))
Expand Down
Loading
Loading