Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

- The on-disk format is now version 3. Mount refuses devices formatted by earlier builds. Version 1 kept one journal for all nodes. Version 2 had room for 4096 inodes, and on devices over 128 GB its refcounts overwrote the snapshot registry and then data blocks. Copy the data off with the old build, reformat with `-format`, and copy it back.
- Devices must be at least 327 MB: the layout reserves 256 MB for 64 node journals of 4 MB and 64 MB for lock records.
- Mount also refuses version 3 devices formatted before lock records had one claim slot per node. Copy the data off with the build that formatted them and reformat.
- Mount also refuses version 3 devices formatted before lock records had one claim slot per node, or before each lock type had its own lock records. Older builds refuse devices this version formats. Copy the data off with the build that formatted them and reformat.
- Node clocks must agree within a few seconds. See the node requirements in `docs/OPERATIONS.md`.
- Upgrade all nodes together: unmount on every node, upgrade, then mount again. Don't mount one shared device with a mix of old and new binaries.
- Older binaries reset an inode's generation counters to 0 when they free it, so other nodes can serve stale data after the inode number is reused.
- Older binaries zero the extent-tree generation (inode bytes 188–195) when they rewrite an inode. Newer binaries then cannot read that file if it has more than two extents, because such files use an extent tree.
- This version places lock records differently (inode table sector locks have their own records), so old and new binaries on one device do not exclude each other.
- Directory entries now fill all 4 KB chunks of a directory block. When an older binary removes the first entry of a later chunk, it grows the last entry of the chunk before it across the boundary. This version then stops listing that entry, and fsck reports the directory block.
- Run `goclusterfs-fsck -repair` before the first mount after upgrading an image that has snapshots or clones. Unmount the device on every node first.
- Older versions marked a file's blocks shared, on snapshot or clone, without counting the owners. Until `goclusterfs-fsck -repair` counts them, this version never frees such blocks, so deleting those files leaks their space, and it refuses to snapshot or clone them (`ENOTSUP`). In-place writes to an old clone still change its source.
Expand Down Expand Up @@ -79,6 +78,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- A node could serve a stale cached inode after another node freed and reused its inode number. Inode generations now survive free and reuse, and opening a file reloads its inode from disk.
- Journal recovery no longer panics, loops forever, or allocates gigabytes on a corrupt journal. It now rejects short entries and bad ring geometry as corrupt.
- Lock-order violations when several nodes create files and directories at the same time.
- Two nodes could each hold a lock and wait for the other's, then fail with `EIO` after 30 seconds, when a write's inode and block locks mapped to records the other node held. Each lock type now has its own lock records.
- Inode allocation races between nodes; allocation now retries with jittered backoff.
- Directory operations now update caches and send broadcast invalidations.
- Docker and VM test suites now run on fresh hosts.
Expand Down
2 changes: 1 addition & 1 deletion docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no

**Read locks are not renewed.** A node whose overlapping lookups keep a directory read-locked for more than 15 seconds can lose that lock to another node's writer. Its readers may then see a half-written directory. Two writers still cannot overlap.

**Locks share records.** The lock region has 256 records, and every lock maps to one of them, so unrelated files and the block bitmap can share a record. A node holding one of those locks blocks other nodes from all of them. If two nodes each hold one record and wait for the other's, both wait until one record is 15 seconds old; the takeover then lets both proceed, with the same risk as a stall. This needs nested locks whose records cross, roughly 1 in 65,000 pairs of overlapping operations.
**Locks of one type share records.** Each lock type has its own records of the 256: 128 for inodes, 97 for block extents, 16 for inode table sectors, 8 for SFB containers, and one for each other type. Two files' inode locks can map to one record, and a node holding one of them blocks other nodes from both. A lock never waits on a record another type uses. A node takes locks in type order and, within a type, in record order, so two nodes cannot each wait for a record the other holds. The exception is a replay of a failed node's journal: the replaying node keeps the failed node's records until the replay ends, so a node that holds a lock the replay needs and waits on one of those records waits until one side gives up after 30 seconds.

**Safe operating limits:**

Expand Down
23 changes: 23 additions & 0 deletions internal/fsck/fsck_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -565,3 +565,26 @@ func TestFsckExtentTree(t *testing.T) {
t.Fatal("data extent pointing at another inode's tree node not reported")
}
}

func TestFsckRefusesHashedLockRecords(t *testing.T) {
dev, cleanup := testDevice(t, 20)
defer cleanup()
sb := superblock.New(dev.Size(), 0)
sb.TotalBlocks, sb.FreeBlocks = 9, 9
sb.RequiredFlags &^= superblock.RequiredTypedLockRecords
if _, err := dev.WriteAt(sb.Serialize(), 0); err != nil {
t.Fatal(err)
}

opts := fsck.DefaultOptions()
opts.Output = &bytes.Buffer{}
result := fsck.New(dev, opts).Check()
found := false
for _, p := range result.Problems {
found = found || p.Phase == fsck.PhaseSuperblock && p.Severity == fsck.Fatal && strings.Contains(p.Message, "reformat with -format")
}
if !result.Aborted || !found {
t.Fatalf("fsck of a volume without per-type lock records: aborted=%v, problems %v; want a fatal superblock problem saying to reformat",
result.Aborted, result.Problems)
}
}
10 changes: 3 additions & 7 deletions internal/fsck/phase1_superblock.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,9 @@ func (c *Checker) checkSuperblock() error {
}
c.logf(" Version: %d (OK)\n", sb.Version)

// Check required flags
unknownFlags := sb.RequiredFlags &^ superblock.FlagsWeSupport
if unknownFlags != 0 {
c.addProblem(PhaseSuperblock, Fatal, "offset 40",
fmt.Sprintf("unknown required flags: 0x%X", unknownFlags),
false)
return fmt.Errorf("unknown required flags")
if err := superblock.CheckRequiredFlags(sb.RequiredFlags); err != nil {
c.addProblem(PhaseSuperblock, Fatal, "offset 40", err.Error(), false)
return fmt.Errorf("unsupported required flags")
}
c.logf(" Required Flags: 0x%X (OK)\n", sb.RequiredFlags)

Expand Down
72 changes: 50 additions & 22 deletions internal/lock/lock.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@
// Level 9: Extent (by block)
// Level 10: InodeSector (by inode table sector), a leaf: held only around
// one sector's read-modify-write, with no other lock taken while held
//
// Each type has its own records, laid out in type order, and locks of one type
// are taken in record order. A goroutine's records then only go up, so waits
// between nodes cannot form a cycle, except through records a node adopts to
// replay a failed node's journal: the replay holds them until it ends and takes
// its own locks in journal order.
package lock

import (
Expand Down Expand Up @@ -277,6 +283,31 @@ func (m *Manager) Stats() Stats {

const numLockTypes = int(LockTypeInodeSector) + 1

// layoutRows is how many records a volume's lock region has; typeRows divides them among the types.
const layoutRows = 256

// typeRows is each type's share of the records, by how many keys of the type nodes hold at once.
var typeRows = [numLockTypes]uint64{
LockTypeVolume: 1,
LockTypeSnapshotReg: 1,
LockTypeSnapshot: 1,
LockTypeJournal: 1,
LockTypeRefCount: 1,
LockTypeLockRegion: 1,
LockTypeBlockBitmap: 1,
LockTypeSFBContainer: 8,
LockTypeInode: 128,
LockTypeExtent: 97,
LockTypeInodeSector: 16,
}

var typeFirstRow = func() (first [numLockTypes]uint64) {
for t := 1; t < numLockTypes; t++ {
first[t] = first[t-1] + typeRows[t-1]
}
return first
}()

// acquireCounts says how one lock type's acquires on this node were served:
// a key this node already held, a join of a held record, a cached record, a
// claim won on disk, and tries that had to wait.
Expand Down Expand Up @@ -349,6 +380,9 @@ func (m *Manager) LogStats() {
// Resources that hash to the same record share it: a node holding the record
// for one resource excludes other nodes from every resource in that record.
func (m *Manager) Acquire(ctx context.Context, lockType LockType, resourceID uint64, mode LockMode) error {
if int(lockType) >= numLockTypes {
return fmt.Errorf("lock: unknown lock type %d", lockType)
}
key := lockKey{Type: lockType, ResourceID: resourceID}
recordID := m.hashToRecord(lockType, resourceID)
gid := getGoroutineID()
Expand All @@ -359,7 +393,7 @@ func (m *Manager) Acquire(ctx context.Context, lockType LockType, resourceID uin
m.countAcquire(lockType, func(c *acquireCounts) *atomic.Uint64 { return &c.held })
return nil
}
if err := m.checkLockOrder(lockType); err != nil {
if err := m.checkLockOrder(key, recordID, gid); err != nil {
m.mu.Unlock()
return err
}
Expand Down Expand Up @@ -684,18 +718,19 @@ func (m *Manager) ReleaseCached() {
wg.Wait()
}

// checkLockOrder ensures we're acquiring locks in hierarchy order.
// Only checks against locks held by the CURRENT goroutine to avoid
// false positives when different goroutines hold different locks concurrently.
func (m *Manager) checkLockOrder(newType LockType) error {
gid := getGoroutineID()
for key, held := range m.heldLocks {
// Only check order for locks held by THIS goroutine
if held.goroutines != nil && held.goroutines[gid] > 0 {
if key.Type > newType {
return fmt.Errorf("%w: cannot acquire %v while holding %v",
ErrDeadlockRisk, newType, key.Type)
}
// checkLockOrder refuses a lock below one this goroutine holds: a lower type, or a lower record of the same type.
func (m *Manager) checkLockOrder(key lockKey, recordID, gid uint64) error {
for heldKey, held := range m.heldLocks {
if heldKey == key || held.goroutines[gid] == 0 {
continue
}
if heldKey.Type > key.Type {
return fmt.Errorf("%w: cannot acquire %v while holding %v",
ErrDeadlockRisk, key.Type, heldKey.Type)
}
if heldKey.Type == key.Type && held.RecordID > recordID {
return fmt.Errorf("%w: cannot acquire type %v in record %d while holding its record %d; take locks of one type in record order",
ErrDeadlockRisk, key.Type, recordID, held.RecordID)
}
}
return nil
Expand All @@ -710,18 +745,11 @@ func (m *Manager) hashToRecord(lockType LockType, resourceID uint64) uint64 {
hash *= 0xFF51AFD7ED558CCD
hash ^= hash >> 33

// Inode table sector locks get their own records. A sector lock is a leaf,
// so its holder never waits on another lock, and no cycle of waits between
// nodes can pass through a record only sector locks use.
records := m.size / RowSize
leaf := records / 16
if leaf == 0 {
if records < layoutRows || int(lockType) >= numLockTypes {
return hash % records
}
if lockType == LockTypeInodeSector {
return records - leaf + hash%leaf
}
return hash % (records - leaf)
return typeFirstRow[lockType] + hash%typeRows[lockType]
}

// SameRecord reports whether two locks map to one record, so one goroutine holding both holds the record in the stronger mode.
Expand Down
35 changes: 0 additions & 35 deletions internal/lock/sector_test.go

This file was deleted.

Loading
Loading