Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ DASHBOARD_DIST=internal/webui/build/dist
GOLANGCI_LINT_VERSION=v2.15.0
GOVULNCHECK_VERSION=v1.9.0
COVER_PROFILE=coverage.out
UNIT_PKGS ?= ./internal/... ./tests/soak
UNIT_PKGS ?= ./internal/... ./tests/soak ./tests/integration/history
UNIT_TIMEOUT ?= 20m

# Default target
Expand Down
2 changes: 1 addition & 1 deletion docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no

**Nodes keep released locks briefly.** After its last user releases a lock, a node keeps it for 10 ms (`CacheGrace`), so back-to-back work on the same node does not write and read the record again. A node reuses a lock it claimed for at most 250 ms (`HoldTurn`). After that, new work on that node waits up to another 250 ms for the current holders to finish. The node then releases the lock and waits 50 ms (`YieldGap`) before taking it again, so another node gets a turn. A node that claims a lock and finds another node's waiting mark gives it up the same way 250 ms after it first saw the mark, even if it released and claimed the lock again in between. Without this, a node in a lower slot that keeps taking a lock back every few milliseconds would shut out a waiting node in a higher slot. A mark counts only while its node is alive in the registry. Another node that wants the lock waits up to about 35 ms while the holder is idle, and up to about 300 ms while the holder keeps using it. A clean unmount releases kept locks at once. A crashed node's last locks, kept or held, stay held until another node proves it dead and replays its journal (see [Node Requirements](#node-requirements)). A lock wait gives up after 30 seconds (`DefaultTimeout`) while the holder is alive. A wait for a failed holder has no time limit: it logs a warning that names the node, and it ends when the replay does. It fails with `EIO` only if the replay fails or is abandoned, which shows as a lapsed recovery lease, with one exception. An operation that already holds a lock and then waits for a failed node's lock keeps the 30-second limit and fails with `EIO`, because the replay may need the lock it holds. An overwrite of a file's existing data does not: it lets go of its locks at once, waits for the block's lock holding none, and retries, so it does not fail. Operations that already changed something first, such as a create or a write that allocated blocks, keep the 30-second limit. For example, a node holds a directory's lock and waits for a block lock the dead node held, while the dead node's journal changes that directory. The replay's own waits have no limit, so it finishes once the operation gives up. The warning for this case names the record and the dead node.

**Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves.
**Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves. A read that overlaps a write of the same range on the same node can return old data in some 4 KB pieces and new data in others, as a Linux local filesystem can.

**Locks of one type share records.** Each lock type has its own records of the 256: 128 for inodes, 97 for block extents, 16 for inode table sectors, 8 for SFB containers, and one for each other type. Two files' inode locks can map to one record, and a node holding one of them blocks other nodes from both. A lock never waits on a record another type uses. A node takes locks in type order and, within a type, in record order, so two nodes cannot each wait for a record the other holds. The exception is a replay of a failed node's journal: the replaying node keeps the failed node's records until the replay ends, so a node that holds a lock the replay needs and waits on one of those records gives up after 30 seconds with `EIO`. The replay's waits have no limit, so it then finishes.

Expand Down
258 changes: 258 additions & 0 deletions tests/integration/coherence_history_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
package integration

import (
"encoding/binary"
"errors"
"fmt"
"io"
"io/fs"
"math/rand"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"syscall"
"testing"
"time"

"github.com/supporttools/GoClusterFS/tests/integration/history"
)

const historyWriteSize = 3 * 4096

var historyNames = []string{"n0", "n1", "n2", "n3", "n4"}

// TestCoherenceHistory runs clients on several nodes against a few shared names,
// records every call with its start and end, and checks the history against the
// documented cross-node guarantees. It runs only with GCFS_HISTORY=1; -short
// runs 2 nodes for 20s. For a long run set GCFS_HISTORY_MINUTES,
// GCFS_HISTORY_NODES (2 to 4) and GCFS_HISTORY_OUT, a directory for each round's
// history.
func TestCoherenceHistory(t *testing.T) {
if os.Getenv("GCFS_HISTORY") != "1" {
t.Skip("set GCFS_HISTORY=1 to run the coherence history checker")
}
nodes, clients, total := 3, 3, 2*time.Minute
if testing.Short() {
nodes, clients, total = 2, 2, 20*time.Second
}
if v, err := strconv.Atoi(os.Getenv("GCFS_HISTORY_MINUTES")); err == nil && v > 0 {
total = time.Duration(v) * time.Minute
}
if v, err := strconv.Atoi(os.Getenv("GCFS_HISTORY_NODES")); err == nil && v >= 2 && v <= 4 {
nodes = v
}
out := os.Getenv("GCFS_HISTORY_OUT")

cluster := NewMultiNodeCluster(t, nodes, 256)
if err := cluster.FormatAndStartFirst(false); err != nil {
t.Fatalf("format and start node 0: %v", err)
}
for i := 1; i < nodes; i++ {
if err := cluster.StartNode(i, false); err != nil {
t.Fatalf("start node %d: %v", i, err)
}
}
rules := history.Rules{Window: cluster.Node(0).attrTimeout() + 500*time.Millisecond, WriteSize: historyWriteSize}

start := time.Now()
for round := 0; time.Since(start) < total; round++ {
dir := fmt.Sprintf("h%d", round)
if err := cluster.Node(0).Mkdir(dir); err != nil {
t.Fatalf("mkdir %s: %v", dir, err)
}
time.Sleep(rules.Window)
ops := runHistoryRound(cluster, dir, clients, min(10*time.Minute, total-time.Since(start)))
violations := history.Check(ops, rules)
t.Logf("round %d: %d calls on %d nodes, %s; %d violations", round, len(ops), nodes, summarize(ops), len(violations))
if out != "" {
saveHistory(t, filepath.Join(out, fmt.Sprintf("round-%03d.txt", round)), ops, violations)
}
for i, v := range violations {
if i == 20 {
t.Errorf("round %d: %d more violations", round, len(violations)-20)
break
}
t.Errorf("round %d: %s", round, v)
}
if err := cluster.Node(0).RemoveAll(dir); err != nil {
t.Logf("remove %s: %v", dir, err)
}
}
}

// runHistoryRound runs clients goroutines per node on the names in dir until d has passed.
func runHistoryRound(cluster *MultiNodeCluster, dir string, clients int, d time.Duration) []history.Op {
start := time.Now()
deadline := start.Add(d)
var wg sync.WaitGroup
var mu sync.Mutex
var ops []history.Op
for n := range cluster.NodeCount() {
node := cluster.Node(n)
for c := range clients {
id := n*clients + c
wg.Add(1)
go func() {
defer wg.Done()
rng := rand.New(rand.NewSource(time.Now().UnixNano() + int64(id)))
var mine []history.Op
for seq := uint64(1); time.Now().Before(deadline); seq++ {
o := historyCall(node, dir, rng, uint64(id+1)<<32|seq, start)
o.Client, o.Node = id, n
mine = append(mine, o)
}
mu.Lock()
ops = append(ops, mine...)
mu.Unlock()
}()
}
}
wg.Wait()
for i := range ops {
ops[i].ID = i
}
return ops
}

// historyCall makes one random call and records it; value is unique to this call.
func historyCall(node *MountedFS, dir string, rng *rand.Rand, value uint64, start time.Time) history.Op {
name := historyNames[rng.Intn(len(historyNames))]
path := node.Path(dir + "/" + name)
o := history.Op{Name: name}
begin := time.Now()
var err error
switch p := rng.Intn(100); {
case p < 25:
o.Kind, o.Value = history.Write, value
err = writeValue(path, value)
case p < 50:
o.Kind = history.Read
o.Content, o.Value, o.Pieces, o.Size, err = readValue(path)
case p < 65:
o.Kind = history.Stat
var fi os.FileInfo
if fi, err = os.Stat(path); err == nil {
o.Size = fi.Size()
}
case p < 73:
o.Kind = history.Readdir
var entries []os.DirEntry
entries, err = os.ReadDir(node.Path(dir))
for _, e := range entries {
o.Names = append(o.Names, e.Name())
}
case p < 81:
o.Kind = history.Create
var f *os.File
if f, err = os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644); err == nil {
err = f.Close()
}
case p < 95:
o.Kind = history.Unlink
err = os.Remove(path)
default:
o.Kind, o.To = history.Rename, historyNames[(slices.Index(historyNames, name)+1+rng.Intn(len(historyNames)-1))%len(historyNames)]
err = os.Rename(path, node.Path(dir+"/"+o.To))
}
o.Start, o.End = begin.Sub(start), time.Since(start)
o.Result = classify(err)
if o.Result != history.OK {
o.Err = err.Error()
}
return o
}

func writeValue(path string, value uint64) error {
buf := make([]byte, historyWriteSize)
for i := 0; i < len(buf); i += 8 {
binary.LittleEndian.PutUint64(buf[i:], value)
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE, 0o644)
if err != nil {
return err
}
_, err = f.WriteAt(buf, 0)
return errors.Join(err, f.Close())
}

// readValue reads path whole: Whole when it holds one nonzero value over the
// write size, else Torn with each run of one value.
func readValue(path string) (history.Content, uint64, []history.Piece, int64, error) {
f, err := os.Open(path)
if err != nil {
return 0, 0, nil, 0, err
}
buf := make([]byte, historyWriteSize+1)
n, err := f.ReadAt(buf, 0)
if errors.Is(err, io.EOF) {
err = nil
}
if err = errors.Join(err, f.Close()); err != nil {
return 0, 0, nil, int64(n), err
}
if n == 0 {
return history.Empty, 0, nil, 0, nil
}
var pieces []history.Piece
for i := 0; i+8 <= n; i += 8 {
if w := binary.LittleEndian.Uint64(buf[i:]); len(pieces) == 0 || pieces[len(pieces)-1].Value != w {
pieces = append(pieces, history.Piece{At: int64(i), Value: w})
}
}
if len(pieces) == 1 && n == historyWriteSize && pieces[0].Value != 0 {
return history.Whole, pieces[0].Value, nil, int64(n), nil
}
return history.Torn, 0, pieces, int64(n), nil
}

func classify(err error) history.Result {
switch {
case err == nil:
return history.OK
case errors.Is(err, fs.ErrNotExist):
return history.NotFound
case errors.Is(err, fs.ErrExist):
return history.Exists
case errors.Is(err, syscall.ENOSYS), errors.Is(err, syscall.ENOTSUP), errors.Is(err, syscall.EXDEV):
return history.Unsupported
}
return history.Failed
}

func summarize(ops []history.Op) string {
counts := map[string]int{}
for _, o := range ops {
counts[o.Kind.String()+" "+o.Result.String()]++
}
var parts []string
for _, k := range []history.Kind{history.Write, history.Read, history.Stat, history.Readdir, history.Create, history.Unlink, history.Rename} {
for _, r := range []history.Result{history.OK, history.NotFound, history.Exists, history.Unsupported, history.Failed} {
if n := counts[k.String()+" "+r.String()]; n > 0 {
parts = append(parts, fmt.Sprintf("%s %s %d", k, r, n))
}
}
}
return strings.Join(parts, ", ")
}

func saveHistory(t *testing.T, path string, ops []history.Op, violations []history.Violation) {
t.Helper()
var b strings.Builder
for _, v := range violations {
b.WriteString(v.String())
}
b.WriteString("--- full history\n")
for _, o := range ops {
b.WriteString(o.String() + "\n")
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Logf("save history: %v", err)
return
}
if err := os.WriteFile(path, []byte(b.String()), 0o644); err != nil {
t.Logf("save history: %v", err)
}
}
Loading
Loading