diff --git a/Makefile b/Makefile index 6e60aed6..97f366db 100644 --- a/Makefile +++ b/Makefile @@ -22,7 +22,7 @@ DASHBOARD_DIST=internal/webui/build/dist GOLANGCI_LINT_VERSION=v2.15.0 GOVULNCHECK_VERSION=v1.9.0 COVER_PROFILE=coverage.out -UNIT_PKGS ?= ./internal/... ./tests/soak +UNIT_PKGS ?= ./internal/... ./tests/soak ./tests/integration/history UNIT_TIMEOUT ?= 20m # Default target diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 2f41408f..1941afe4 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -289,7 +289,7 @@ Lock slot reads and writes do not take the device's per-region I/O lock, so a no **Nodes keep released locks briefly.** After its last user releases a lock, a node keeps it for 10 ms (`CacheGrace`), so back-to-back work on the same node does not write and read the record again. A node reuses a lock it claimed for at most 250 ms (`HoldTurn`). After that, new work on that node waits up to another 250 ms for the current holders to finish. The node then releases the lock and waits 50 ms (`YieldGap`) before taking it again, so another node gets a turn. A node that claims a lock and finds another node's waiting mark gives it up the same way 250 ms after it first saw the mark, even if it released and claimed the lock again in between. Without this, a node in a lower slot that keeps taking a lock back every few milliseconds would shut out a waiting node in a higher slot. A mark counts only while its node is alive in the registry. Another node that wants the lock waits up to about 35 ms while the holder is idle, and up to about 300 ms while the holder keeps using it. A clean unmount releases kept locks at once. A crashed node's last locks, kept or held, stay held until another node proves it dead and replays its journal (see [Node Requirements](#node-requirements)). A lock wait gives up after 30 seconds (`DefaultTimeout`) while the holder is alive. A wait for a failed holder has no time limit: it logs a warning that names the node, and it ends when the replay does. It fails with `EIO` only if the replay fails or is abandoned, which shows as a lapsed recovery lease, with one exception. An operation that already holds a lock and then waits for a failed node's lock keeps the 30-second limit and fails with `EIO`, because the replay may need the lock it holds. An overwrite of a file's existing data does not: it lets go of its locks at once, waits for the block's lock holding none, and retries, so it does not fail. Operations that already changed something first, such as a create or a write that allocated blocks, keep the 30-second limit. For example, a node holds a directory's lock and waits for a block lock the dead node held, while the dead node's journal changes that directory. The replay's own waits have no limit, so it finishes once the operation gives up. The warning for this case names the record and the dead node. -**Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves. +**Writes to one region from two nodes do not mix within a block.** A node takes a block's lock for every write to an existing block, so two writes to the same block land one after the other. A write that spans several blocks takes each block's lock in turn, so if two nodes write overlapping multi-block ranges of a file at the same time, the result can mix both writes, one block at a time. Applications that write the same region from several nodes must coordinate those writes themselves. A read that overlaps a write of the same range on the same node can return old data in some 4 KB pieces and new data in others, as a Linux local filesystem can. **Locks of one type share records.** Each lock type has its own records of the 256: 128 for inodes, 97 for block extents, 16 for inode table sectors, 8 for SFB containers, and one for each other type. Two files' inode locks can map to one record, and a node holding one of them blocks other nodes from both. A lock never waits on a record another type uses. A node takes locks in type order and, within a type, in record order, so two nodes cannot each wait for a record the other holds. The exception is a replay of a failed node's journal: the replaying node keeps the failed node's records until the replay ends, so a node that holds a lock the replay needs and waits on one of those records gives up after 30 seconds with `EIO`. The replay's waits have no limit, so it then finishes. diff --git a/tests/integration/coherence_history_test.go b/tests/integration/coherence_history_test.go new file mode 100644 index 00000000..53c1b88c --- /dev/null +++ b/tests/integration/coherence_history_test.go @@ -0,0 +1,258 @@ +package integration + +import ( + "encoding/binary" + "errors" + "fmt" + "io" + "io/fs" + "math/rand" + "os" + "path/filepath" + "slices" + "strconv" + "strings" + "sync" + "syscall" + "testing" + "time" + + "github.com/supporttools/GoClusterFS/tests/integration/history" +) + +const historyWriteSize = 3 * 4096 + +var historyNames = []string{"n0", "n1", "n2", "n3", "n4"} + +// TestCoherenceHistory runs clients on several nodes against a few shared names, +// records every call with its start and end, and checks the history against the +// documented cross-node guarantees. It runs only with GCFS_HISTORY=1; -short +// runs 2 nodes for 20s. For a long run set GCFS_HISTORY_MINUTES, +// GCFS_HISTORY_NODES (2 to 4) and GCFS_HISTORY_OUT, a directory for each round's +// history. +func TestCoherenceHistory(t *testing.T) { + if os.Getenv("GCFS_HISTORY") != "1" { + t.Skip("set GCFS_HISTORY=1 to run the coherence history checker") + } + nodes, clients, total := 3, 3, 2*time.Minute + if testing.Short() { + nodes, clients, total = 2, 2, 20*time.Second + } + if v, err := strconv.Atoi(os.Getenv("GCFS_HISTORY_MINUTES")); err == nil && v > 0 { + total = time.Duration(v) * time.Minute + } + if v, err := strconv.Atoi(os.Getenv("GCFS_HISTORY_NODES")); err == nil && v >= 2 && v <= 4 { + nodes = v + } + out := os.Getenv("GCFS_HISTORY_OUT") + + cluster := NewMultiNodeCluster(t, nodes, 256) + if err := cluster.FormatAndStartFirst(false); err != nil { + t.Fatalf("format and start node 0: %v", err) + } + for i := 1; i < nodes; i++ { + if err := cluster.StartNode(i, false); err != nil { + t.Fatalf("start node %d: %v", i, err) + } + } + rules := history.Rules{Window: cluster.Node(0).attrTimeout() + 500*time.Millisecond, WriteSize: historyWriteSize} + + start := time.Now() + for round := 0; time.Since(start) < total; round++ { + dir := fmt.Sprintf("h%d", round) + if err := cluster.Node(0).Mkdir(dir); err != nil { + t.Fatalf("mkdir %s: %v", dir, err) + } + time.Sleep(rules.Window) + ops := runHistoryRound(cluster, dir, clients, min(10*time.Minute, total-time.Since(start))) + violations := history.Check(ops, rules) + t.Logf("round %d: %d calls on %d nodes, %s; %d violations", round, len(ops), nodes, summarize(ops), len(violations)) + if out != "" { + saveHistory(t, filepath.Join(out, fmt.Sprintf("round-%03d.txt", round)), ops, violations) + } + for i, v := range violations { + if i == 20 { + t.Errorf("round %d: %d more violations", round, len(violations)-20) + break + } + t.Errorf("round %d: %s", round, v) + } + if err := cluster.Node(0).RemoveAll(dir); err != nil { + t.Logf("remove %s: %v", dir, err) + } + } +} + +// runHistoryRound runs clients goroutines per node on the names in dir until d has passed. +func runHistoryRound(cluster *MultiNodeCluster, dir string, clients int, d time.Duration) []history.Op { + start := time.Now() + deadline := start.Add(d) + var wg sync.WaitGroup + var mu sync.Mutex + var ops []history.Op + for n := range cluster.NodeCount() { + node := cluster.Node(n) + for c := range clients { + id := n*clients + c + wg.Add(1) + go func() { + defer wg.Done() + rng := rand.New(rand.NewSource(time.Now().UnixNano() + int64(id))) + var mine []history.Op + for seq := uint64(1); time.Now().Before(deadline); seq++ { + o := historyCall(node, dir, rng, uint64(id+1)<<32|seq, start) + o.Client, o.Node = id, n + mine = append(mine, o) + } + mu.Lock() + ops = append(ops, mine...) + mu.Unlock() + }() + } + } + wg.Wait() + for i := range ops { + ops[i].ID = i + } + return ops +} + +// historyCall makes one random call and records it; value is unique to this call. +func historyCall(node *MountedFS, dir string, rng *rand.Rand, value uint64, start time.Time) history.Op { + name := historyNames[rng.Intn(len(historyNames))] + path := node.Path(dir + "/" + name) + o := history.Op{Name: name} + begin := time.Now() + var err error + switch p := rng.Intn(100); { + case p < 25: + o.Kind, o.Value = history.Write, value + err = writeValue(path, value) + case p < 50: + o.Kind = history.Read + o.Content, o.Value, o.Pieces, o.Size, err = readValue(path) + case p < 65: + o.Kind = history.Stat + var fi os.FileInfo + if fi, err = os.Stat(path); err == nil { + o.Size = fi.Size() + } + case p < 73: + o.Kind = history.Readdir + var entries []os.DirEntry + entries, err = os.ReadDir(node.Path(dir)) + for _, e := range entries { + o.Names = append(o.Names, e.Name()) + } + case p < 81: + o.Kind = history.Create + var f *os.File + if f, err = os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644); err == nil { + err = f.Close() + } + case p < 95: + o.Kind = history.Unlink + err = os.Remove(path) + default: + o.Kind, o.To = history.Rename, historyNames[(slices.Index(historyNames, name)+1+rng.Intn(len(historyNames)-1))%len(historyNames)] + err = os.Rename(path, node.Path(dir+"/"+o.To)) + } + o.Start, o.End = begin.Sub(start), time.Since(start) + o.Result = classify(err) + if o.Result != history.OK { + o.Err = err.Error() + } + return o +} + +func writeValue(path string, value uint64) error { + buf := make([]byte, historyWriteSize) + for i := 0; i < len(buf); i += 8 { + binary.LittleEndian.PutUint64(buf[i:], value) + } + f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE, 0o644) + if err != nil { + return err + } + _, err = f.WriteAt(buf, 0) + return errors.Join(err, f.Close()) +} + +// readValue reads path whole: Whole when it holds one nonzero value over the +// write size, else Torn with each run of one value. +func readValue(path string) (history.Content, uint64, []history.Piece, int64, error) { + f, err := os.Open(path) + if err != nil { + return 0, 0, nil, 0, err + } + buf := make([]byte, historyWriteSize+1) + n, err := f.ReadAt(buf, 0) + if errors.Is(err, io.EOF) { + err = nil + } + if err = errors.Join(err, f.Close()); err != nil { + return 0, 0, nil, int64(n), err + } + if n == 0 { + return history.Empty, 0, nil, 0, nil + } + var pieces []history.Piece + for i := 0; i+8 <= n; i += 8 { + if w := binary.LittleEndian.Uint64(buf[i:]); len(pieces) == 0 || pieces[len(pieces)-1].Value != w { + pieces = append(pieces, history.Piece{At: int64(i), Value: w}) + } + } + if len(pieces) == 1 && n == historyWriteSize && pieces[0].Value != 0 { + return history.Whole, pieces[0].Value, nil, int64(n), nil + } + return history.Torn, 0, pieces, int64(n), nil +} + +func classify(err error) history.Result { + switch { + case err == nil: + return history.OK + case errors.Is(err, fs.ErrNotExist): + return history.NotFound + case errors.Is(err, fs.ErrExist): + return history.Exists + case errors.Is(err, syscall.ENOSYS), errors.Is(err, syscall.ENOTSUP), errors.Is(err, syscall.EXDEV): + return history.Unsupported + } + return history.Failed +} + +func summarize(ops []history.Op) string { + counts := map[string]int{} + for _, o := range ops { + counts[o.Kind.String()+" "+o.Result.String()]++ + } + var parts []string + for _, k := range []history.Kind{history.Write, history.Read, history.Stat, history.Readdir, history.Create, history.Unlink, history.Rename} { + for _, r := range []history.Result{history.OK, history.NotFound, history.Exists, history.Unsupported, history.Failed} { + if n := counts[k.String()+" "+r.String()]; n > 0 { + parts = append(parts, fmt.Sprintf("%s %s %d", k, r, n)) + } + } + } + return strings.Join(parts, ", ") +} + +func saveHistory(t *testing.T, path string, ops []history.Op, violations []history.Violation) { + t.Helper() + var b strings.Builder + for _, v := range violations { + b.WriteString(v.String()) + } + b.WriteString("--- full history\n") + for _, o := range ops { + b.WriteString(o.String() + "\n") + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Logf("save history: %v", err) + return + } + if err := os.WriteFile(path, []byte(b.String()), 0o644); err != nil { + t.Logf("save history: %v", err) + } +} diff --git a/tests/integration/history/history.go b/tests/integration/history/history.go new file mode 100644 index 00000000..d1f25ce0 --- /dev/null +++ b/tests/integration/history/history.go @@ -0,0 +1,457 @@ +// Package history checks a record of concurrent file calls from several nodes +// against GoClusterFS's documented cross-node guarantees. +package history + +import ( + "fmt" + "slices" + "sort" + "strings" + "time" +) + +type Kind uint8 + +const ( + Create Kind = iota // open with O_CREAT|O_EXCL, then close + Write // open with O_CREAT, write one whole value at offset 0, close + Read // open, read the whole file, close + Stat // + Readdir // list the directory that holds every name + Unlink // + Rename // Name to To +) + +func (k Kind) String() string { + return [...]string{"create", "write", "read", "stat", "readdir", "unlink", "rename"}[k] +} + +type Result uint8 + +const ( + OK Result = iota + NotFound // ENOENT + Exists // EEXIST + Unsupported // the call is not implemented; it changed nothing + Failed // any other error: the call may or may not have taken effect +) + +func (r Result) String() string { + return [...]string{"ok", "ENOENT", "EEXIST", "unsupported", "failed"}[r] +} + +// Content is what a successful Read saw. +type Content uint8 + +const ( + Empty Content = iota // no bytes + Whole // exactly one value, Value, over the whole write size + Torn // anything else: two values, zeros, or a short file +) + +// Op is one call, timed by a clock all clients share. +type Op struct { + ID int // unique within a history + Client int + Node int + Kind Kind + Name, To string + Start time.Duration + End time.Duration + Result Result + Err string + Value uint64 // Write: the value written. Read: the value seen, when Content is Whole. + Pieces []Piece // Read: each run of one value, when Content is Torn + Content Content // Read + Size int64 // Stat + Names []string // Readdir +} + +func (o Op) String() string { + s := fmt.Sprintf("[%9.3fs %9.3fs] node %d client %d %s %s", o.Start.Seconds(), o.End.Seconds(), o.Node, o.Client, o.Kind, o.Name) + if o.Kind == Rename { + s += " -> " + o.To + } + switch { + case o.Result != OK: + s += " = " + o.Result.String() + if o.Err != "" { + s += " (" + o.Err + ")" + } + case o.Kind == Write: + s += fmt.Sprintf(" %#x", o.Value) + case o.Kind == Read && o.Content == Whole: + s += fmt.Sprintf(" = %#x", o.Value) + case o.Kind == Read && o.Content == Torn: + s += fmt.Sprintf(" = torn: %d bytes", o.Size) + for _, p := range o.Pieces { + s += fmt.Sprintf(", %#x from %d", p.Value, p.At) + } + case o.Kind == Read: + s += " = empty" + case o.Kind == Stat: + s += fmt.Sprintf(" = size %d", o.Size) + case o.Kind == Readdir: + s += " = [" + strings.Join(o.Names, " ") + "]" + } + return s +} + +// Piece is a run of one value in a torn read, starting at byte At. +type Piece struct { + At int64 + Value uint64 +} + +// Rules are the guarantees a history is checked against. +type Rules struct { + // Window is how long stat, lookups and listings on one node may show a name + // as it was before another node's change: -attr-timeout plus some slack. + Window time.Duration + // WriteSize is the size of every Write. Stat sizes must be 0 or WriteSize. + WriteSize int64 +} + +// Violation is an observation no allowed order of the history explains. +type Violation struct { + What string + Op Op + Allowed string + History []Op // the observation and every mutation that decides what it may see +} + +func (v Violation) String() string { + var b strings.Builder + fmt.Fprintf(&b, "%s: %s\n allowed: %s\n history:\n", v.What, v.Op, v.Allowed) + for _, o := range v.History { + fmt.Fprintf(&b, " %s\n", o) + } + return b.String() +} + +// state is one possible content of a name. +type state struct { + present bool + empty bool + any bool // a rename moved some content here + value uint64 +} + +func (s state) String() string { + switch { + case !s.present: + return "absent" + case s.any: + return "present" + case s.empty: + return "empty" + } + return fmt.Sprintf("%#x", s.value) +} + +// mutation is an Op's effect on one name. +type mutation struct { + op Op + after state + maybe bool // the call failed, so it may not have taken effect +} + +type nameLog struct { + byEnd []mutation // sorted by op.End + maxStart []time.Duration + maybes []mutation + maxDur time.Duration +} + +// Check returns every observation in ops that violates r. +func Check(ops []Op, r Rules) []Violation { + logs := map[string]*nameLog{} + add := func(name string, m mutation) { + l := logs[name] + if l == nil { + l = &nameLog{} + logs[name] = l + } + if m.maybe { + l.maybes = append(l.maybes, m) + } else { + l.byEnd = append(l.byEnd, m) + } + l.maxDur = max(l.maxDur, m.op.End-m.op.Start) + } + for _, o := range ops { + maybe := o.Result == Failed + if o.Result != OK && !maybe { + continue + } + switch o.Kind { + case Create: + add(o.Name, mutation{o, state{present: true, empty: true}, maybe}) + case Write: + add(o.Name, mutation{o, state{present: true, value: o.Value}, maybe}) + case Unlink: + add(o.Name, mutation{o, state{}, maybe}) + case Rename: + add(o.Name, mutation{o, state{}, maybe}) + add(o.To, mutation{o, state{present: true, any: true}, maybe}) + } + } + for _, l := range logs { + sort.Slice(l.byEnd, func(i, j int) bool { return l.byEnd[i].op.End < l.byEnd[j].op.End }) + l.maxStart = make([]time.Duration, len(l.byEnd)) + for i, m := range l.byEnd { + l.maxStart[i] = m.op.Start + if i > 0 { + l.maxStart[i] = max(l.maxStart[i], l.maxStart[i-1]) + } + } + } + + var out []Violation + for _, o := range ops { + switch { + case o.Result == Failed: + out = append(out, Violation{What: o.Kind.String() + " failed", Op: o, Allowed: "success, ENOENT or EEXIST", History: []Op{o}}) + case o.Kind == Write && o.Result == Exists: + out = append(out, Violation{What: "open with O_CREAT and no O_EXCL failed with EEXIST", Op: o, Allowed: "the file opened", History: []Op{o}}) + case o.Result != OK && o.Result != NotFound && o.Result != Exists: + case o.Kind == Read && o.Content == Torn && sectorTear(o, logs[o.Name], r): + case o.Kind == Read || o.Kind == Stat: + if v, bad := checkName(o, o.Name, observed(o, r), logs[o.Name], r); bad { + out = append(out, v) + } + case o.Kind == Unlink || o.Kind == Create: + if v, bad := checkName(o, o.Name, observed(o, r), logs[o.Name], r); bad { + out = append(out, v) + } + case o.Kind == Readdir: + for name, l := range logs { + listed := slices.Contains(o.Names, name) + if v, bad := checkName(o, name, func(s state) bool { return s.present == listed }, l, r); bad { + v.What = map[bool]string{true: "listing shows a name that should be gone", false: "listing misses a name that should exist"}[listed] + " (" + name + ")" + out = append(out, v) + } + } + } + } + return append(out, checkRenames(ops, logs, r)...) +} + +// observed returns which states explain what o saw of its name. +func observed(o Op, r Rules) func(state) bool { + switch { + case o.Kind == Read && o.Result == NotFound, o.Kind == Stat && o.Result == NotFound, + o.Kind == Unlink && o.Result == NotFound, o.Kind == Create && o.Result == OK: + return func(s state) bool { return !s.present } + case o.Kind == Unlink && o.Result == OK, o.Kind == Create && o.Result == Exists: + return func(s state) bool { return s.present } + case o.Kind == Read && o.Content == Torn: + return func(state) bool { return false } + case o.Kind == Read && o.Content == Empty: + return func(s state) bool { return s.present && (s.empty || s.any) } + case o.Kind == Read: + return func(s state) bool { return s.present && (s.any || !s.empty && s.value == o.Value) } + case o.Kind == Stat && o.Size == 0: + return func(s state) bool { return s.present && (s.empty || s.any) } + case o.Kind == Stat && o.Size == r.WriteSize: + return func(s state) bool { return s.present && (s.any || !s.empty) } + } + return func(state) bool { return false } +} + +// checkName finds the states o may see of name and reports a violation when seen accepts none of them. +func checkName(o Op, name string, seen func(state) bool, l *nameLog, r Rules) (Violation, bool) { + window := r.Window + if o.Kind == Read && stableName(o, l, r) { + window = 0 + } + allowed, deciding := possible(o, l, o.Start-window) + for _, s := range allowed { + if seen(s) { + return Violation{}, false + } + } + v := Violation{Op: o, What: describe(o, allowed, window == 0, r), History: append(deciding, o)} + var names []string + for _, s := range allowed { + names = append(names, s.String()) + } + v.Allowed = strings.Join(names, " or ") + if o.Kind == Read && o.Content == Whole && l != nil { + for _, m := range l.byEnd { + if m.op.Kind == Write && m.op.Value == o.Value && !slices.ContainsFunc(v.History, func(h Op) bool { return h.ID == m.op.ID }) { + v.History = append(v.History, m.op) + } + } + } + sort.SliceStable(v.History, func(i, j int) bool { return v.History[i].Start < v.History[j].Start }) + return v, true +} + +// possible returns the states o may see when changes that ended before settled +// must show, with the mutations that decide them. o's own effect is excluded. +func possible(o Op, l *nameLog, settled time.Duration) ([]state, []Op) { + if l == nil { + return []state{{}}, nil + } + var states []state + var deciding []Op + take := func(m mutation) { + if m.op.ID == o.ID { + return + } + if !slices.Contains(states, m.after) { + states = append(states, m.after) + } + deciding = append(deciding, m.op) + } + n := sort.Search(len(l.byEnd), func(i int) bool { return l.byEnd[i].op.End >= settled }) + latest := time.Duration(-1 << 62) + if n == 0 { + states = append(states, state{}) + } else { + latest = l.maxStart[n-1] + for i := sort.Search(n, func(i int) bool { return l.byEnd[i].op.End >= latest }); i < n; i++ { + take(l.byEnd[i]) + } + if len(states) == 0 { + states = append(states, state{}) + } + } + writing := false + l.during(settled, o.End, func(m mutation) { + take(m) + writing = writing || m.op.Kind == Write && m.op.ID != o.ID + }) + for _, m := range l.maybes { + if m.op.Start < o.End && m.op.End >= latest { + take(m) + writing = writing || m.op.Kind == Write + } + } + created := state{present: true, empty: true} + if writing && slices.Contains(states, state{}) && !slices.Contains(states, created) { + states = append(states, created) + } + return states, deciding +} + +// during calls fn for each sure mutation that ended at or after from and started before to. +func (l *nameLog) during(from, to time.Duration, fn func(mutation)) { + for i := sort.Search(len(l.byEnd), func(i int) bool { return l.byEnd[i].op.End >= from }); i < len(l.byEnd) && l.byEnd[i].op.End <= to+l.maxDur; i++ { + if l.byEnd[i].op.Start < to { + fn(l.byEnd[i]) + } + } +} + +// sectorTear reports whether a torn read only mixes values split at 4 KB +// boundaries while a write on the reader's node overlapped it, as Linux local +// filesystems allow, and every piece is a value the read may see. +func sectorTear(o Op, l *nameLog, r Rules) bool { + if o.Size != r.WriteSize || len(o.Pieces) < 2 || l == nil { + return false + } + allowed, _ := possible(o, l, o.Start-r.Window) + overlapping := false + for _, p := range o.Pieces { + if p.At%4096 != 0 || p.Value == 0 || !slices.Contains(allowed, state{present: true, value: p.Value}) { + return false + } + l.during(o.Start, o.End, func(m mutation) { + overlapping = overlapping || m.op.Kind == Write && m.op.Node == o.Node && m.op.Value == p.Value + }) + } + return overlapping +} + +// stableName reports whether name kept one file for a whole window before o +// and during it: then o's open finds that file, and must see its completed writes. +func stableName(o Op, l *nameLog, r Rules) bool { + if l == nil { + return false + } + before, _ := possible(o, l, o.Start-r.Window) + if slices.ContainsFunc(before, func(s state) bool { return !s.present }) { + return false + } + moved := false + l.during(o.Start-r.Window, o.End, func(m mutation) { moved = moved || m.op.Kind != Write }) + return !moved && !slices.ContainsFunc(l.maybes, func(m mutation) bool { + return m.op.Kind != Write && m.op.End >= o.Start-r.Window && m.op.Start < o.End + }) +} + +func describe(o Op, allowed []state, strict bool, r Rules) string { + present := slices.ContainsFunc(allowed, func(s state) bool { return s.present }) + absent := slices.ContainsFunc(allowed, func(s state) bool { return !s.present }) + suffix := "" + if strict { + suffix = " right after open" + } + switch { + case o.Kind == Read && o.Result == OK && o.Content == Torn: + return "torn read" + case o.Kind == Stat && o.Result == OK && o.Size != 0 && o.Size != r.WriteSize: + return fmt.Sprintf("stat size %d is no write's size", o.Size) + case !present && o.Result == OK && (o.Kind == Read || o.Kind == Stat || o.Kind == Unlink), + !present && o.Kind == Create && o.Result == Exists: + return o.Kind.String() + " found a name that should be gone" + case !absent && (o.Result == NotFound || o.Kind == Create && o.Result == OK): + return o.Kind.String() + " missed a name that should exist" + suffix + case o.Kind == Read && o.Content == Whole: + return "stale or unknown value" + suffix + } + return o.Kind.String() + " saw a state no order explains" + suffix +} + +// checkRenames checks that a rename onto a free name, with no other change to +// either name nearby, never shows in a listing as both names or neither. +func checkRenames(ops []Op, logs map[string]*nameLog, r Rules) []Violation { + var listings []Op + var longest time.Duration + for _, o := range ops { + if o.Kind == Readdir && o.Result == OK { + listings = append(listings, o) + longest = max(longest, o.End-o.Start) + } + } + sort.Slice(listings, func(i, j int) bool { return listings[i].Start < listings[j].Start }) + var out []Violation + for _, rn := range ops { + if rn.Kind != Rename || rn.Result != OK || !quiet(rn, logs[rn.Name], r) || !quiet(rn, logs[rn.To], r) { + continue + } + if before, _ := possible(rn, logs[rn.To], rn.Start-r.Window); slices.ContainsFunc(before, func(s state) bool { return s.present }) { + continue + } + for i := sort.Search(len(listings), func(i int) bool { return listings[i].Start >= rn.Start-longest }); i < len(listings) && listings[i].Start <= rn.End+r.Window; i++ { + d := listings[i] + if d.End < rn.Start { + continue + } + from, to := slices.Contains(d.Names, rn.Name), slices.Contains(d.Names, rn.To) + if from == to { + out = append(out, Violation{ + What: map[bool]string{true: "listing shows both names of a rename", false: "listing shows neither name of a rename"}[from], + Op: d, + Allowed: "exactly one of " + rn.Name + ", " + rn.To, + History: []Op{rn, d}, + }) + } + } + } + return out +} + +// quiet reports whether nothing but rn changed the name logged in l within a window of rn. +func quiet(rn Op, l *nameLog, r Rules) bool { + if l == nil { + return true + } + busy := false + l.during(rn.Start-r.Window, rn.End+r.Window, func(m mutation) { busy = busy || m.op.ID != rn.ID }) + return !busy && !slices.ContainsFunc(l.maybes, func(m mutation) bool { + return m.op.ID != rn.ID && m.op.End >= rn.Start-r.Window && m.op.Start < rn.End+r.Window + }) +} diff --git a/tests/integration/history/history_test.go b/tests/integration/history/history_test.go new file mode 100644 index 00000000..daa5498a --- /dev/null +++ b/tests/integration/history/history_test.go @@ -0,0 +1,178 @@ +package history + +import ( + "strings" + "testing" + "time" +) + +var rules = Rules{Window: time.Second, WriteSize: 12288} + +type builder struct{ ops []Op } + +func (b *builder) add(o Op, startMs, endMs int) *Op { + o.ID = len(b.ops) + o.Start, o.End = time.Duration(startMs)*time.Millisecond, time.Duration(endMs)*time.Millisecond + b.ops = append(b.ops, o) + return &b.ops[len(b.ops)-1] +} + +func (b *builder) write(name string, v uint64, s, e int) { + b.add(Op{Kind: Write, Name: name, Value: v}, s, e) +} +func (b *builder) read(name string, v uint64, s, e int) { + b.add(Op{Kind: Read, Name: name, Value: v, Content: Whole, Node: 1}, s, e) +} +func (b *builder) gone(kind Kind, name string, s, e int) { + b.add(Op{Kind: kind, Name: name, Result: NotFound, Node: 1}, s, e) +} +func (b *builder) unlink(name string, s, e int) { b.add(Op{Kind: Unlink, Name: name}, s, e) } +func (b *builder) list(s, e int, names ...string) { + b.add(Op{Kind: Readdir, Names: names, Node: 1}, s, e) +} + +func expect(t *testing.T, b *builder, want ...string) { + t.Helper() + got := Check(b.ops, rules) + if len(got) != len(want) { + for _, v := range got { + t.Log(v) + } + t.Fatalf("%d violations, want %d %q", len(got), len(want), want) + } + for i, v := range got { + if !strings.Contains(v.What, want[i]) { + t.Errorf("violation %d is %q, want %q\n%s", i, v.What, want[i], v) + } + } +} + +func TestConcurrentHistoryPasses(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.write("a", 2, 100, 200) + b.read("a", 1, 150, 160) + b.read("a", 2, 150, 160) + b.read("a", 2, 300, 310) + b.add(Op{Kind: Stat, Name: "a", Size: 12288}, 300, 301) + b.unlink("a", 400, 410) + b.read("a", 2, 500, 510) + b.gone(Read, "a", 500, 510) + b.gone(Stat, "a", 1500, 1510) + b.list(1500, 1510) + b.add(Op{Kind: Create, Name: "b"}, 1600, 1610) + b.add(Op{Kind: Read, Name: "b", Content: Empty}, 1700, 1710) + b.add(Op{Kind: Create, Name: "b", Result: Exists}, 1700, 1710) + b.list(1700, 1710) + b.list(2700, 2710, "b") + expect(t, b) +} + +func TestStaleReadRightAfterOpen(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.write("a", 2, 2000, 2010) + b.read("a", 1, 2050, 2060) + expect(t, b, "stale or unknown value right after open") +} + +func TestNameChangesMayLagTheWindow(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.unlink("a", 2000, 2010) + b.write("a", 2, 2020, 2030) + b.gone(Read, "a", 2100, 2110) + b.read("a", 1, 2100, 2110) + b.gone(Read, "a", 3100, 3110) + expect(t, b, "read missed a name that should exist") +} + +func TestTornRead(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.add(Op{Kind: Read, Name: "a", Content: Torn, Size: 12288, Pieces: []Piece{{0, 1}, {968, 2}}}, 20, 30) + expect(t, b, "torn read") +} + +func TestSectorTearDuringALocalWriteIsAllowed(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.add(Op{Kind: Write, Name: "a", Value: 2, Node: 1}, 20, 40) + tear := Op{Kind: Read, Name: "a", Content: Torn, Size: 12288, Node: 1, Pieces: []Piece{{0, 1}, {4096, 2}}} + b.add(tear, 25, 30) + b.add(Op{Kind: Write, Name: "b", Value: 3, Node: 0}, 20, 40) + b.add(Op{Kind: Read, Name: "b", Content: Torn, Size: 12288, Node: 1, Pieces: []Piece{{0, 0}, {4096, 3}}}, 25, 30) + tear.Node = 0 + b.add(tear, 25, 30) + expect(t, b, "torn read", "torn read") +} + +func TestUnknownValue(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.read("a", 7, 20, 30) + expect(t, b, "stale or unknown value") +} + +func TestResurrectedName(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.unlink("a", 20, 30) + b.add(Op{Kind: Stat, Name: "a", Size: 12288}, 2000, 2010) + b.list(2000, 2010, "a") + b.add(Op{Kind: Create, Name: "a", Result: Exists}, 2000, 2010) + expect(t, b, "stat found a name that should be gone", "listing shows a name that should be gone (a)", "create found a name that should be gone") +} + +func TestListingMissesCreatedName(t *testing.T) { + b := &builder{} + b.add(Op{Kind: Create, Name: "a"}, 0, 10) + b.list(2000, 2010) + b.gone(Unlink, "a", 2000, 2010) + expect(t, b, "listing misses a name that should exist (a)", "unlink missed a name that should exist") +} + +func TestFailedCallsMayOrMayNotTakeEffect(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.add(Op{Kind: Write, Name: "a", Value: 2, Result: Failed}, 100, 110) + b.read("a", 1, 2000, 2010) + b.read("a", 2, 2000, 2010) + b.add(Op{Kind: Unlink, Name: "b", Result: NotFound}, 100, 110) + b.add(Op{Kind: Rename, Name: "a", To: "c", Result: Unsupported}, 100, 110) + b.list(3000, 3010, "a") + expect(t, b, "write failed") +} + +func TestFailedCallSupersededByALaterOne(t *testing.T) { + b := &builder{} + b.add(Op{Kind: Write, Name: "a", Value: 2, Result: Failed}, 0, 10) + b.write("a", 1, 100, 110) + b.read("a", 2, 2000, 2010) + expect(t, b, "write failed", "stale or unknown value right after open") +} + +func TestCreatingWritePassesThroughEmpty(t *testing.T) { + b := &builder{} + b.write("a", 1, 100, 110) + b.add(Op{Kind: Stat, Name: "a"}, 105, 106) + b.add(Op{Kind: Read, Name: "a", Content: Empty}, 105, 106) + b.write("a", 2, 2000, 2010) + b.add(Op{Kind: Read, Name: "a", Content: Empty}, 2005, 2006) + expect(t, b, "read saw a state no order explains right after open") +} + +func TestNonExclusiveOpenNeverFailsWithEEXIST(t *testing.T) { + b := &builder{} + b.add(Op{Kind: Write, Name: "a", Value: 1, Result: Exists}, 0, 10) + expect(t, b, "open with O_CREAT and no O_EXCL failed with EEXIST") +} + +func TestRenameShowsExactlyOneName(t *testing.T) { + b := &builder{} + b.write("a", 1, 0, 10) + b.add(Op{Kind: Rename, Name: "a", To: "b"}, 2000, 2010) + b.list(2005, 2015, "a", "b") + b.list(2005, 2015, "b") + expect(t, b, "listing shows both names of a rename") +} diff --git a/tests/scripts/unit-groups.sh b/tests/scripts/unit-groups.sh index 0cf113ee..45bf9c57 100755 --- a/tests/scripts/unit-groups.sh +++ b/tests/scripts/unit-groups.sh @@ -12,7 +12,7 @@ default_seconds=5 list_packages() { local out - out=$(go list ./internal/... ./tests/soak) + out=$(go list ./internal/... ./tests/soak ./tests/integration/history) echo "$out" } diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index bb321cc6..bb8acd36 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -4524,3 +4524,83 @@ scenarios: priority: go_only environments: go: internal/cli/completion_test.go::TestBashCompletion + + - id: history-checker-accepts-concurrent + name: Coherence checker accepts concurrent histories + description: Overlapping writes, reads within the coherence window, unlinks and creates that the documented guarantees allow give no violations + priority: go_only + environments: + go: history/history_test.go::TestConcurrentHistoryPasses + + - id: history-checker-stale-read + name: Coherence checker flags a stale read after open + description: A read right after open that misses a write completed before it is a violation + priority: go_only + environments: + go: history/history_test.go::TestStaleReadRightAfterOpen + + - id: history-checker-name-window + name: Coherence checker allows name changes to lag the window + description: A name created or removed within -attr-timeout may still look old; after the window it may not + priority: go_only + environments: + go: history/history_test.go::TestNameChangesMayLagTheWindow + + - id: history-checker-torn-read + name: Coherence checker flags torn reads + description: A read that mixes values mid-sector, or with no write on its node overlapping, is a violation + priority: go_only + environments: + go: history/history_test.go::TestTornRead + + - id: history-checker-sector-tear + name: Coherence checker allows 4 KB tears during a local write + description: A read split at 4 KB boundaries between values a write on the same node was storing is allowed + priority: go_only + environments: + go: history/history_test.go::TestSectorTearDuringALocalWriteIsAllowed + + - id: history-checker-resurrection + name: Coherence checker flags resurrected names + description: A stat, listing or create that finds a name unlinked more than a window ago is a violation + priority: go_only + environments: + go: history/history_test.go::TestResurrectedName + + - id: history-checker-listing + name: Coherence checker flags listings missing created names + description: A listing or unlink that misses a name created more than a window ago is a violation + priority: go_only + environments: + go: history/history_test.go::TestListingMissesCreatedName + + - id: history-checker-failed-calls + name: Coherence checker treats failed calls as maybe applied + description: A failed write may or may not have taken effect, and is itself reported + priority: go_only + environments: + go: history/history_test.go::TestFailedCallsMayOrMayNotTakeEffect + + - id: history-checker-rename + name: Coherence checker checks rename atomicity in listings + description: A listing during or after a rename onto a free name shows exactly one of the two names + priority: go_only + environments: + go: history/history_test.go::TestRenameShowsExactlyOneName + + - id: history-checker-eexist + name: Coherence checker flags EEXIST from a non-exclusive open + description: open with O_CREAT and no O_EXCL must open the file, never fail with EEXIST + priority: go_only + environments: + go: history/history_test.go::TestNonExclusiveOpenNeverFailsWithEEXIST + + # Records every call of 2-4 mounts on one clock in one process; docker exec and ssh would add their own latency to each timing. Runs only with GCFS_HISTORY=1. + - id: coherence-history + name: Concurrent multi-node history meets the documented guarantees + description: Clients on 2-4 nodes create, write, read, stat, list, unlink and rename a few shared names; the recorded history has no violations + priority: medium + environments: + go: coherence_history_test.go::TestCoherenceHistory + docker: TODO + vm: TODO