From c6880484cf0e07725c7332fbb7e0352922a0c789 Mon Sep 17 00:00:00 2001 From: Matthew Mattox Date: Sat, 14 Nov 2026 10:02:08 -0600 Subject: [PATCH 1/3] fix(goclusterfs): refresh the cached inode when getting a file's attributes On a node whose broadcast messages from a writer did not arrive, a file held open could return short reads for up to 9 seconds after the writer grew it. Getattr read the new size from disk, so fstat showed it within -attr-timeout, but reads used the cached inode until -cache-ttl passed; with the writer still changing the file, each cache refresh caught a version that was already old, and the two kept disagreeing. Getattr now stores the inode it read, so reads catch up whenever the kernel rechecks the attributes, about every -attr-timeout while the file is read. Measured in Docker with tc netem and iptables (writer on one node, readers through an open handle, stat and a fresh open on three others): with multicast dropped or a one-way partition, the open handle's worst lag went from 4.7-9.4 s (4 of 10 runs over the 6 s bound) to 0.9-2.0 s. Loss up to 30%, duplication, reordering, 2 s delay and a flood under loss stayed within 2 s before and after. tests/docker/staleness is the measuring tool; six Docker tests run it under each fault, and TestMultiNodeOpenFileSeesGrowthWithoutBroadcast checks the fix without any network fault. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- docs/API-CONFIG.md | 2 +- docs/OPERATIONS.md | 2 +- internal/goclusterfs/fs.go | 4 + tests/docker/Dockerfile | 2 + tests/docker/scripts/run-tests.sh | 95 +++++++- tests/docker/staleness/main.go | 358 ++++++++++++++++++++++++++++ tests/integration/multinode_test.go | 50 ++++ tests/test-manifest.yaml | 68 ++++++ 10 files changed, 580 insertions(+), 4 deletions(-) create mode 100644 tests/docker/staleness/main.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 10eae16f..33b82c7d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -100,6 +100,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - Directory operations now update caches and send broadcast invalidations. - The broadcast peer table never dropped an entry: every remount and every forged heartbeat added one. Peers are now kept by address, at most `-broadcast-max-peers` of them, and so are the nodes whose message numbers a node tracks. - A cache that kept evicting entries grew in memory past its limit, because a Go map keeps the room of deleted keys: the extent cache took 5.8 MB when full and 10.9 MB after 40 times its size went through it. Each cache shard now rebuilds its map after it has evicted as many entries as it holds. +- A file held open on one node could return short reads for up to 9 seconds after another node grew it, when that node's broadcast messages did not arrive. `stat` read the new size from disk, but reads used the cached inode until `-cache-ttl` passed, and while the other node kept writing the two kept disagreeing. Getting a file's attributes now refreshes the cached inode too, so open files catch up within about twice `-attr-timeout`. - A read right after `open` could stop early when this node's kernel had cached the file's size during another node's write, for example from a desktop volume monitor or `ls -l` scanning the mount. Opening a file now makes the kernel fetch its size again. - A node that lost broadcast invalidations, for example when its receive buffer filled during a burst of writes on another node, kept stale inodes until `-cache-ttl` and logged nothing. It now sees the gap in the sender's sequence numbers, drops its inode and extent caches, and logs it at most every 10 seconds. The unmount stats line reports `lost=N`. - Broadcast deduplication kept every message for 5 minutes: about 224 MB per 2.4 million messages, and cleaning it up held up message handling for up to 200 ms. It now keeps a 64-message window per sender. diff --git a/README.md b/README.md index 6aba45f8..e54b9e6a 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod - **Linux only.** GoClusterFS needs FUSE and O_DIRECT. - **Every file with data uses at least one full 1 MB block.** The block size is 1 MB, and small-file packing (SFB) is disabled since [#12](https://github.com/SupportTools/GoClusterFS/pull/12), so `-sfb` has no effect. - **`ls -l` of a large directory is slow.** Each lookup reads two inodes from disk, about 0.3 ms on our test host, and `ls -l` repeats its lookups once it runs longer than `-attr-timeout`. In one test, `ls -l` took 0.14 s on 1,000 files and 14 s on 10,000. -- **Other nodes' changes can show up late.** The kernel caches file attributes and directory entries for `-attr-timeout` (default 1s). For that long, `stat` and `ls` can show another node's file with its old size and mtime. A file that stays open can use the old size and block mappings for up to `-cache-ttl` (default 5s). Opening the file again reads the current data. See [Cache Flags](docs/API-CONFIG.md#cache-flags). +- **Other nodes' changes can show up late.** The kernel caches file attributes and directory entries for `-attr-timeout` (default 1s). For that long, `stat` and `ls` can show another node's file with its old size and mtime. A file that stays open can use the old size and block mappings for about twice `-attr-timeout`. Opening the file again reads the current data. See [Cache Flags](docs/API-CONFIG.md#cache-flags). - **On-disk format version 3.** Mount refuses images formatted by earlier builds, including version 3 images formatted before [#85](https://github.com/SupportTools/GoClusterFS/pull/85) gave each lock type its own lock records. Copy the data off with the build that formatted the image, reformat, and copy it back. - **The inode count is set at format time.** `-format` creates one inode per `-bytes-per-inode` bytes of device (default 1 MB, so one per data block), between 4096 and 16,777,216. It cannot change later. - **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md). diff --git a/docs/API-CONFIG.md b/docs/API-CONFIG.md index 77294d6e..c529ec55 100644 --- a/docs/API-CONFIG.md +++ b/docs/API-CONFIG.md @@ -54,7 +54,7 @@ These are the `goclusterfs-mount` flags: Each cache splits its limit evenly over 16 shards, so it holds a multiple of 16 entries: `-cache-negative 1000` holds 992, and a limit from 1 to 15 holds 16. A full cache drops the entry used longest ago. -The kernel caches file attributes and directory entries for `-attr-timeout`. For that long after another node changes a file, `stat` and path lookups on this node can return the old size. Opening a file reads its inode from disk and makes the kernel fetch its size again, so a read after `open` sees the other node's completed writes. Reads through a file that is already open use the inode cache and can use the old size and block mappings for up to `-cache-ttl`, including a block that another node has since freed and reused for other data. With broadcast enabled, writes and directory changes invalidate peers' caches sooner; truncate, fallocate and unlink do not. +The kernel caches file attributes and directory entries for `-attr-timeout`. For that long after another node changes a file, `stat` and path lookups on this node can return the old size. Opening a file reads its inode from disk and makes the kernel fetch its size again, so a read after `open` sees the other node's completed writes. Reads through a file that is already open use the inode cache. Each time the kernel rechecks the file's attributes, every `-attr-timeout` while the file is read, the mount refreshes that cache from disk, so an open file sees another node's completed writes within about twice `-attr-timeout` (2s at the default), even when broadcast messages are lost. Until then it can use the old size and block mappings, including a block that another node has since freed and reused for other data. With broadcast enabled, writes and directory changes invalidate peers' caches sooner; truncate, fallocate and unlink do not. `-attr-timeout=0` makes `stat` and path lookups always current, but each `stat` and path step is then a lookup that reads the directory's and the file's inodes from disk: about 0.7 ms per `stat` on the test host, against well under 0.1 ms from the kernel cache. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 86a5f59e..ac642198 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -544,7 +544,7 @@ The flags set how many entries each cache holds and how long cached data may be | Flag | Default | Effect | |------|---------|--------| | `-attr-timeout` | `1s` | How long the kernel caches attributes and directory entries. `stat` and `ls` can show another node's change up to this late. `0` makes every `stat` and path step a lookup that reads the directory's and the file's inodes from disk. | -| `-cache-ttl` | `5s` | How long the mount trusts a cached inode before it checks the disk. Open files can lag another node's change by up to this long. | +| `-cache-ttl` | `5s` | How long the mount trusts a cached inode before it checks the disk. Directory lookups can lag another node's change by up to this long; open files catch up within about twice `-attr-timeout`. | | `-cache-strict` | `false` | Check the disk on every cache hit | | `-cache-inodes` | `10000` | Inodes kept in memory, about 480 bytes each | | `-cache-extents` | `50000` | Block mappings kept in memory, about 120 bytes each | diff --git a/internal/goclusterfs/fs.go b/internal/goclusterfs/fs.go index ca6f0a36..dfb7aeeb 100644 --- a/internal/goclusterfs/fs.go +++ b/internal/goclusterfs/fs.go @@ -1690,6 +1690,10 @@ func (f *File) Getattr(ctx context.Context, fh fs.FileHandle, out *fuse.AttrOut) } logging.Debugf("File.Getattr: inode=%d Size=%d AllocatedBlocks=%d", f.inodeNum, ino.Size, ino.AllocatedBlocks) + // Reads use the cached inode; they must not stop at an older size than stat shows. + if ino.Magic == inode.InodeMagic { + f.fs.cacheInode(ino) + } // Zero means go-fuse applies the mount's -attr-timeout. out.AttrValid = 0 diff --git a/tests/docker/Dockerfile b/tests/docker/Dockerfile index a9cca5fc..b54743b0 100644 --- a/tests/docker/Dockerfile +++ b/tests/docker/Dockerfile @@ -17,6 +17,7 @@ COPY . . # Build the mount binary RUN CGO_ENABLED=0 GOOS=linux go build -o goclusterfs-mount ./cmd/goclusterfs-mount +RUN CGO_ENABLED=0 GOOS=linux go build -o gcfs-staleness ./tests/docker/staleness # ============================================================================= # Stage 2: Runtime - Minimal image with FUSE and network tools @@ -42,6 +43,7 @@ RUN mkdir -p /mnt/goclusterfs /data /scripts # Copy binary from builder COPY --from=builder /build/goclusterfs-mount /usr/local/bin/ +COPY --from=builder /build/gcfs-staleness /usr/local/bin/ # Copy container scripts COPY tests/docker/scripts/ /scripts/ diff --git a/tests/docker/scripts/run-tests.sh b/tests/docker/scripts/run-tests.sh index ad382f51..7fd99d0a 100644 --- a/tests/docker/scripts/run-tests.sh +++ b/tests/docker/scripts/run-tests.sh @@ -273,7 +273,7 @@ in_test_group() { [[ -z "$TEST_GROUP" ]] && return 0 local group=1 case "$1" in - "Container Kill Recovery"|Journal*|"Network Partition"|Broadcast*|"Inode Reuse Stale Read"|"Rolling Restart") + "Container Kill Recovery"|Journal*|"Network Partition"|Broadcast*|Staleness*|"Inode Reuse Stale Read"|"Rolling Restart") group=2 ;; # Takes about 14 minutes in CI, so it runs alone. "More Than 4096 Files") @@ -2466,6 +2466,93 @@ test_dir_index_follows_other_node() { return 0 } +heal_network_faults() { + local node + for node in "${NODE_ARRAY[@]}"; do + exec_on_node "$node" sh -c 'tc qdisc del dev eth0 root 2>/dev/null; iptables -F INPUT; iptables -F OUTPUT' || true + done +} + +# measure_staleness has node 1 rewrite a file 20 times, growing and truncating +# it, while node 2 reads it through a file it holds open, node 3 stats it, and +# node 4 opens it afresh for each look (or floods invalidations with "flood"). +# Fails if any of them sees a version later than its bound allows, with the +# default -attr-timeout 1s and -cache-ttl 5s. +measure_staleness() { + local name=$1 flood=${2:-} + local writer="${NODE_ARRAY[0]}" held="${NODE_ARRAY[1]}" stat="${NODE_ARRAY[2]}" fresh="${NODE_ARRAY[3]}" + local file + file="/mnt/goclusterfs/staleness_${name}_$(date +%s%N)" + local out="$RESULTS_DIR/staleness_$name" + local pids=() logs="$out.open,$out.stat" + + exec_on_node "$held" gcfs-staleness read -file "$file" -duration 20s -modes open > "$out.open" 2>&1 & + pids+=($!) + exec_on_node "$stat" gcfs-staleness read -file "$file" -duration 20s -modes stat > "$out.stat" 2>&1 & + pids+=($!) + if [[ "$flood" == flood ]]; then + exec_on_node "$fresh" gcfs-staleness flood -dir "/mnt/goclusterfs/flood_$name" -duration 12s > "$out.flood" 2>&1 & + else + exec_on_node "$fresh" gcfs-staleness read -file "$file" -duration 20s -modes reopen > "$out.reopen" 2>&1 & + logs="$logs,$out.reopen" + fi + pids+=($!) + sleep 0.5 + exec_on_node "$writer" gcfs-staleness write -file "$file" -versions 20 > "$out.w" 2>&1 + wait "${pids[@]}" + heal_network_faults + + local result=0 + gcfs-staleness analyze -writer "$out.w" -reader "$logs" -bound-open 3s -bound-stat 1500ms -bound-reopen 500ms > "$out.result" 2>&1 || result=1 + while read -r line; do log " $line"; done < "$out.result" + exec_on_node "$writer" rm -rf "$file" "/mnt/goclusterfs/flood_$name" + return $result +} + +apply_fault() { + local node=$1 + shift + if ! exec_on_node "$node" "$@"; then + log "ERROR: could not apply network fault on $node: $*" + heal_network_faults + return 1 + fi +} + +test_staleness_packet_loss() { + apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem loss 30% || return 1 + measure_staleness packet_loss +} + +test_staleness_duplication_and_reordering() { + apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem delay 50ms reorder 50% 50% duplicate 30% || return 1 + measure_staleness reordering +} + +test_staleness_delay() { + apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem delay 1000ms 1000ms || return 1 + measure_staleness delay +} + +test_staleness_one_way_partition() { + local writer_ip + writer_ip=$(exec_on_node "${NODE_ARRAY[0]}" hostname -i | awk '{print $1}') + apply_fault "${NODE_ARRAY[1]}" iptables -I INPUT -s "$writer_ip" -j DROP || return 1 + measure_staleness one_way_partition +} + +# The reader hears no multicast at all, as behind a switch that drops it. +test_staleness_multicast_dropped() { + apply_fault "${NODE_ARRAY[1]}" iptables -I INPUT -d 239.192.74.75 -j DROP || return 1 + measure_staleness multicast_dropped +} + +test_staleness_flood_under_loss() { + apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem loss 10% || return 1 + apply_fault "${NODE_ARRAY[1]}" iptables -I INPUT -p udp -m statistic --mode random --probability 0.1 -j DROP || return 1 + measure_staleness flood_under_loss flood +} + # Print the free 1MB blocks that statfs reports on node $1 statfs_free_blocks() { exec_on_node "$1" stat -f -c %f /mnt/goclusterfs @@ -3330,6 +3417,12 @@ run_test "Broadcast Invalidation" test_broadcast_invalidation run_test "Broadcast Directory Invalidation" test_broadcast_directory_invalidation run_test "Broadcast Multiple Updates" test_broadcast_multiple_updates run_test "Broadcast Bidirectional" test_broadcast_bidirectional +run_test "Staleness Under Packet Loss" test_staleness_packet_loss +run_test "Staleness Under Duplication and Reordering" test_staleness_duplication_and_reordering +run_test "Staleness Under Delay" test_staleness_delay +run_test "Staleness Under One-Way Partition" test_staleness_one_way_partition +run_test "Staleness With Multicast Dropped" test_staleness_multicast_dropped +run_test "Staleness During Flood Under Loss" test_staleness_flood_under_loss run_test "Inode Reuse Stale Read" test_inode_reuse_stale_read run_test "More Than 4096 Files" test_more_than_4096_files run_test "Extent Cache Coherency" test_extent_cache_coherency diff --git a/tests/docker/staleness/main.go b/tests/docker/staleness/main.go new file mode 100644 index 00000000..0ffd2d3d --- /dev/null +++ b/tests/docker/staleness/main.go @@ -0,0 +1,358 @@ +// Command gcfs-staleness measures how late one node sees another node's +// writes. The writer stamps each version into the file's size and its last +// bytes, which every other version moves into blocks it maps again. The reader watches the file three ways and logs every change it +// sees, and analyze turns both logs into per-version lag. +// +// gcfs-staleness write -file F -versions N +// gcfs-staleness read -file F -duration D -modes open,stat,reopen +// gcfs-staleness flood -dir D -duration D +// gcfs-staleness analyze -writer W -reader R -bound-open 6s -bound-stat 1.5s -bound-reopen 500ms +package main + +import ( + "bufio" + "flag" + "fmt" + "math/rand/v2" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +const ( + headerLen = 11 + bigBase = 2 << 20 + smallBase = 8192 + sizeStep = 16 +) + +// sizeOf alternates between a 2 MB file and an 8 KB one, so every other +// version truncates away blocks and the next one maps new ones. +func sizeOf(v int) int64 { + if v%2 == 0 { + return bigBase + int64(v)*sizeStep + } + return smallBase + int64(v)*sizeStep +} + +func versionOfSize(size int64) int { + base := int64(smallBase) + if size >= bigBase { + base = bigBase + } + if size < base || (size-base)%sizeStep != 0 { + return -1 + } + return int((size - base) / sizeStep) +} + +func header(v int) []byte { return []byte(fmt.Sprintf("%010d\n", v)) } + +func versionOfHeader(b []byte) int { + if len(b) < headerLen || b[headerLen-1] != '\n' { + return -1 + } + v, err := strconv.Atoi(string(b[:headerLen-1])) + if err != nil { + return -1 + } + return v +} + +func fail(format string, args ...any) { + fmt.Fprintf(os.Stderr, format+"\n", args...) + os.Exit(2) +} + +func write(args []string) { + fs := flag.NewFlagSet("write", flag.ExitOnError) + file := fs.String("file", "", "file to write") + versions := fs.Int("versions", 100, "number of versions") + minGap := fs.Duration("min", 100*time.Millisecond, "shortest pause between versions") + maxGap := fs.Duration("max", 600*time.Millisecond, "longest pause between versions") + _ = fs.Parse(args) + + out := bufio.NewWriter(os.Stdout) + defer out.Flush() + for v := 1; v <= *versions; v++ { + f, err := os.OpenFile(*file, os.O_RDWR|os.O_CREATE, 0o644) + if err != nil { + fail("open: %v", err) + } + if err := f.Truncate(sizeOf(v)); err != nil { + fail("truncate v%d: %v", v, err) + } + if _, err := f.WriteAt(header(v), sizeOf(v)-headerLen); err != nil { + fail("write v%d: %v", v, err) + } + if err := f.Close(); err != nil { + fail("close v%d: %v", v, err) + } + fmt.Fprintf(out, "W %d %d\n", time.Now().UnixNano(), v) + out.Flush() + gap := *minGap + time.Duration(rand.Int64N(int64(*maxGap-*minGap)+1)) + time.Sleep(gap) + } +} + +// observation is what one way of looking at the file saw: the version its +// size says and the version its last bytes say. -1 means unreadable. +type observation struct{ size, data int } + +func read(args []string) { + fs := flag.NewFlagSet("read", flag.ExitOnError) + file := fs.String("file", "", "file to watch") + duration := fs.Duration("duration", time.Minute, "how long to watch") + poll := fs.Duration("poll", 10*time.Millisecond, "pause between looks") + modes := fs.String("modes", "open,stat,reopen", "ways to look; one node's watchers share its kernel caches, so run one per node to measure each alone") + _ = fs.Parse(args) + + var mu sync.Mutex + out := bufio.NewWriter(os.Stdout) + defer out.Flush() + logChange := func(mode string, o observation) { + mu.Lock() + fmt.Fprintf(out, "R %d %s %d %d\n", time.Now().UnixNano(), mode, o.size, o.data) + out.Flush() + mu.Unlock() + } + watch := func(mode string, look func() observation) { + last := observation{-2, -2} + for end := time.Now().Add(*duration); time.Now().Before(end); time.Sleep(*poll) { + if o := look(); o != last { + logChange(mode, o) + last = o + } + } + } + readTail := func(f *os.File, size int64) int { + buf := make([]byte, headerLen) + if n, err := f.ReadAt(buf, size-headerLen); n < headerLen && err != nil { + return -1 + } + return versionOfHeader(buf) + } + + var held *os.File + for deadline := time.Now().Add(30 * time.Second); held == nil; time.Sleep(*poll) { + f, err := os.Open(*file) + if err == nil { + held = f + } else if time.Now().After(deadline) { + fail("open %s: %v", *file, err) + } + } + defer held.Close() + + looks := map[string]func() observation{ + "open": func() observation { + fi, err := held.Stat() + if err != nil { + return observation{-1, -1} + } + return observation{versionOfSize(fi.Size()), readTail(held, fi.Size())} + }, + "stat": func() observation { + fi, err := os.Stat(*file) + if err != nil { + return observation{-1, -1} + } + return observation{versionOfSize(fi.Size()), 0} + }, + "reopen": func() observation { + f, err := os.Open(*file) + if err != nil { + return observation{-1, -1} + } + defer f.Close() + fi, err := f.Stat() + if err != nil { + return observation{-1, -1} + } + return observation{versionOfSize(fi.Size()), readTail(f, fi.Size())} + }, + } + var wg sync.WaitGroup + for _, mode := range strings.Split(*modes, ",") { + look, ok := looks[mode] + if !ok { + fail("unknown mode %q", mode) + } + wg.Add(1) + go func() { + defer wg.Done() + watch(mode, look) + }() + } + wg.Wait() +} + +func flood(args []string) { + fs := flag.NewFlagSet("flood", flag.ExitOnError) + dir := fs.String("dir", "", "directory for the flood files") + files := fs.Int("files", 200, "number of files to rewrite in turn") + duration := fs.Duration("duration", time.Minute, "how long to flood") + _ = fs.Parse(args) + + if err := os.MkdirAll(*dir, 0o755); err != nil { + fail("mkdir: %v", err) + } + handles := make([]*os.File, *files) + for i := range handles { + f, err := os.OpenFile(filepath.Join(*dir, fmt.Sprintf("f%04d", i)), os.O_RDWR|os.O_CREATE, 0o644) + if err != nil { + fail("open: %v", err) + } + defer f.Close() + handles[i] = f + } + buf := make([]byte, 512) + writes := 0 + for end := time.Now().Add(*duration); time.Now().Before(end); writes++ { + if _, err := handles[writes%len(handles)].WriteAt(buf, int64(writes/len(handles))%64*512); err != nil { + fail("write: %v", err) + } + } + fmt.Printf("flood: %d writes in %v\n", writes, *duration) +} + +type change struct { + at int64 + obs observation +} + +func analyze(args []string) { + fs := flag.NewFlagSet("analyze", flag.ExitOnError) + writerLog := fs.String("writer", "", "writer log") + readerLog := fs.String("reader", "", "reader logs, comma separated") + bounds := map[string]*time.Duration{ + "open": fs.Duration("bound-open", 6*time.Second, "most lag allowed through a file held open"), + "stat": fs.Duration("bound-stat", 1500*time.Millisecond, "most lag allowed for stat on the path"), + "reopen": fs.Duration("bound-reopen", 500*time.Millisecond, "most lag allowed for a fresh open"), + } + _ = fs.Parse(args) + + done := map[int]int64{} + maxVersion := 0 + scan(*writerLog, func(f []string) { + if f[0] == "W" { + at, _ := strconv.ParseInt(f[1], 10, 64) + v, _ := strconv.Atoi(f[2]) + done[v] = at + maxVersion = max(maxVersion, v) + } + }) + changes := map[string][]change{} + for _, readerLog := range strings.Split(*readerLog, ",") { + scan(readerLog, func(f []string) { + if f[0] == "R" && len(f) == 5 { + at, _ := strconv.ParseInt(f[1], 10, 64) + size, _ := strconv.Atoi(f[3]) + data, _ := strconv.Atoi(f[4]) + changes[f[2]] = append(changes[f[2]], change{at, observation{size, data}}) + } + }) + } + if maxVersion == 0 { + fail("writer log has no versions") + } + + failed := false + for _, mode := range []string{"reopen", "stat", "open"} { + cs, ok := changes[mode] + if !ok { + continue + } + var lags []time.Duration + var worst time.Duration + worstVersion, bad := 0, 0 + for v := 1; v <= maxVersion; v++ { + seen := int64(-1) + for _, c := range cs { + if c.obs.size >= v && (mode == "stat" || c.obs.data >= v) { + seen = c.at + break + } + } + if seen < 0 { + continue + } + lag := time.Duration(max(0, seen-done[v])) + lags = append(lags, lag) + if lag > worst { + worst, worstVersion = lag, v + } + } + regressions, highest := 0, observation{} + for _, c := range cs { + if c.obs.size == -1 || (mode != "stat" && c.obs.data == -1) { + bad++ + continue + } + if c.obs.size < highest.size || c.obs.data < highest.data { + regressions++ + } + highest = observation{max(highest.size, c.obs.size), max(highest.data, c.obs.data)} + } + if regressions > 0 { + failed = true + } + sort.Slice(lags, func(i, j int) bool { return lags[i] < lags[j] }) + median := time.Duration(0) + if len(lags) > 0 { + median = lags[len(lags)/2] + } + verdict := "ok" + if worst > *bounds[mode] { + verdict = "OUT OF BOUND" + failed = true + } + if regressions > 0 { + verdict = "WENT BACKWARDS" + } + fmt.Printf("%-6s versions seen %d/%d median %v worst %v (v%d) bound %v unreadable looks %d regressions %d %s\n", + mode, len(lags), maxVersion, median.Round(time.Millisecond), worst.Round(time.Millisecond), worstVersion, *bounds[mode], bad, regressions, verdict) + if len(lags) < maxVersion-1 { + fmt.Printf("%-6s missed %d versions entirely\n", mode, maxVersion-len(lags)) + } + } + if failed { + os.Exit(1) + } +} + +func scan(path string, each func([]string)) { + f, err := os.Open(path) + if err != nil { + fail("%v", err) + } + defer f.Close() + s := bufio.NewScanner(f) + for s.Scan() { + if fields := strings.Fields(s.Text()); len(fields) > 0 { + each(fields) + } + } +} + +func main() { + if len(os.Args) < 2 { + fail("usage: gcfs-staleness write|read|flood|analyze [flags]") + } + switch os.Args[1] { + case "write": + write(os.Args[2:]) + case "read": + read(os.Args[2:]) + case "flood": + flood(os.Args[2:]) + case "analyze": + analyze(os.Args[2:]) + default: + fail("unknown command %q", os.Args[1]) + } +} diff --git a/tests/integration/multinode_test.go b/tests/integration/multinode_test.go index 48cf6fb5..e6fa1e8c 100644 --- a/tests/integration/multinode_test.go +++ b/tests/integration/multinode_test.go @@ -1280,6 +1280,56 @@ func TestMultiNodeOpenRefreshesCachedSize(t *testing.T) { expectContent(t, reader, name, want) } +// TestMultiNodeOpenFileSeesGrowthWithoutBroadcast checks that a file held open +// on a node that gets no broadcast reads another node's growth once its stat +// shows it. With -cache-ttl at a minute, only the stat refreshing the cached +// inode makes the new end of the file readable in time. +func TestMultiNodeOpenFileSeesGrowthWithoutBroadcast(t *testing.T) { + cluster := NewMultiNodeCluster(t, 2, 50) + if err := cluster.FormatAndStartFirst(false); err != nil { + t.Fatalf("Failed to format and start first node: %v", err) + } + cluster.Node(1).ExtraArgs = []string{"-broadcast=false", "-cache-ttl=1m"} + if err := cluster.StartNode(1, false); err != nil { + t.Fatalf("Failed to start second node: %v", err) + } + writer, reader := cluster.Node(0), cluster.Node(1) + + const name = "held.bin" + writeOrFail(t, writer, name, bytes.Repeat([]byte("a"), 8<<10), 0) + f, err := os.Open(reader.Path(name)) + if err != nil { + t.Fatal(err) + } + defer f.Close() + buf := make([]byte, 8) + if _, err := f.ReadAt(buf, 8<<10-8); err != nil { + t.Fatalf("reading the first version: %v", err) + } + + const grown = 2<<20 + 100 + writeOrFail(t, writer, name, []byte("tailtail"), grown-8) + start := time.Now() + for { + fi, err := f.Stat() + if err != nil { + t.Fatal(err) + } + if fi.Size() == grown { + n, err := f.ReadAt(buf, grown-8) + if n == 8 && string(buf) == "tailtail" { + t.Logf("read the new end %v after the write", time.Since(start).Round(time.Millisecond)) + return + } + t.Logf("stat shows %d bytes, read at the end: %d bytes, %v", fi.Size(), n, err) + } + if time.Since(start) > 3*time.Second { + t.Fatalf("could not read the new end of %s 3s after the write; stat shows %d bytes", name, fi.Size()) + } + time.Sleep(50 * time.Millisecond) + } +} + func expectSize(t *testing.T, m *MountedFS, name string, want int64) { t.Helper() fi, err := m.Stat(name) diff --git a/tests/test-manifest.yaml b/tests/test-manifest.yaml index ce5ae0dc..ef353400 100644 --- a/tests/test-manifest.yaml +++ b/tests/test-manifest.yaml @@ -484,6 +484,74 @@ scenarios: docker: run-tests.sh::test_network_partition vm: run-vm-tests.sh::test_network_partition + - id: open-file-growth-without-broadcast + name: Open file reads growth without broadcast + description: A node with broadcast off and a long cache TTL holds a file open while another node grows it; once stat shows the new size, a read at the new end returns the data + priority: high + environments: + go: multinode_test.go::TestMultiNodeOpenFileSeesGrowthWithoutBroadcast + docker: run-tests.sh::test_staleness_multicast_dropped + vm: TODO + + # Staleness under network faults. Each test has node 1 rewrite a file while + # node 2 reads it through an open handle, node 3 stats it and node 4 opens it + # afresh, and checks how late each sees every version. Integration tests run + # every node in one network namespace, so tc and iptables cannot fault one + # node's traffic there. + - id: staleness-packet-loss + name: Staleness under packet loss + description: 30% loss on the writer's traffic; open, stat and fresh-open reads stay within their staleness bounds + priority: docker_only + environments: + go: SKIPPED + docker: run-tests.sh::test_staleness_packet_loss + vm: TODO + + - id: staleness-duplication-reordering + name: Staleness under duplication and reordering + description: 30% duplicated and 50% reordered writer traffic; reads stay within their staleness bounds + priority: docker_only + environments: + go: SKIPPED + docker: run-tests.sh::test_staleness_duplication_and_reordering + vm: TODO + + - id: staleness-delay + name: Staleness under delay + description: Writer traffic delayed 0 to 2 seconds; reads stay within their staleness bounds + priority: docker_only + environments: + go: SKIPPED + docker: run-tests.sh::test_staleness_delay + vm: TODO + + - id: staleness-one-way-partition + name: Staleness under a one-way partition + description: The reader hears nothing from the writer, which still hears the reader; reads stay within their staleness bounds + priority: docker_only + environments: + go: SKIPPED + docker: run-tests.sh::test_staleness_one_way_partition + vm: TODO + + - id: staleness-multicast-dropped + name: Staleness with multicast dropped + description: The reader drops all multicast while unicast works; reads stay within their staleness bounds + priority: docker_only + environments: + go: SKIPPED + docker: run-tests.sh::test_staleness_multicast_dropped + vm: TODO + + - id: staleness-flood-under-loss + name: Staleness during an invalidation flood under loss + description: A third node floods invalidations while the writer and reader lose 10% of their traffic; reads stay within their staleness bounds + priority: docker_only + environments: + go: SKIPPED + docker: run-tests.sh::test_staleness_flood_under_loss + vm: TODO + - id: rolling-restart name: Rolling restart description: Nodes can be restarted one at a time From 6b5223350ced389aa727c5a20c07f98fad7c5d08 Mon Sep 17 00:00:00 2001 From: Matthew Mattox Date: Sat, 14 Nov 2026 10:17:36 -0600 Subject: [PATCH 2/3] test: satisfy errcheck in the staleness tool and the open-file test Co-Authored-By: Claude Opus 5.5 --- tests/docker/staleness/main.go | 18 ++++++------------ tests/integration/multinode_test.go | 2 +- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/tests/docker/staleness/main.go b/tests/docker/staleness/main.go index 0ffd2d3d..a6255ec5 100644 --- a/tests/docker/staleness/main.go +++ b/tests/docker/staleness/main.go @@ -76,8 +76,6 @@ func write(args []string) { maxGap := fs.Duration("max", 600*time.Millisecond, "longest pause between versions") _ = fs.Parse(args) - out := bufio.NewWriter(os.Stdout) - defer out.Flush() for v := 1; v <= *versions; v++ { f, err := os.OpenFile(*file, os.O_RDWR|os.O_CREATE, 0o644) if err != nil { @@ -92,8 +90,7 @@ func write(args []string) { if err := f.Close(); err != nil { fail("close v%d: %v", v, err) } - fmt.Fprintf(out, "W %d %d\n", time.Now().UnixNano(), v) - out.Flush() + fmt.Printf("W %d %d\n", time.Now().UnixNano(), v) gap := *minGap + time.Duration(rand.Int64N(int64(*maxGap-*minGap)+1)) time.Sleep(gap) } @@ -112,12 +109,9 @@ func read(args []string) { _ = fs.Parse(args) var mu sync.Mutex - out := bufio.NewWriter(os.Stdout) - defer out.Flush() logChange := func(mode string, o observation) { mu.Lock() - fmt.Fprintf(out, "R %d %s %d %d\n", time.Now().UnixNano(), mode, o.size, o.data) - out.Flush() + fmt.Printf("R %d %s %d %d\n", time.Now().UnixNano(), mode, o.size, o.data) mu.Unlock() } watch := func(mode string, look func() observation) { @@ -146,7 +140,7 @@ func read(args []string) { fail("open %s: %v", *file, err) } } - defer held.Close() + defer func() { _ = held.Close() }() looks := map[string]func() observation{ "open": func() observation { @@ -168,7 +162,7 @@ func read(args []string) { if err != nil { return observation{-1, -1} } - defer f.Close() + defer func() { _ = f.Close() }() fi, err := f.Stat() if err != nil { return observation{-1, -1} @@ -207,7 +201,7 @@ func flood(args []string) { if err != nil { fail("open: %v", err) } - defer f.Close() + defer func() { _ = f.Close() }() handles[i] = f } buf := make([]byte, 512) @@ -330,7 +324,7 @@ func scan(path string, each func([]string)) { if err != nil { fail("%v", err) } - defer f.Close() + defer func() { _ = f.Close() }() s := bufio.NewScanner(f) for s.Scan() { if fields := strings.Fields(s.Text()); len(fields) > 0 { diff --git a/tests/integration/multinode_test.go b/tests/integration/multinode_test.go index e6fa1e8c..b0a9c224 100644 --- a/tests/integration/multinode_test.go +++ b/tests/integration/multinode_test.go @@ -1301,7 +1301,7 @@ func TestMultiNodeOpenFileSeesGrowthWithoutBroadcast(t *testing.T) { if err != nil { t.Fatal(err) } - defer f.Close() + defer func() { _ = f.Close() }() buf := make([]byte, 8) if _, err := f.ReadAt(buf, 8<<10-8); err != nil { t.Fatalf("reading the first version: %v", err) From bd987d4dc7d724ddf40561b28421816cfa8ef1a1 Mon Sep 17 00:00:00 2001 From: Matthew Mattox Date: Sat, 14 Nov 2026 10:20:53 -0600 Subject: [PATCH 3/3] test(staleness): fail when a reader never reaches a version A reader frozen at an early version passed: versions it never saw were skipped, and a reader that logged nothing dropped out of the report. Readers watch until well after the writer's last version, so each must reach every version, and every reader log must have looks. Co-Authored-By: Claude Opus 5.5 --- tests/docker/staleness/main.go | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/tests/docker/staleness/main.go b/tests/docker/staleness/main.go index a6255ec5..33fa9604 100644 --- a/tests/docker/staleness/main.go +++ b/tests/docker/staleness/main.go @@ -240,22 +240,28 @@ func analyze(args []string) { maxVersion = max(maxVersion, v) } }) + if maxVersion == 0 { + fail("writer log has no versions") + } + failed := false changes := map[string][]change{} for _, readerLog := range strings.Split(*readerLog, ",") { + looks := 0 scan(readerLog, func(f []string) { if f[0] == "R" && len(f) == 5 { at, _ := strconv.ParseInt(f[1], 10, 64) size, _ := strconv.Atoi(f[3]) data, _ := strconv.Atoi(f[4]) changes[f[2]] = append(changes[f[2]], change{at, observation{size, data}}) + looks++ } }) - } - if maxVersion == 0 { - fail("writer log has no versions") + if looks == 0 { + fmt.Printf("%s has no looks: the reader never opened the file or failed\n", readerLog) + failed = true + } } - failed := false for _, mode := range []string{"reopen", "stat", "open"} { cs, ok := changes[mode] if !ok { @@ -308,11 +314,14 @@ func analyze(args []string) { if regressions > 0 { verdict = "WENT BACKWARDS" } + // Readers watch until well after the writer's last version, so each + // must reach it. + if len(lags) < maxVersion { + verdict = fmt.Sprintf("NEVER SAW %d VERSIONS", maxVersion-len(lags)) + failed = true + } fmt.Printf("%-6s versions seen %d/%d median %v worst %v (v%d) bound %v unreadable looks %d regressions %d %s\n", mode, len(lags), maxVersion, median.Round(time.Millisecond), worst.Round(time.Millisecond), worstVersion, *bounds[mode], bad, regressions, verdict) - if len(lags) < maxVersion-1 { - fmt.Printf("%-6s missed %d versions entirely\n", mode, maxVersion-len(lags)) - } } if failed { os.Exit(1)