Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- Directory operations now update caches and send broadcast invalidations.
- The broadcast peer table never dropped an entry: every remount and every forged heartbeat added one. Peers are now kept by address, at most `-broadcast-max-peers` of them, and so are the nodes whose message numbers a node tracks.
- A cache that kept evicting entries grew in memory past its limit, because a Go map keeps the room of deleted keys: the extent cache took 5.8 MB when full and 10.9 MB after 40 times its size went through it. Each cache shard now rebuilds its map after it has evicted as many entries as it holds.
- A file held open on one node could return short reads for up to 9 seconds after another node grew it, when that node's broadcast messages did not arrive. `stat` read the new size from disk, but reads used the cached inode until `-cache-ttl` passed, and while the other node kept writing the two kept disagreeing. Getting a file's attributes now refreshes the cached inode too, so open files catch up within about twice `-attr-timeout`.
- A read right after `open` could stop early when this node's kernel had cached the file's size during another node's write, for example from a desktop volume monitor or `ls -l` scanning the mount. Opening a file now makes the kernel fetch its size again.
- A node that lost broadcast invalidations, for example when its receive buffer filled during a burst of writes on another node, kept stale inodes until `-cache-ttl` and logged nothing. It now sees the gap in the sender's sequence numbers, drops its inode and extent caches, and logs it at most every 10 seconds. The unmount stats line reports `lost=N`.
- Broadcast deduplication kept every message for 5 minutes: about 224 MB per 2.4 million messages, and cleaning it up held up message handling for up to 200 ms. It now keeps a 64-message window per sender.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ A VMFS-like clustered filesystem written in Go. GoClusterFS enables multiple nod
- **Linux only.** GoClusterFS needs FUSE and O_DIRECT.
- **Every file with data uses at least one full 1 MB block.** The block size is 1 MB, and small-file packing (SFB) is disabled since [#12](https://github.com/SupportTools/GoClusterFS/pull/12), so `-sfb` has no effect.
- **`ls -l` of a large directory is slow.** Each lookup reads two inodes from disk, about 0.3 ms on our test host, and `ls -l` repeats its lookups once it runs longer than `-attr-timeout`. In one test, `ls -l` took 0.14 s on 1,000 files and 14 s on 10,000.
- **Other nodes' changes can show up late.** The kernel caches file attributes and directory entries for `-attr-timeout` (default 1s). For that long, `stat` and `ls` can show another node's file with its old size and mtime. A file that stays open can use the old size and block mappings for up to `-cache-ttl` (default 5s). Opening the file again reads the current data. See [Cache Flags](docs/API-CONFIG.md#cache-flags).
- **Other nodes' changes can show up late.** The kernel caches file attributes and directory entries for `-attr-timeout` (default 1s). For that long, `stat` and `ls` can show another node's file with its old size and mtime. A file that stays open can use the old size and block mappings for about twice `-attr-timeout`. Opening the file again reads the current data. See [Cache Flags](docs/API-CONFIG.md#cache-flags).
- **On-disk format version 3.** Mount refuses images formatted by earlier builds, including version 3 images formatted before [#85](https://github.com/SupportTools/GoClusterFS/pull/85) gave each lock type its own lock records. Copy the data off with the build that formatted the image, reformat, and copy it back.
- **The inode count is set at format time.** `-format` creates one inode per `-bytes-per-inode` bytes of device (default 1 MB, so one per data block), between 4096 and 16,777,216. It cannot change later.
- **Upgrade all nodes at once.** Unmount on every node, upgrade, then mount again. Mixing old and new binaries on one device can serve stale data or make files unreadable. See the upgrade notes in [CHANGELOG.md](CHANGELOG.md).
Expand Down
2 changes: 1 addition & 1 deletion docs/API-CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ These are the `goclusterfs-mount` flags:

Each cache splits its limit evenly over 16 shards, so it holds a multiple of 16 entries: `-cache-negative 1000` holds 992, and a limit from 1 to 15 holds 16. A full cache drops the entry used longest ago.

The kernel caches file attributes and directory entries for `-attr-timeout`. For that long after another node changes a file, `stat` and path lookups on this node can return the old size. Opening a file reads its inode from disk and makes the kernel fetch its size again, so a read after `open` sees the other node's completed writes. Reads through a file that is already open use the inode cache and can use the old size and block mappings for up to `-cache-ttl`, including a block that another node has since freed and reused for other data. With broadcast enabled, writes and directory changes invalidate peers' caches sooner; truncate, fallocate and unlink do not.
The kernel caches file attributes and directory entries for `-attr-timeout`. For that long after another node changes a file, `stat` and path lookups on this node can return the old size. Opening a file reads its inode from disk and makes the kernel fetch its size again, so a read after `open` sees the other node's completed writes. Reads through a file that is already open use the inode cache. Each time the kernel rechecks the file's attributes, every `-attr-timeout` while the file is read, the mount refreshes that cache from disk, so an open file sees another node's completed writes within about twice `-attr-timeout` (2s at the default), even when broadcast messages are lost. Until then it can use the old size and block mappings, including a block that another node has since freed and reused for other data. With broadcast enabled, writes and directory changes invalidate peers' caches sooner; truncate, fallocate and unlink do not.

`-attr-timeout=0` makes `stat` and path lookups always current, but each `stat` and path step is then a lookup that reads the directory's and the file's inodes from disk: about 0.7 ms per `stat` on the test host, against well under 0.1 ms from the kernel cache.

Expand Down
2 changes: 1 addition & 1 deletion docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,7 @@ The flags set how many entries each cache holds and how long cached data may be
| Flag | Default | Effect |
|------|---------|--------|
| `-attr-timeout` | `1s` | How long the kernel caches attributes and directory entries. `stat` and `ls` can show another node's change up to this late. `0` makes every `stat` and path step a lookup that reads the directory's and the file's inodes from disk. |
| `-cache-ttl` | `5s` | How long the mount trusts a cached inode before it checks the disk. Open files can lag another node's change by up to this long. |
| `-cache-ttl` | `5s` | How long the mount trusts a cached inode before it checks the disk. Directory lookups can lag another node's change by up to this long; open files catch up within about twice `-attr-timeout`. |
| `-cache-strict` | `false` | Check the disk on every cache hit |
| `-cache-inodes` | `10000` | Inodes kept in memory, about 480 bytes each |
| `-cache-extents` | `50000` | Block mappings kept in memory, about 120 bytes each |
Expand Down
4 changes: 4 additions & 0 deletions internal/goclusterfs/fs.go
Original file line number Diff line number Diff line change
Expand Up @@ -1690,6 +1690,10 @@ func (f *File) Getattr(ctx context.Context, fh fs.FileHandle, out *fuse.AttrOut)
}

logging.Debugf("File.Getattr: inode=%d Size=%d AllocatedBlocks=%d", f.inodeNum, ino.Size, ino.AllocatedBlocks)
// Reads use the cached inode; they must not stop at an older size than stat shows.
if ino.Magic == inode.InodeMagic {
f.fs.cacheInode(ino)
}

// Zero means go-fuse applies the mount's -attr-timeout.
out.AttrValid = 0
Expand Down
2 changes: 2 additions & 0 deletions tests/docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ COPY . .

# Build the mount binary
RUN CGO_ENABLED=0 GOOS=linux go build -o goclusterfs-mount ./cmd/goclusterfs-mount
RUN CGO_ENABLED=0 GOOS=linux go build -o gcfs-staleness ./tests/docker/staleness

# =============================================================================
# Stage 2: Runtime - Minimal image with FUSE and network tools
Expand All @@ -42,6 +43,7 @@ RUN mkdir -p /mnt/goclusterfs /data /scripts

# Copy binary from builder
COPY --from=builder /build/goclusterfs-mount /usr/local/bin/
COPY --from=builder /build/gcfs-staleness /usr/local/bin/

# Copy container scripts
COPY tests/docker/scripts/ /scripts/
Expand Down
95 changes: 94 additions & 1 deletion tests/docker/scripts/run-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ in_test_group() {
[[ -z "$TEST_GROUP" ]] && return 0
local group=1
case "$1" in
"Container Kill Recovery"|Journal*|"Network Partition"|Broadcast*|"Inode Reuse Stale Read"|"Rolling Restart")
"Container Kill Recovery"|Journal*|"Network Partition"|Broadcast*|Staleness*|"Inode Reuse Stale Read"|"Rolling Restart")
group=2 ;;
# Takes about 14 minutes in CI, so it runs alone.
"More Than 4096 Files")
Expand Down Expand Up @@ -2466,6 +2466,93 @@ test_dir_index_follows_other_node() {
return 0
}

heal_network_faults() {
local node
for node in "${NODE_ARRAY[@]}"; do
exec_on_node "$node" sh -c 'tc qdisc del dev eth0 root 2>/dev/null; iptables -F INPUT; iptables -F OUTPUT' || true
done
}

# measure_staleness has node 1 rewrite a file 20 times, growing and truncating
# it, while node 2 reads it through a file it holds open, node 3 stats it, and
# node 4 opens it afresh for each look (or floods invalidations with "flood").
# Fails if any of them sees a version later than its bound allows, with the
# default -attr-timeout 1s and -cache-ttl 5s.
measure_staleness() {
local name=$1 flood=${2:-}
local writer="${NODE_ARRAY[0]}" held="${NODE_ARRAY[1]}" stat="${NODE_ARRAY[2]}" fresh="${NODE_ARRAY[3]}"
local file
file="/mnt/goclusterfs/staleness_${name}_$(date +%s%N)"
local out="$RESULTS_DIR/staleness_$name"
local pids=() logs="$out.open,$out.stat"

exec_on_node "$held" gcfs-staleness read -file "$file" -duration 20s -modes open > "$out.open" 2>&1 &
pids+=($!)
exec_on_node "$stat" gcfs-staleness read -file "$file" -duration 20s -modes stat > "$out.stat" 2>&1 &
pids+=($!)
if [[ "$flood" == flood ]]; then
exec_on_node "$fresh" gcfs-staleness flood -dir "/mnt/goclusterfs/flood_$name" -duration 12s > "$out.flood" 2>&1 &
else
exec_on_node "$fresh" gcfs-staleness read -file "$file" -duration 20s -modes reopen > "$out.reopen" 2>&1 &
logs="$logs,$out.reopen"
fi
pids+=($!)
sleep 0.5
exec_on_node "$writer" gcfs-staleness write -file "$file" -versions 20 > "$out.w" 2>&1
wait "${pids[@]}"
heal_network_faults

local result=0
gcfs-staleness analyze -writer "$out.w" -reader "$logs" -bound-open 3s -bound-stat 1500ms -bound-reopen 500ms > "$out.result" 2>&1 || result=1
while read -r line; do log " $line"; done < "$out.result"
exec_on_node "$writer" rm -rf "$file" "/mnt/goclusterfs/flood_$name"
return $result
}

apply_fault() {
local node=$1
shift
if ! exec_on_node "$node" "$@"; then
log "ERROR: could not apply network fault on $node: $*"
heal_network_faults
return 1
fi
}

test_staleness_packet_loss() {
apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem loss 30% || return 1
measure_staleness packet_loss
}

test_staleness_duplication_and_reordering() {
apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem delay 50ms reorder 50% 50% duplicate 30% || return 1
measure_staleness reordering
}

test_staleness_delay() {
apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem delay 1000ms 1000ms || return 1
measure_staleness delay
}

test_staleness_one_way_partition() {
local writer_ip
writer_ip=$(exec_on_node "${NODE_ARRAY[0]}" hostname -i | awk '{print $1}')
apply_fault "${NODE_ARRAY[1]}" iptables -I INPUT -s "$writer_ip" -j DROP || return 1
measure_staleness one_way_partition
}

# The reader hears no multicast at all, as behind a switch that drops it.
test_staleness_multicast_dropped() {
apply_fault "${NODE_ARRAY[1]}" iptables -I INPUT -d 239.192.74.75 -j DROP || return 1
measure_staleness multicast_dropped
}

test_staleness_flood_under_loss() {
apply_fault "${NODE_ARRAY[0]}" tc qdisc add dev eth0 root netem loss 10% || return 1
apply_fault "${NODE_ARRAY[1]}" iptables -I INPUT -p udp -m statistic --mode random --probability 0.1 -j DROP || return 1
measure_staleness flood_under_loss flood
}

# Print the free 1MB blocks that statfs reports on node $1
statfs_free_blocks() {
exec_on_node "$1" stat -f -c %f /mnt/goclusterfs
Expand Down Expand Up @@ -3330,6 +3417,12 @@ run_test "Broadcast Invalidation" test_broadcast_invalidation
run_test "Broadcast Directory Invalidation" test_broadcast_directory_invalidation
run_test "Broadcast Multiple Updates" test_broadcast_multiple_updates
run_test "Broadcast Bidirectional" test_broadcast_bidirectional
run_test "Staleness Under Packet Loss" test_staleness_packet_loss
run_test "Staleness Under Duplication and Reordering" test_staleness_duplication_and_reordering
run_test "Staleness Under Delay" test_staleness_delay
run_test "Staleness Under One-Way Partition" test_staleness_one_way_partition
run_test "Staleness With Multicast Dropped" test_staleness_multicast_dropped
run_test "Staleness During Flood Under Loss" test_staleness_flood_under_loss
run_test "Inode Reuse Stale Read" test_inode_reuse_stale_read
run_test "More Than 4096 Files" test_more_than_4096_files
run_test "Extent Cache Coherency" test_extent_cache_coherency
Expand Down
Loading
Loading