Skip to content

feat(fs): reads never return part of another node's write - #188

Open
mattmattox wants to merge 5 commits into
mainfrom
claude/atomic-reads
Open

mattmattox wants to merge 5 commits into
mainfrom
claude/atomic-reads

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Reads no longer return part of another node's write. Each FUSE read takes a per-file cluster lock shared; each write, truncate and fallocate takes it exclusive; snapshots and clones take it shared. The lock covers the whole request (GFS2/OCFS2 model).

Guarantee. Per request, and a request carries at most 1 MB (MaxWrite and max_read raised from 128 KB, readahead to 1 MB). A write() over 1 MB is several requests. A read through the page cache can still combine pages cached before another node's write with pages fetched after it, until the next open or attribute refresh (-attr-timeout). O_DIRECT reads, or the new -direct-io mount option, close that gap. Same-node page-cache tears (#174) are unchanged. Readers wait behind a writer's mark for up to 500 ms, so neither side starves.

Needs a reformat. File I/O locks take 39 lock rows (32 from extents, 7 from SFB), so new volumes carry required flag 0x40 (unknown required flags 0x40 on older builds) and mount and fsck refuse older volumes. Each registry slot records its lock layout; mount refuses to join while a live node shows another, naming it.

Measured on .68 (load ~60, ZFS image, two runs each, against main 48d1e51, before the rebase). One-node and reader/writer rows come from the second commit (315bdeb), which differs from the final code only in the row split; one-file and probe rows from the third (63ae45f).

One node main branch -direct-io
read 1 MB seq, O_DIRECT 214–221 MB/s 271–317 252–372
read 1 MB seq, buffered 191–194 MB/s 170–189 223–233
read 64 KB random, O_DIRECT 2.7–2.8k/s 2.4–2.6k 2.5k
read 4 KB random, O_DIRECT 11.1–12.3k/s 9.0–9.9k 10.3–10.8k
read 4 KB, lock gone cold (20 ms gaps), p50 0.2 ms 1.1–5.2 ms 1.5–1.8 ms
write 1 MB seq 80 MB/s 181 209–217
write 64 KB / 4 KB random 552–584 / 672–756 /s 697–714 / 879–901 800–845 / 980–1,038

Write amplification is unchanged: device bytes written per user byte 1.04 → 1.01 at 1 MB, 1.18 at 64 KB, 3.8 at 4 KB.

Several nodes main branch
4 nodes writing one file, 1 MB random, total 81–95 MB/s 182–186 MB/s
same, 4 separate files 243–255 MB/s 440–468 MB/s
64 KB random, one file vs separate, 4 nodes 41–42 vs 60–64 MB/s 42–45 vs 70–73 MB/s
per reader (64 KB O_DIRECT), 1–4 nodes reading while node 0 writes 1 MB 2,950–3,980/s 1,430–1,990/s
same, no writer 2,790–3,980/s 2,840–3,800/s
writer p99 with those readers 83–89 ms 290–306 ms
probe, 4 nodes: writes / deletes of 1 MB files 191–195 / 375–382 /s 304–312 / 429–451 /s

Writes from several nodes to one file now run one at a time: total throughput stays at one node's rate (~185 MB/s) while separate files scale to ~450 MB/s. Readers and a writer of one file take 250 ms turns, hence the reader drop and the writer p99.

Profiling the read path found two lock-manager costs, fixed here: finding the goroutine ID formats the whole stack (~55 µs per read; file I/O locks now use a per-request owner), and a holder gave up every record each 250 ms turn even when nobody waited (it now checks the row and keeps it). Other lock types still pay the stack walk; follow-up.

Tests. 3 FUSE nodes, one writing 1 MB values, all reading whole with O_DIRECT for 8 s: main 4,218 of 5,658 reads torn, this branch 0 (also 0 with -direct-io). In-process tests for a write paused between blocks, a truncate, and a stale cached inode, each failing with its piece removed. Writer-preference test: with it off, the writer never gets the lock in 10 s. Docker and VM run the same tear loop with dd iflag=direct.

Evidence on .68 with this branch rebased on c009926 (a rerun on the current base b5f69e3, after rename and symlinks landed, is in progress): unit 30/30 packages; -race ×3 on lock, journal, inode, dir and once on goclusterfs, node, superblock, fsck: no races; lint 0 issues; integration passes except TestMultiNodeRestartKeepsPeerJournal, the known flake (interleaved: main 2/7, this branch 2/7, same ENOENT on the crashed node's last file); history checker, 3 nodes × 3 min: 69,365 calls, 0 violations; VM suite (2 nodes) 75 passed, 0 failed, and the VM tear test mixed 0 of 165 reads over two runs (main fails it: 97 of 207). Shared mmap on a -direct-io mount fails with ENODEV, private works (checked on a mount).

🤖 Generated with Claude Code

mattmattox and others added 5 commits November 16, 2026 21:07
A read request took no lock, so it could return part of another node's
write, or of a truncate. Each read, write, truncate and fallocate now
takes a per-file I/O lock for the whole request: shared for reads,
exclusive for changes, with a local stripe for the node's own requests.
Snapshots and clones take it shared. A reader waits behind a writer's
waiting mark for up to two turns, so readers on lower slots cannot shut
a writer out. The first read after each claim reads the inode again.

File I/O locks get 39 of the 256 lock records, so the record layout
changes: RequiredFileIOLocks marks new volumes, mount refuses older
ones, and each registry slot records its lock layout, which mount
checks against live nodes. FUSE requests now carry up to 1 MB, and
-direct-io bypasses the page cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…file I/O locks

Reads now take a lock per request, which showed two costs. Finding the
goroutine ID formats the caller's whole stack, about 55 us per read on
FUSE stacks: file I/O locks now go through AcquireAs and ReleaseAs with a
per-request owner. And a holder gave up every record at the end of each
250 ms turn and waited 50 ms even when no other node wanted it: it now
reads the row when the turn ends and starts a new turn if the row shows
no conflicting claim and no live mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With 96 inode rows, unrelated inodes shared records more often: in the
4-node probe, deletes ran 5 to 13% slower than main and inode lock waits
rose from 125% to about 150% of node time. Whole-block writes take no
extent locks now that requests carry 1 MB, so extents give up 32 rows and
inodes keep 128. Deletes then ran 429 to 451 files/s at 4 nodes (main
375 to 382), and overwrites of separate files from 4 nodes stayed within
noise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…junction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant