Repository navigation
feat(fs): reads never return part of another node's write - #188
Open
mattmattox wants to merge 5 commits into
Open
mattmattox wants to merge 5 commits into
mattmattox wants to merge 5 commits into
Conversation
A read request took no lock, so it could return part of another node's write, or of a truncate. Each read, write, truncate and fallocate now takes a per-file I/O lock for the whole request: shared for reads, exclusive for changes, with a local stripe for the node's own requests. Snapshots and clones take it shared. A reader waits behind a writer's waiting mark for up to two turns, so readers on lower slots cannot shut a writer out. The first read after each claim reads the inode again. File I/O locks get 39 of the 256 lock records, so the record layout changes: RequiredFileIOLocks marks new volumes, mount refuses older ones, and each registry slot records its lock layout, which mount checks against live nodes. FUSE requests now carry up to 1 MB, and -direct-io bypasses the page cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…file I/O locks Reads now take a lock per request, which showed two costs. Finding the goroutine ID formats the caller's whole stack, about 55 us per read on FUSE stacks: file I/O locks now go through AcquireAs and ReleaseAs with a per-request owner. And a holder gave up every record at the end of each 250 ms turn and waited 50 ms even when no other node wanted it: it now reads the row when the turn ends and starts a new turn if the row shows no conflicting claim and no live mark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With 96 inode rows, unrelated inodes shared records more often: in the 4-node probe, deletes ran 5 to 13% slower than main and inode lock waits rose from 125% to about 150% of node time. Whole-block writes take no extent locks now that requests carry 1 MB, so extents give up 32 rows and inodes keep 128. Deletes then ran 429 to 451 files/s at 4 nodes (main 375 to 382), and overwrites of separate files from 4 nodes stayed within noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…junction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reads no longer return part of another node's write. Each FUSE read takes a per-file cluster lock shared; each write, truncate and fallocate takes it exclusive; snapshots and clones take it shared. The lock covers the whole request (GFS2/OCFS2 model).
Guarantee. Per request, and a request carries at most 1 MB (MaxWrite and max_read raised from 128 KB, readahead to 1 MB). A
write()over 1 MB is several requests. A read through the page cache can still combine pages cached before another node's write with pages fetched after it, until the next open or attribute refresh (-attr-timeout).O_DIRECTreads, or the new-direct-iomount option, close that gap. Same-node page-cache tears (#174) are unchanged. Readers wait behind a writer's mark for up to 500 ms, so neither side starves.Needs a reformat. File I/O locks take 39 lock rows (32 from extents, 7 from SFB), so new volumes carry required flag
0x40(unknown required flags 0x40on older builds) and mount and fsck refuse older volumes. Each registry slot records its lock layout; mount refuses to join while a live node shows another, naming it.Measured on .68 (load ~60, ZFS image, two runs each, against main 48d1e51, before the rebase). One-node and reader/writer rows come from the second commit (315bdeb), which differs from the final code only in the row split; one-file and probe rows from the third (63ae45f).
-direct-ioWrite amplification is unchanged: device bytes written per user byte 1.04 → 1.01 at 1 MB, 1.18 at 64 KB, 3.8 at 4 KB.
Writes from several nodes to one file now run one at a time: total throughput stays at one node's rate (~185 MB/s) while separate files scale to ~450 MB/s. Readers and a writer of one file take 250 ms turns, hence the reader drop and the writer p99.
Profiling the read path found two lock-manager costs, fixed here: finding the goroutine ID formats the whole stack (~55 µs per read; file I/O locks now use a per-request owner), and a holder gave up every record each 250 ms turn even when nobody waited (it now checks the row and keeps it). Other lock types still pay the stack walk; follow-up.
Tests. 3 FUSE nodes, one writing 1 MB values, all reading whole with O_DIRECT for 8 s: main 4,218 of 5,658 reads torn, this branch 0 (also 0 with
-direct-io). In-process tests for a write paused between blocks, a truncate, and a stale cached inode, each failing with its piece removed. Writer-preference test: with it off, the writer never gets the lock in 10 s. Docker and VM run the same tear loop withdd iflag=direct.Evidence on .68 with this branch rebased on c009926 (a rerun on the current base b5f69e3, after rename and symlinks landed, is in progress): unit 30/30 packages;
-race×3 on lock, journal, inode, dir and once on goclusterfs, node, superblock, fsck: no races; lint 0 issues; integration passes exceptTestMultiNodeRestartKeepsPeerJournal, the known flake (interleaved: main 2/7, this branch 2/7, same ENOENT on the crashed node's last file); history checker, 3 nodes × 3 min: 69,365 calls, 0 violations; VM suite (2 nodes) 75 passed, 0 failed, and the VM tear test mixed 0 of 165 reads over two runs (main fails it: 97 of 207). Sharedmmapon a-direct-iomount fails with ENODEV, private works (checked on a mount).🤖 Generated with Claude Code