Skip to content

fix: never let a read show bytes nobody wrote to the file - #145

Merged
mattmattox merged 4 commits into
mainfrom
fix/stale-bytes
Nov 16, 2026
Merged

mattmattox merged 4 commits into
mainfrom
fix/stale-bytes

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The multi-node soak found this after 1.5 minutes. On main, reads can return a deleted file's data, or a file's own data from before a truncate.

Cause 1, the worse one. A partial write into a newly allocated block writes only its own range. The rest of the block keeps whatever the disk held, often a freed file's data, and reads return it as soon as the file covers that range. A single sparse write into a new file is enough.

Cause 2. A truncate into the middle of a block leaves the old bytes after the cut. Growing the file later, by truncate up or a sparse write, shows them.

Fixes.

  • Every allocation path in the write zeroes a new block before a partial write into it. That includes blocks from the node's reserve (feat(alloc): hand writes data blocks from a per-node reserve #142).
  • Truncate zeroes the rest of the last block before it shrinks the file. It writes through Write, so a shared block is copied first.
  • With both, every byte past EOF in a mapped block is zero, so growing a file can only show zeros.
  • Fallocate was already correct, and the tests confirm it.

Tests. stale_bytes_test.go writes a marker file A, deletes it, and checks that file B never shows A's bytes or its own truncated bytes.

  • It covers: sparse write, growing within a new block, truncate down then up, truncate down then a sparse write, fallocate, copy-on-write of a partial block, clone, clone of a snapshot of a truncated file, and the reserve.
  • Each case asserts that B really reuses A's blocks.
  • Two power-cut tests cut power at every write: of the truncate, and of a first write into a reused block, with and without the reserve.
  • On main's fs.go, 8 of the 9 non-crash cases fail, 6 of them by showing A's data, and both power-cut tests fail.

Crash behaviour. If a truncate is cut between zeroing and shrinking, the file keeps its old size, with zeros or its own bytes in the cut range. It never shows anything else.

Cost. The first partial write into a block now also writes up to 1 MB of zeros.

  • 60 small-file writes took about 1.8 times as long on tmpfs.
  • go test -race ./internal/goclusterfs on .66 went from 351–520 s to 819 s. All 26 packages pass.
  • A follow-up could track how much of each block is written, or zero with BLKZEROOUT / FALLOC_FL_ZERO_RANGE, to avoid moving the zeros.

Images written by older builds may already hold stale bytes past EOF. This fix doesn't scrub them.

🤖 Generated with Claude Code

mattmattox and others added 3 commits November 14, 2026 09:26
A partial write into a newly allocated block wrote only its own range, so
the rest of the block kept what the disk held, often a deleted file's
data, and reads showed it once the file covered that range. Writes now
zero a new block before a partial write into it, on every allocation
path, including blocks from the node's reserve.

A truncate into the middle of a block left the old bytes after the cut,
and growing the file later showed them. Truncate now zeroes them, through
the write path so a shared block is copied first, before it shrinks.

Found by the multi-node soak.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… SFB move

zeroPastEnd read a cached inode and failed a truncate with EIO when the
extent tree changed under it. It now reads fresh, retries on a stale tree,
and hands setattr the generation it decided on; setattr starts over if the
inode changed since. That also covers a file older versions kept in an SFB
slot: setattr moves it to a block of its own first, and the retry zeroes
past the cut in that block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The generation check made a truncate start over whenever a local write
landed between the zeroing and the size change, and after 5 tries it
returned EAGAIN. Setattr now holds the inode lock from the zeroing
through the size change, and a truncate with nothing to zero, such as to
0 or to a block boundary, skips the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit d7f5420 into main Nov 16, 2026
17 checks passed
mattmattox added a commit that referenced this pull request Nov 16, 2026
* perf: zero only what a read could see, not every new block

#145 zeroed every new block before a partial write. FUSE hands writes over
in 128 KiB pieces, so that doubled the bytes written for all new data.

Bytes past the end of a file cannot be read, so they may stay as the disk
held them. A first write into a block now zeroes only the parts below the
end of the file that it does not cover. A write, truncate or fallocate
that grows the file zeroes the rest of its old last block first, through
the write path, so a block a snapshot or clone shares is copied. Truncate
no longer zeroes on shrink.

A growing write holds the inode lock exclusively from its first data write
through its size update, and if it drops the lock mid-write it first
covers what it wrote in the old last block, so another node's growth never
zeroes data it has not yet sized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: retry a growing write when a peer resized the file after it planned the gap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 16, 2026
The 48h soak on daa33a6 found another file's block with its first
176128 bytes zeroed on disk after a write into a hole on a different
node. That range matches the head zero of the hole write, which points
at #160's narrower zeroing. #145's full-block zeroing ran clean for
26,792 soak ops. Restore it until #160's cause is found.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant