Skip to content

fix: stop TRIM discarding reallocated blocks and dedupe broadcasts per sender - #37

Merged
mattmattox merged 9 commits into
mainfrom
claude/trim-broadcast-safety
Oct 24, 2026
Merged

mattmattox merged 9 commits into
mainfrom
claude/trim-broadcast-safety

Conversation

@mattmattox

@mattmattox mattmattox commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

TRIM, data loss on devices that support discard (fixed). bitmap.Free cleared the bit and queued the block. The discard ran up to 5s or 256 frees later. Allocation takes the lowest free block, so this node or another one often allocated and wrote the block first, and BLKDISCARD then destroyed the new data. Image files reject BLKDISCARD, so no test noticed. Now the flush marks the still-free blocks as allocated under the bitmap lock, releases the lock, discards them, then frees them again under the lock. No lock is held during the discard, so a slow discard can't outlast the lock's 15s stale threshold. Freeing the reservation ignores the flush deadline and retries until it succeeds, so a slow discard can't leak the blocks. A crash mid-discard leaves them leaked, which fsck reports. A full batch wakes the flush loop instead of flushing inside Free, which holds the bitmap lock. Tests run 30× on .68: with the guard disabled, the reallocation tests fail 30/30. With the earlier hold-the-lock guard, the two-node real-lock test fails.

Checked, no change needed: shared blocks reach Free only at refcount 0. The discard queue is in memory only, so a crash leaves blocks undiscarded, which wastes space but loses no data.

Broadcast (fixed):

  • Dedup was keyed on sequence alone across all senders. Sequences start at boot time in ns, so a busy node's range can overlap another's and drop its invalidations. Now keyed on sender + sequence.
  • Unmarshal accepted trailing bytes outside the HMAC. Now rejected.

Checked, no change needed: every message carries one inode (94 bytes with HMAC, max 1200), so there are no batches and no IP fragmentation. Dropped or truncated messages fall back to the 5s TTL. New tests flip every bit, cut at every length, strip the HMAC, and fuzz Unmarshal (21M runs clean).

Not changed here: cmd/mount stops the node registry before the final TRIM flush. The flush now holds the bitmap lock only for two brief bit flips, so that window is milliseconds. VM tests still issue no real discards: the shared drive lacks discard=unmap, and the vm-harness branches rewrite that file.

Known narrow window: while blocks are reserved, cross-node journal replay of a stale BlockFree (applyBlockFree) could free one mid-discard, and another node could then allocate it before the discard ends. That needs a replay and a discard to overlap, and stale BlockFree replay is already risky without TRIM.

Not fixed: signed messages can be replayed after the 5-minute dedup window, because the timestamp is never checked. That allows forced cache clears and stale peer entries; a fix needs clock-skew tolerance.

On .68: all unit tests pass with -race, and the integration suite passes with -short.

🤖 Generated with Claude Code

@mattmattox
mattmattox force-pushed the claude/trim-broadcast-safety branch from f7b4b27 to 5c635a3 Compare October 24, 2026 23:23
mattmattox and others added 8 commits October 24, 2026 18:31
bitmap.Free cleared the bit and queued the block for TRIM, and the
discard ran up to MaxAge (5s) or 256 frees later. Allocation scans from
the lowest free block, so any node could allocate the block and write
to it first, and BLKDISCARD then destroyed the new data. Image files
reject BLKDISCARD, so no test saw it; on a device that supports
discard it loses data.

The TRIM manager now discards through a guard the bitmap registers in
SetTrimManager. The guard takes the bitmap lock, discards only the
blocks still free on disk, and holds the lock until the discard ends.
Queue no longer flushes inline: Free holds the bitmap lock, so a full
batch wakes the flush loop instead. Duplicate queued blocks (freed,
reallocated, freed again) are discarded once.

Shared blocks reach Free only at refcount 0, so they are covered by
the same check. Pending discards live only in memory: a crash before
the flush leaves those blocks free and undiscarded, which costs space
on thin storage but no data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receiver dropped any message whose sequence number it had seen in
the last 5 minutes, from any sender. Each node's sequence starts at its
boot time in nanoseconds and counts up by one, so once one node has sent
more messages than the gap between two boot times (1e7 for a 10ms gap),
its sequence numbers overlap the other's and real invalidations from
the other node are dropped as duplicates. Peers then serve stale data
until the cache TTL.

Dedup now keys on sender UUID and sequence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unmarshal accepted datagrams with extra bytes after the payload and
HMAC. They were ignored, but they were not authenticated. It now
rejects them, so an accepted datagram is authenticated in full.

New tests flip every bit position of a signed message, cut it at every
length, strip its HMAC, and check that the largest message fits in one
datagram on a 1500-byte MTU. Every invalidation carries one inode, so
messages are 62 bytes (94 with HMAC) and never need IP fragmentation.
FuzzUnmarshal checks that any accepted signed datagram is exactly the
bytes Marshal produces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…map lock

The guard held the bitmap lock for the whole discard. Nothing renews
that lock and peers take it over once it is 15s old, so a discard
slower than about 9-15s (thin LVM, SAN UNMAP, zeroing devices) let
another node allocate and write a block that was still being
discarded. It also stalled every node's allocations meanwhile.

The guard now marks the still-free blocks allocated under the lock,
releases it, discards, then frees exactly those blocks under the lock.
A crash during the discard leaves them allocated and unreferenced,
which fsck reports as leaked. SetFreeGuard is now part of TrimQueuer,
so a TRIM manager that does not accept the guard no longer compiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d with real locks

TestTrimSkipsBlockReallocatedBeforeFlush waited for the queue to empty,
which happens before the discard runs, so it passed 13 of 30 runs with
the guard removed. It now also frees a higher block and waits for that
block's discard. It fails 30 of 30 runs without the guard.

TestTrimDiscardHoldsNoBitmapLock uses two real lock managers. While
node A discards, the block must read as allocated and node B must still
allocate a different block within 5s. It fails with the earlier guard
that held the lock during the discard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The flush passed one 30s context to the guard, and the final free of
the reserved blocks used it too. After a discard longer than 30s that
context was done, so the free got only the lock manager's minimum
retries (about 100ms). If another node held the bitmap lock then, the
blocks stayed allocated for good; fsck reports them but cannot repair
them.

The free now ignores the caller's deadline, takes up to 30s per
attempt, and retries every second with a WARN until every reserved
block is free. Each retry covers only the blocks still reserved, so a
block freed in an earlier round and allocated by another node is not
touched again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox force-pushed the claude/trim-broadcast-safety branch from 5c635a3 to e18d8fe Compare October 24, 2026 23:38
releaseReserved retried forever. With a persistent device error the
TRIM worker spun and unmount's final flush never returned. It now stops
after 5 minutes and logs an ERROR naming the blocks still reserved,
which fsck will report as leaked. Lock contention clears well within
that time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 0158c8c into main Oct 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant