Repository navigation
fix: stop TRIM discarding reallocated blocks and dedupe broadcasts per sender - #37
Merged
Merged
Conversation
mattmattox
force-pushed
the
claude/trim-broadcast-safety
branch
from
October 24, 2026 23:23
f7b4b27 to
5c635a3
Compare
bitmap.Free cleared the bit and queued the block for TRIM, and the discard ran up to MaxAge (5s) or 256 frees later. Allocation scans from the lowest free block, so any node could allocate the block and write to it first, and BLKDISCARD then destroyed the new data. Image files reject BLKDISCARD, so no test saw it; on a device that supports discard it loses data. The TRIM manager now discards through a guard the bitmap registers in SetTrimManager. The guard takes the bitmap lock, discards only the blocks still free on disk, and holds the lock until the discard ends. Queue no longer flushes inline: Free holds the bitmap lock, so a full batch wakes the flush loop instead. Duplicate queued blocks (freed, reallocated, freed again) are discarded once. Shared blocks reach Free only at refcount 0, so they are covered by the same check. Pending discards live only in memory: a crash before the flush leaves those blocks free and undiscarded, which costs space on thin storage but no data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receiver dropped any message whose sequence number it had seen in the last 5 minutes, from any sender. Each node's sequence starts at its boot time in nanoseconds and counts up by one, so once one node has sent more messages than the gap between two boot times (1e7 for a 10ms gap), its sequence numbers overlap the other's and real invalidations from the other node are dropped as duplicates. Peers then serve stale data until the cache TTL. Dedup now keys on sender UUID and sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unmarshal accepted datagrams with extra bytes after the payload and HMAC. They were ignored, but they were not authenticated. It now rejects them, so an accepted datagram is authenticated in full. New tests flip every bit position of a signed message, cut it at every length, strip its HMAC, and check that the largest message fits in one datagram on a 1500-byte MTU. Every invalidation carries one inode, so messages are 62 bytes (94 with HMAC) and never need IP fragmentation. FuzzUnmarshal checks that any accepted signed datagram is exactly the bytes Marshal produces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…map lock The guard held the bitmap lock for the whole discard. Nothing renews that lock and peers take it over once it is 15s old, so a discard slower than about 9-15s (thin LVM, SAN UNMAP, zeroing devices) let another node allocate and write a block that was still being discarded. It also stalled every node's allocations meanwhile. The guard now marks the still-free blocks allocated under the lock, releases it, discards, then frees exactly those blocks under the lock. A crash during the discard leaves them allocated and unreferenced, which fsck reports as leaked. SetFreeGuard is now part of TrimQueuer, so a TRIM manager that does not accept the guard no longer compiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d with real locks TestTrimSkipsBlockReallocatedBeforeFlush waited for the queue to empty, which happens before the discard runs, so it passed 13 of 30 runs with the guard removed. It now also frees a higher block and waits for that block's discard. It fails 30 of 30 runs without the guard. TestTrimDiscardHoldsNoBitmapLock uses two real lock managers. While node A discards, the block must read as allocated and node B must still allocate a different block within 5s. It fails with the earlier guard that held the lock during the discard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The flush passed one 30s context to the guard, and the final free of the reserved blocks used it too. After a discard longer than 30s that context was done, so the free got only the lock manager's minimum retries (about 100ms). If another node held the bitmap lock then, the blocks stayed allocated for good; fsck reports them but cannot repair them. The free now ignores the caller's deadline, takes up to 30s per attempt, and retries every second with a WARN until every reserved block is free. Each retry covers only the blocks still reserved, so a block freed in an earlier round and allocated by another node is not touched again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
force-pushed
the
claude/trim-broadcast-safety
branch
from
October 24, 2026 23:38
5c635a3 to
e18d8fe
Compare
releaseReserved retried forever. With a persistent device error the TRIM worker spun and unmount's final flush never returned. It now stops after 5 minutes and logs an ERROR naming the blocks still reserved, which fsck will report as leaked. Lock contention clears well within that time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TRIM, data loss on devices that support discard (fixed).
bitmap.Freecleared the bit and queued the block. The discard ran up to 5s or 256 frees later. Allocation takes the lowest free block, so this node or another one often allocated and wrote the block first, and BLKDISCARD then destroyed the new data. Image files reject BLKDISCARD, so no test noticed. Now the flush marks the still-free blocks as allocated under the bitmap lock, releases the lock, discards them, then frees them again under the lock. No lock is held during the discard, so a slow discard can't outlast the lock's 15s stale threshold. Freeing the reservation ignores the flush deadline and retries until it succeeds, so a slow discard can't leak the blocks. A crash mid-discard leaves them leaked, which fsck reports. A full batch wakes the flush loop instead of flushing insideFree, which holds the bitmap lock. Tests run 30× on .68: with the guard disabled, the reallocation tests fail 30/30. With the earlier hold-the-lock guard, the two-node real-lock test fails.Checked, no change needed: shared blocks reach
Freeonly at refcount 0. The discard queue is in memory only, so a crash leaves blocks undiscarded, which wastes space but loses no data.Broadcast (fixed):
Unmarshalaccepted trailing bytes outside the HMAC. Now rejected.Checked, no change needed: every message carries one inode (94 bytes with HMAC, max 1200), so there are no batches and no IP fragmentation. Dropped or truncated messages fall back to the 5s TTL. New tests flip every bit, cut at every length, strip the HMAC, and fuzz
Unmarshal(21M runs clean).Not changed here:
cmd/mountstops the node registry before the final TRIM flush. The flush now holds the bitmap lock only for two brief bit flips, so that window is milliseconds. VM tests still issue no real discards: the shared drive lacksdiscard=unmap, and the vm-harness branches rewrite that file.Known narrow window: while blocks are reserved, cross-node journal replay of a stale BlockFree (
applyBlockFree) could free one mid-discard, and another node could then allocate it before the discard ends. That needs a replay and a discard to overlap, and stale BlockFree replay is already risky without TRIM.Not fixed: signed messages can be replayed after the 5-minute dedup window, because the timestamp is never checked. That allows forced cache clears and stale peer entries; a fix needs clock-skew tolerance.
On .68: all unit tests pass with
-race, and the integration suite passes with-short.🤖 Generated with Claude Code