Repository navigation
test: fuzz every on-disk decoder, journal replay and API writes - #153
Merged
Merged
Conversation
DeserializeBlockRuns expanded every run a record claimed, so a 124-byte BlockReserve or BlockUse record with a run of 4 billion blocks made replay allocate 32 GB and the mount run out of memory. Records now hold at most MaxRecordBlocks (1<<20) blocks, and the block reserve size is clamped to the same limit so no writer produces a record replay refuses. FuzzTxnRecord decodes block runs and keeps the crasher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deserializeSlot accepted any slot whose stored checksum was 0, so a slot torn before its last bytes landed, or zeroed there, decoded as valid with whatever fields it held. Only an all-zero slot, one no node wrote, skips the checksum now. FuzzRegistrySlot keeps the crasher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parseEntry checked a name against the buffer, not the entry's record, so a damaged entry read its name out of the next record. fsck already treats that as corruption; walkChunk now stops there too. FuzzDirChunk keeps the crasher and checks that chunkRoom agrees with what addToChunk can place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FuzzLockRow builds rows of valid, torn and checksum-passing garbage claims and checks what conflicts with a claim, with and without an owner view, and that AdoptOwner and ClearForeignClaims write only the slot they may. FuzzRefcountSector runs refcount operations over a fuzzed sector against a model. FuzzParseTreeNode also decodes extents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
freeSFBSlot trusted the container header's SlotCount, so a header that passed its checksum with more than 256 slots let a slot past the 32-byte bitmap index out of range, and unlinking such a file crashed the mount. A slot past the bitmap now leaks with the existing warning, like any other bad header. FuzzSFBContainer frees and reads slots of fuzzed containers; its seeds include the crasher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FuzzReplay writes journals of well-formed records with the real journal, completes or tears some, replays them over a small filesystem and runs fsck: replay must finish, and every error must name an inode or block the records touched or one those map. FuzzWriteRequests checks that a write reaches the service only with the token, the same origin and one strict JSON object. FuzzFileOpsPaths checks the path, name and destination checks and that an accepted path opens exactly the file it names under the mount root. Manifest entries for every new target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
currentRecord compared the record length, rounded up to a sector, with the bytes left in the journal. A damaged record claiming nearly 4 GB rounded up past the uint32 range to 0, passed the check, and panicked the recovery scan slicing a zero-length buffer. The length is now checked before rounding. FuzzJournalScan keeps the crasher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A superblock whose journal region does not split into MaxNodes journals only gets a Critical problem in phase 1, so the later phases still run. reserves() then read a journal for each of MaxNodes slots, and a damaged count of 419 million kept fsck reading the same sector for hours. It now stops at the 64 slots the node registry has. FuzzCheck keeps the crasher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every input under 32 KB that the fuzzers on .66 found during the two-hour runs, 2449 files and 5.3 MB. Plain go test runs them as seeds in about 10 seconds across all packages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 14, 2026
The registry fuzz test from #153 called failed(), which this branch replaces. It now checks that a slot seen for the first time is never stale, that an active slot's heartbeat age agrees with LeaseExpiry by this host's clock, and that a dead slot's lease ends recoveryLease after it started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox
added a commit
that referenced
this pull request
Nov 14, 2026
…ans off the heartbeat (#151) * fix(node): judge peers on this node's own clock, and keep registry scans off the heartbeat Nodes judged each other's heartbeats and recovery leases by wall clock: - a node mounting with its clock more than 10s ahead took a live node's registry slot, and that node stopped; - a crashed node whose clock ran ahead was replayed only once its lead ran out, so lock waits on its records failed with EIO meanwhile; - a replay whose node's clock was behind looked abandoned. A node now records each slot it reads and counts the slot's node as failing, dead or abandoned only once it has read the same heartbeat or lease for long enough by its own monotonic clock. A replay renews its lease on its own clock, to a value that always changes. Register takes an empty slot first and takes another host's slot only after watching its heartbeat not change. Under slow storage a node was declared dead while only slow: registry scans held the heartbeat's lock across 64 reads, so at 1s per I/O a peer replayed the node. Scans now read the registry in one I/O without that lock, and registry I/O skips the per-region shard lock, as lock slots already do. storage.Device gains SetReadFault, a read-side twin of SetWriteGate for tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(node): watch leases lapse in fsck's release tests, and renew test leases as replays do fsck's release tests backdated a lease or heartbeat on disk and expected a peer to act on it at once; peers now act only on what they watch stay unchanged. The crash test lets a 1s lease lapse for real through node.SetRecoveryLease, and the late mount comes back as the same host, the one way it still takes a stale slot while another is empty. The test helper that renews a lease now always changes it, as a replay's renewal does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(node): let a mount take another host's silent slot, after watching it Taking empty slots first removed the way a mount on another host takes a crashed node's slot and replays its journal, and with it -skip-recovery for that journal. A mount now again takes another host's slot whose heartbeat looks old by its clock, but only after watching the heartbeat stay unchanged for HeartbeatTimeout plus two beats; a live node whose clock is behind beats within that time and keeps its slot. Integration mounts may now take up to 30s to become ready. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(node): claim a watched slot only if it still holds what was watched, and take empty slots first Register claimed a slot it had watched stay silent with a compare-and-swap against a fresh read, so a slot a peer marked dead to replay it, or whose node beat again, in between was taken anyway. It now swaps against the bytes it watched, so any change makes the claim fail. A mount takes an empty slot first, so it never waits while one is free. Another host's silent slot is taken only when -skip-recovery names it, before any other slot, or no slot is empty. A named slot under another node's recovery is taken once its lease stays unrenewed for the lease term. Failed replays now name -skip-recovery in their log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: notice a mount that exited while waiting for it WaitForMount saw an exit only through ProcessState, which stays nil until someone calls Wait, so a mount meant to fail always waited the full timeout. It now also checks whether the process is a zombie. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(node): end the same-host namespace watch as soon as a slot's heartbeat is old A restarted container keeps its hostname but gets a new PID namespace, so its mount watched its old slot. The watch always ran its full 12s; it now ends, as before, once the slot's heartbeat is older than HeartbeatTimeout by this host's clock, which wrote it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(node): a rebooted host takes back the slot it held, after watching it After a crash and reboot a host has a new boot ID, so its mount no longer counted its old slot as its own and took an empty one, leaving the old slot and its dirty journal for peers. Until a peer replayed it, the old node's locks blocked the new mount, and an fsck run in between refused to repair. A mount now watches the slots its hostname held in an earlier boot, whose heartbeat looks old, before taking an empty one, and takes back one that stays unchanged, replaying its journal before serving. Another host with the same name beats during the watch and keeps its slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(node): a live host that shares the hostname keeps its slot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(node): judge fuzzed slots by the new liveness rules The registry fuzz test from #153 called failed(), which this branch replaces. It now checks that a slot seen for the first time is never stale, that an active slot's heartbeat age agrees with LeaseExpiry by this host's clock, and that a dead slot's lease ends recoveryLease after it started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds fuzz targets for the decoders that had none: lock rows, registry slots, refcount sectors, directory chunks and SFB containers. Also adds journal replay checked by fsck, the API write endpoints and fileops paths.
All 18 targets ran for 2h on .66, each capped at 8G. The 6 crashers are fixed here, one commit each, and each crasher stays in the corpus. None needed a design change.
FuzzJournalScan and FuzzCheck stopped early on their crashers. Their 2h reruns end around 13:45 CST; I'll push any new corpus entries then.
Not covered yet: the broadcast message decoder.
🤖 Generated with Claude Code