Skip to content

test: fuzz every on-disk decoder, journal replay and API writes - #153

Merged
mattmattox merged 11 commits into
mainfrom
test-fuzz-hardening
Nov 14, 2026
Merged

mattmattox merged 11 commits into
mainfrom
test-fuzz-hardening

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

Adds fuzz targets for the decoders that had none: lock rows, registry slots, refcount sectors, directory chunks and SFB containers. Also adds journal replay checked by fsck, the API write endpoints and fileops paths.

All 18 targets ran for 2h on .66, each capped at 8G. The 6 crashers are fixed here, one commit each, and each crasher stays in the corpus. None needed a design change.

FuzzJournalScan and FuzzCheck stopped early on their crashers. Their 2h reruns end around 13:45 CST; I'll push any new corpus entries then.

Not covered yet: the broadcast message decoder.

🤖 Generated with Claude Code

mattmattox and others added 11 commits November 14, 2026 11:56
DeserializeBlockRuns expanded every run a record claimed, so a 124-byte
BlockReserve or BlockUse record with a run of 4 billion blocks made
replay allocate 32 GB and the mount run out of memory. Records now hold
at most MaxRecordBlocks (1<<20) blocks, and the block reserve size is
clamped to the same limit so no writer produces a record replay
refuses. FuzzTxnRecord decodes block runs and keeps the crasher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deserializeSlot accepted any slot whose stored checksum was 0, so a slot
torn before its last bytes landed, or zeroed there, decoded as valid
with whatever fields it held. Only an all-zero slot, one no node wrote,
skips the checksum now. FuzzRegistrySlot keeps the crasher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parseEntry checked a name against the buffer, not the entry's record,
so a damaged entry read its name out of the next record. fsck already
treats that as corruption; walkChunk now stops there too.
FuzzDirChunk keeps the crasher and checks that chunkRoom agrees with
what addToChunk can place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FuzzLockRow builds rows of valid, torn and checksum-passing garbage
claims and checks what conflicts with a claim, with and without an
owner view, and that AdoptOwner and ClearForeignClaims write only the
slot they may. FuzzRefcountSector runs refcount operations over a
fuzzed sector against a model. FuzzParseTreeNode also decodes extents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
freeSFBSlot trusted the container header's SlotCount, so a header that
passed its checksum with more than 256 slots let a slot past the 32-byte
bitmap index out of range, and unlinking such a file crashed the mount.
A slot past the bitmap now leaks with the existing warning, like any
other bad header. FuzzSFBContainer frees and reads slots of fuzzed
containers; its seeds include the crasher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FuzzReplay writes journals of well-formed records with the real journal,
completes or tears some, replays them over a small filesystem and runs
fsck: replay must finish, and every error must name an inode or block
the records touched or one those map. FuzzWriteRequests checks that a
write reaches the service only with the token, the same origin and one
strict JSON object. FuzzFileOpsPaths checks the path, name and
destination checks and that an accepted path opens exactly the file it
names under the mount root. Manifest entries for every new target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
currentRecord compared the record length, rounded up to a sector, with
the bytes left in the journal. A damaged record claiming nearly 4 GB
rounded up past the uint32 range to 0, passed the check, and panicked
the recovery scan slicing a zero-length buffer. The length is now
checked before rounding. FuzzJournalScan keeps the crasher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A superblock whose journal region does not split into MaxNodes journals
only gets a Critical problem in phase 1, so the later phases still run.
reserves() then read a journal for each of MaxNodes slots, and a
damaged count of 419 million kept fsck reading the same sector for
hours. It now stops at the 64 slots the node registry has. FuzzCheck
keeps the crasher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every input under 32 KB that the fuzzers on .66 found during the
two-hour runs, 2449 files and 5.3 MB. Plain go test runs them as seeds
in about 10 seconds across all packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 4de255a into main Nov 14, 2026
17 checks passed
mattmattox added a commit that referenced this pull request Nov 14, 2026
The registry fuzz test from #153 called failed(), which this branch
replaces. It now checks that a slot seen for the first time is never
stale, that an active slot's heartbeat age agrees with LeaseExpiry by this
host's clock, and that a dead slot's lease ends recoveryLease after it
started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mattmattox added a commit that referenced this pull request Nov 14, 2026
…ans off the heartbeat (#151)

* fix(node): judge peers on this node's own clock, and keep registry scans off the heartbeat

Nodes judged each other's heartbeats and recovery leases by wall clock:
- a node mounting with its clock more than 10s ahead took a live node's
  registry slot, and that node stopped;
- a crashed node whose clock ran ahead was replayed only once its lead ran
  out, so lock waits on its records failed with EIO meanwhile;
- a replay whose node's clock was behind looked abandoned.

A node now records each slot it reads and counts the slot's node as failing,
dead or abandoned only once it has read the same heartbeat or lease for long
enough by its own monotonic clock. A replay renews its lease on its own clock,
to a value that always changes. Register takes an empty slot first and takes
another host's slot only after watching its heartbeat not change.

Under slow storage a node was declared dead while only slow: registry scans
held the heartbeat's lock across 64 reads, so at 1s per I/O a peer replayed
the node. Scans now read the registry in one I/O without that lock, and
registry I/O skips the per-region shard lock, as lock slots already do.

storage.Device gains SetReadFault, a read-side twin of SetWriteGate for tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(node): watch leases lapse in fsck's release tests, and renew test leases as replays do

fsck's release tests backdated a lease or heartbeat on disk and expected a
peer to act on it at once; peers now act only on what they watch stay
unchanged. The crash test lets a 1s lease lapse for real through
node.SetRecoveryLease, and the late mount comes back as the same host, the
one way it still takes a stale slot while another is empty. The test helper
that renews a lease now always changes it, as a replay's renewal does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(node): let a mount take another host's silent slot, after watching it

Taking empty slots first removed the way a mount on another host takes a
crashed node's slot and replays its journal, and with it -skip-recovery for
that journal. A mount now again takes another host's slot whose heartbeat
looks old by its clock, but only after watching the heartbeat stay unchanged
for HeartbeatTimeout plus two beats; a live node whose clock is behind beats
within that time and keeps its slot. Integration mounts may now take up to
30s to become ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(node): claim a watched slot only if it still holds what was watched, and take empty slots first

Register claimed a slot it had watched stay silent with a compare-and-swap
against a fresh read, so a slot a peer marked dead to replay it, or whose node
beat again, in between was taken anyway. It now swaps against the bytes it
watched, so any change makes the claim fail.

A mount takes an empty slot first, so it never waits while one is free.
Another host's silent slot is taken only when -skip-recovery names it, before
any other slot, or no slot is empty. A named slot under another node's
recovery is taken once its lease stays unrenewed for the lease term. Failed
replays now name -skip-recovery in their log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: notice a mount that exited while waiting for it

WaitForMount saw an exit only through ProcessState, which stays nil until
someone calls Wait, so a mount meant to fail always waited the full timeout.
It now also checks whether the process is a zombie.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(node): end the same-host namespace watch as soon as a slot's heartbeat is old

A restarted container keeps its hostname but gets a new PID namespace, so
its mount watched its old slot. The watch always ran its full 12s; it now
ends, as before, once the slot's heartbeat is older than HeartbeatTimeout by
this host's clock, which wrote it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(node): a rebooted host takes back the slot it held, after watching it

After a crash and reboot a host has a new boot ID, so its mount no longer
counted its old slot as its own and took an empty one, leaving the old slot
and its dirty journal for peers. Until a peer replayed it, the old node's
locks blocked the new mount, and an fsck run in between refused to repair.

A mount now watches the slots its hostname held in an earlier boot, whose
heartbeat looks old, before taking an empty one, and takes back one that
stays unchanged, replaying its journal before serving. Another host with the
same name beats during the watch and keeps its slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(node): a live host that shares the hostname keeps its slot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(node): judge fuzzed slots by the new liveness rules

The registry fuzz test from #153 called failed(), which this branch
replaces. It now checks that a slot seen for the first time is never
stale, that an active slot's heartbeat age agrees with LeaseExpiry by this
host's clock, and that a dead slot's lease ends recoveryLease after it
started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant