Repository navigation
feat(alloc): hand writes data blocks from a per-node reserve - #142
Merged
Merged
Conversation
Each node takes up to 64 blocks from the bitmap under one lock hold and hands them to writes without the global bitmap lock. The reserve is a journal record, so replay and fsck free a crashed node's unused blocks. An idle reserve goes back after 10s, and a node that finds the bitmap full asks peers for their reserves and waits up to 10s before ENOSPC. No reserves below max(1%, 2 x size x live nodes) free blocks. New -block-reserve flag (0 turns it off) and reserve metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…replay Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A write that refilled the reserve lowered df by the whole reserve, and an unmount raised it again, so the statfs tests saw free space jump. df now shows what this node can write without waiting. Other nodes' reserves still count as used until they go back, so the shell tests that compare free space wait out the idle return first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #142 found that Take held the reserve's mutexes while waiting for the cluster bitmap lock, and writers took the same mutexes under an inode lock. A node dying with the bitmap lock could then hang the cluster: its replay needs an inode lock that a writer holds while it waits behind the refill. - The reserve mutex guards only memory. Refill, return, renew and record ends run outside it; writers never wait for them. - Each refill is its own record. A record ends once all its blocks are used, or early when the reserve goes back or falls far behind the journal tail. A crash then leaks in-flight blocks, never frees twice. - Blocks are freed only after their record end is synced (CompleteIntentDurable keeps the intent pending until then). - A failed bitmap write in a refill ends the record, then restores the sectors, under the bitmap lock. - A failed use commit leaves its blocks allocated instead of reusing them; failed commits are aborted. - Peer return requests set a flag the reserve's tick reads, so a flood costs one return per second. The asker re-asks every second and waits 12s, longer than an idle return. - Statfs clamps free blocks to the total. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second review of #142: - CompleteIntentDurable could never end a reserve record that is the oldest intent in a ring too full for a completion record, so every commit on the node then failed. It now moves the head past such an intent instead; a synced head past a record ends it for replay. - A record that a failed refresh reopened after its last blocks were used never ended. Reopen now queues it as fully used. - WaitForPeers called ReturnAll, which could wait on endMu while its holder waited for a bitmap claim that only the waiting replay frees. WaitForPeers now only asks for a return, and returns and tidies free blocks after dropping endMu, so its holders wait only for the journal. - When a refill's record cannot be ended after a failed bitmap write, Reserve writes the remaining sectors again and keeps the blocks. - A use commit that hit a full journal puts its blocks back. - The cached free count saturates at zero instead of wrapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Third review of #142: if ending the oldest intent in a full ring moved the in-memory head but the superblock write or the sync failed, CompleteIntentDurable put the intent back as pending behind the head. The next checkpoint then wrote a LastCheckpointSeq the head no longer matched, and replay skipped every pending intent. Once the head has moved, the intent now stays out of the pending set and is marked as an unsynced end; the retry rewrites the superblock and syncs before it reports the record ended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes take data blocks from a per-node reserve instead of taking the global bitmap lock per block. Builds on #135 (free path) and #139 (reserve records in the journal, replay and fsck).
Concurrent writes now scale with nodes. Probe on the final code: 300 x 1 MB files per node, 2 runs each, integration harness on one host (load average 43–49).
Deletes stay at 640–760/s; the post-release fix for their bitmap wait is in the probe notes.
Things to review:
TakeReady,UntakeandLoggedunder an inode lock. The reserve's mutex covers only memory; refill, give-back, renewal and record ends run outside it, underendMu, which writers never take.TestWritersDoNotWaitForBitmapLockandTestInodeLockHoldersDoNotWaitForRefillfail if refill holds the mutex across the bitmap wait.CompleteIntentDurable), and not at all while any end is failing.endMu, and a full journal ring ends the oldest reserve record by moving the head past it.dfcounts this node's unused reserve as free and other nodes' reserves as used until they go back, so two nodes can differ by up to 64 MB each. The Docker and VM tests that compare free space wait out the idle return.Crash coverage:
TestReplayKeepsMappedBlocksOfCrashedNodesReservewrites two files from a reserve (one through the BlockUse path), crashes the node, replays, and checks the data, the free count, fsck, and that allocating every free block never returns a mapped one. It fails if replay ignores uses. The VM testtest_reserve_returned_after_crashalso checks the killed node's file md5 after a 128 MB first-fit write on the survivor and after the node restarts.Tests on 646cc7c: VM 2-node suite 65/0, Docker suite 54/0, full integration suite, unit -race, lint and parity all clean.
🤖 Generated with Claude Code