Skip to content

feat(alloc): hand writes data blocks from a per-node reserve - #142

Merged
mattmattox merged 7 commits into
mainfrom
feat/block-reserve-runtime
Nov 14, 2026
Merged

mattmattox merged 7 commits into
mainfrom
feat/block-reserve-runtime

Conversation

@mattmattox

@mattmattox mattmattox commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Writes take data blocks from a per-node reserve instead of taking the global bitmap lock per block. Builds on #135 (free path) and #139 (reserve records in the journal, replay and fsck).

Concurrent writes now scale with nodes. Probe on the final code: 300 x 1 MB files per node, 2 runs each, integration harness on one host (load average 43–49).

Nodes Writes/s before → after Slowest/fastest node Bitmap wait during writes
1 99 → 97 1.00 → 1.00 0–5% → 0%
2 106–108 → 169–180 0.73 → 0.95–0.97 39–41% → 0%
3 102–107 → 200 0.41–0.47 → 0.75–0.82 52–55% → 0%
4 104–108 → 223–254 0.26–0.41 → 0.71–0.86 60–64% → 1%

Deletes stay at 640–760/s; the post-release fix for their bitmap wait is in the probe notes.

Things to review:

  • Writers call TakeReady, Untake and Logged under an inode lock. The reserve's mutex covers only memory; refill, give-back, renewal and record ends run outside it, under endMu, which writers never take. TestWritersDoNotWaitForBitmapLock and TestInodeLockHoldersDoNotWaitForRefill fail if refill holds the mutex across the bitmap wait.
  • Each refill is its own BlockReserve record. A record ends when all its blocks are used, or early when the reserve goes back or the record falls a quarter ring behind the tail. A crash then leaks in-flight blocks; replay never frees a block twice. Blocks are freed only after their record's end is synced (CompleteIntentDurable), and not at all while any end is failing.
  • A refill writes its record before marking the bitmap. If a sector write fails, it ends the record, then restores the sectors, under the bitmap lock.
  • Record ends never wait for frees: returns free blocks after dropping endMu, and a full journal ring ends the oldest reserve record by moving the head past it.
  • A full bitmap logs at INFO, re-asks peers every second and waits up to 12s (an idle peer returns within 11s) before ENOSPC. Peer requests set a flag the reserve's 1s tick reads, so a flood costs one return per second.
  • df counts this node's unused reserve as free and other nodes' reserves as used until they go back, so two nodes can differ by up to 64 MB each. The Docker and VM tests that compare free space wait out the idle return.

Crash coverage: TestReplayKeepsMappedBlocksOfCrashedNodesReserve writes two files from a reserve (one through the BlockUse path), crashes the node, replays, and checks the data, the free count, fsck, and that allocating every free block never returns a mapped one. It fails if replay ignores uses. The VM test test_reserve_returned_after_crash also checks the killed node's file md5 after a 128 MB first-fit write on the survivor and after the node restarts.

Tests on 646cc7c: VM 2-node suite 65/0, Docker suite 54/0, full integration suite, unit -race, lint and parity all clean.

🤖 Generated with Claude Code

mattmattox and others added 7 commits November 14, 2026 00:30
Each node takes up to 64 blocks from the bitmap under one lock hold and
hands them to writes without the global bitmap lock. The reserve is a
journal record, so replay and fsck free a crashed node's unused blocks.
An idle reserve goes back after 10s, and a node that finds the bitmap
full asks peers for their reserves and waits up to 10s before ENOSPC.
No reserves below max(1%, 2 x size x live nodes) free blocks.

New -block-reserve flag (0 turns it off) and reserve metrics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…replay

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A write that refilled the reserve lowered df by the whole reserve, and
an unmount raised it again, so the statfs tests saw free space jump.
df now shows what this node can write without waiting. Other nodes'
reserves still count as used until they go back, so the shell tests
that compare free space wait out the idle return first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #142 found that Take held the reserve's mutexes while waiting
for the cluster bitmap lock, and writers took the same mutexes under an
inode lock. A node dying with the bitmap lock could then hang the
cluster: its replay needs an inode lock that a writer holds while it
waits behind the refill.

- The reserve mutex guards only memory. Refill, return, renew and
  record ends run outside it; writers never wait for them.
- Each refill is its own record. A record ends once all its blocks are
  used, or early when the reserve goes back or falls far behind the
  journal tail. A crash then leaks in-flight blocks, never frees twice.
- Blocks are freed only after their record end is synced
  (CompleteIntentDurable keeps the intent pending until then).
- A failed bitmap write in a refill ends the record, then restores the
  sectors, under the bitmap lock.
- A failed use commit leaves its blocks allocated instead of reusing
  them; failed commits are aborted.
- Peer return requests set a flag the reserve's tick reads, so a flood
  costs one return per second. The asker re-asks every second and
  waits 12s, longer than an idle return.
- Statfs clamps free blocks to the total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second review of #142:

- CompleteIntentDurable could never end a reserve record that is the
  oldest intent in a ring too full for a completion record, so every
  commit on the node then failed. It now moves the head past such an
  intent instead; a synced head past a record ends it for replay.
- A record that a failed refresh reopened after its last blocks were
  used never ended. Reopen now queues it as fully used.
- WaitForPeers called ReturnAll, which could wait on endMu while its
  holder waited for a bitmap claim that only the waiting replay frees.
  WaitForPeers now only asks for a return, and returns and tidies free
  blocks after dropping endMu, so its holders wait only for the journal.
- When a refill's record cannot be ended after a failed bitmap write,
  Reserve writes the remaining sectors again and keeps the blocks.
- A use commit that hit a full journal puts its blocks back.
- The cached free count saturates at zero instead of wrapping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Third review of #142: if ending the oldest intent in a full ring moved
the in-memory head but the superblock write or the sync failed,
CompleteIntentDurable put the intent back as pending behind the head.
The next checkpoint then wrote a LastCheckpointSeq the head no longer
matched, and replay skipped every pending intent.

Once the head has moved, the intent now stays out of the pending set and
is marked as an unsynced end; the retry rewrites the superblock and
syncs before it reports the record ended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 6ca8273 into main Nov 14, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant