Skip to content

csnode: FORK HELD follow-ups: clean first start after a cut, exact fork point - #192

Draft
EnzinoBB wants to merge 5 commits into
masterfrom
fix/resolve-fork-followups
Draft

EnzinoBB wants to merge 5 commits into
masterfrom
fix/resolve-fork-followups

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Two follow-ups to the FORK HELD handling of #189. Both are local to the node: block content, validation and what the node accepts are unchanged.

1. Clean first start after a cut. After --resolve-fork or --set-bc-top, the first start stopped with exit code 255: the shutdown checkpoint qs/0 had been saved at the old head, the node loaded it, found that its blocks were no longer on disk, set it aside and exited; only the second start worked. A successful cut now also removes:

  • the block hashes above the new top (caches/hashdb, seqdb). Before, a checkpoint above the top still looked like part of this chain.
  • every quick-start checkpoint above the new top, with its .prev backup. A qs/0 without head.bin goes as well, because it cannot be placed.

The next start then quick-starts from the highest checkpoint at or below the top, or reads the chain from the start.

Review fixes:

  • Checkpoints above the top are set aside as qs/<n>.bad, not deleted.
  • On a storage that does not start at block 0 (pruned, or a backfill in progress), a cut with no checkpoint left at or below the new top is refused (TruncateNoCheckpoint, exit 3, nothing removed). Such a node cannot slow-start.
  • The block hashes are removed through the last block stored before the cut, across gaps. The transaction index is cut as before (trimToFloor). The checkpoint-ahead robustness scenario now checks that the cut removed the checkpoints. It then puts them back, as a restored backup would, so it still tests setting them aside.

2. The node finds the exact fork point. After a long partition the node knew only an upper bound of the fork point ("block #930 or lower" for a fork at #533). Each --resolve-fork cut one block, and the operator had to find the last common block by hand. While FORK HELD the node now asks the peer of the refused block for a few of its blocks at or below the bound, using the usual BlockRequest. It compares their hashes with its own and narrows down to the last block both chains share (cs::forkguard::ForkLocator). The answer goes into fork-held.json ("fork_point_status": "exact", with the original upper bound and the peer) and into the detail line, so --resolve-fork cuts there in one run.

  • Read-only: nothing is stored, unwound or approved. The peer's answers must hold together: matches below differences, and consecutive blocks chained. Otherwise the search fails and starts again 2 minutes later.
  • Verified before it counts. A peer can answer wrongly, so the located point becomes the fork point ("exact") only if the peer's block right after it verifies as a synced block would after this node's block there (BlockChain::verifiesAfterOurBlock): it must be built on that block, its signatures must verify, and a majority of the confidants of this node's block must have signed its round confirmations. A bootstrap block never verifies.
  • Otherwise the point is a hint: "located (unverified)", or "deeper" when the peer's block differs already at the lowest block searched. --resolve-fork then still cuts at the upper bound. It cuts at the hint only when the operator adds --to-located (Docker: resolve-fork to-located). Before it removes anything, --resolve-fork prints the node's head, the upper bound, the located point and its verification status, where it cuts and how many blocks that removes.
  • A reply counts for the search only if it holds no more blocks than asked for, each of a different height asked for. Each block is verified once at most, and only when it is not ours. A reply from the searched peer that is not for the search is dropped when nothing else waits for one.
  • A pruned peer that answers NoSuchBlocks raises the search floor past the lowest height asked for, a limited number of times. Duplicate-request refusals are retried, which can stretch the search to a few minutes on a busy peer.
  • Bounded and rate-limited: one request in flight, at most one every 6 s, up to 8 heights per request. The search goes down to 20 × max_auto_rollback blocks below the bound (1,000 to 10,000; 2,000 when it is 0). It never goes below the first stored block, and never asks for block 0, which both chains share (the official peer's genesis block hashes differently).
  • The safety properties of csnode: bound automatic chain reorganisations #189 stay: nothing unwinds automatically, and the cut still checks that the stored block at the fork point is the recorded one (TruncateMismatch). A located point approves no branch: resolve-approved.json is written only for an exact point named by the refused block itself, as before.

Docs: docs/quick-start.md ("If your node reports FORK HELD"), docs/differences.md, packaging/docker/README.md.

Tests:

  • Unit tests (tests/csnode/forklocator_tests.cpp) cover:
    • an exact fork point, including one at the bound and one at the floor;
    • a fork deeper than the bound, on full and pruned storage;
    • a lying peer (our own blocks everywhere, matches above a difference, a forged block after the fork point, unchained blocks, blocks nobody asked for);
    • a lying peer that reports a deeper point: the point is not verified and gives no deeper cut without --to-located;
    • a bootstrap block after the point, which stays unverified;
    • the checks of the next block (BlockChain.LocatedForkPointNeedsAVerifiedNextBlock);
    • no peer, no reply, and refusals;
    • rate limiting;
    • the record and the --resolve-fork plan for exact and deeper points;
    • BlockChain taking the answer only for the record searched for;
    • setting checkpoints aside after a cut;
    • a pruned cut below every checkpoint (TruncateNoCheckpoint);
    • --to-located with a changed block at the located point (TruncateMismatch);
    • removing block hashes across a gap;
    • a reply with duplicated blocks;
    • a pruned peer.
  • official-partition-restart-resolve (the csnode: community bootstrap fallback (opt-in) and official-stop CI scenarios #188 scenario) ran on the temporary branch ci/resolve-followups-partition (csnode: community bootstrap fallback (opt-in) and official-stop CI scenarios #188 + this PR, node_alt_ref=official). Every community node found the fork point exact and verified at #506 from the upper bound #1001, in 4 requests (about 30 s). One --resolve-fork per node, no --set-bc-top, no failed start after the cut: PASS at sequence 2000 with one chain. That branch's run.sh waits for fork_point_status before --resolve-fork; csnode: community bootstrap fallback (opt-in) and official-stop CI scenarios #188 can take that change.
  • An earlier run, before block 0 was excluded, showed the first-start fix alone: 10 cuts, 0 failed starts.
  • Robustness: a targeted subset (checkpoint-ahead, mixed-crash-lmdb, lmdb-pruned-crash, net-partition, all-storage-power-cut, hashdb-damage) passed; the full suite at 7 nodes ran on the final commit.

🤖 Generated with Claude Code

EnzinoBB and others added 4 commits November 20, 2026 20:43
A cut at start-up (--resolve-fork, --set-bc-top) now removes the block
hashes and every quick-start checkpoint above the new top, the .prev
backups included. The shutdown checkpoint qs/0 was saved at the old head:
the first start after a cut loaded it, found its blocks gone, set it
aside and exited 255; only the second start worked.

While FORK HELD with only an upper bound of the fork point, the node now
looks for the last common block itself (cs::forkguard::ForkLocator): it
asks the peer of the refused block for a few of its blocks at or below the
bound with the usual BlockRequest, compares their hashes with its own and
narrows down, one request every 6 s at most, down to 20 x max_auto_rollback
blocks (1,000 to 10,000). The answer goes into fork-held.json
("fork_point_status": "exact", or "deeper" beyond the bound), so that
--resolve-fork cuts at the exact point in one run. Read-only: nothing is
stored, unwound or approved; the peer's blocks only locate the point and
must hold together (matches below differences, consecutive blocks chained).
A located point approves no branch; the cut still checks the stored hash
there. A deeper fork makes --resolve-fork remove nothing.

The checkpoint-ahead scenario puts the removed checkpoints back after its
cut, so that it still tests setting them aside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The search for the last common block uses a peer's answers, which could
be wrong. The point it finds now becomes the fork point ("exact") only if
the peer's block right after it verifies as a synced block would after
our block there: built on it, its signatures against its confidants, and
round confirmations by a majority of the confidants of our block at the
point (BlockChain::verifiesAfterOurBlock). A bootstrap block never
verifies.

Otherwise the point is a hint ("located (unverified)"; "deeper" is a hint
too): fork_point stays the upper bound and --resolve-fork still cuts
there; it cuts at the hint only with --resolve-fork --to-located (Docker:
resolve-fork to-located). Before cutting, --resolve-fork prints our head,
the bound, the located point, its verification status, the cut and the
number of blocks it removes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In the official-partition-restart-resolve run every search stopped on
block 0: the official peer's genesis block did not match ours, which read
as a fork below block 474 that matched. Both chains share the genesis
block by definition; the search starts at block 1 at the lowest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EnzinoBB added a commit that referenced this pull request Nov 21, 2026
…nt, one cut only

Before --resolve-fork, wait (RESOLVE_LOCATE_WAIT, 6 min) until every node holding the fork reports
"fork_point_status" in fork-held.json; an unverified located point is cut with --to-located. The
run fails when a cut lands on an upper bound or a start right after the cut stops with an error.
Needs the fork locator of #192 on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EnzinoBB added a commit that referenced this pull request Nov 21, 2026
Option A after a partition (FORK HELD, the operator follows the official chain with the located
fork point of #192, --to-located for an unverified one, --set-bc-top otherwise, or keeps the
community chain with dismiss-fork); why the option stays out of the set-up; the gate for
H_activate_decentralization of BK's plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A cut (--resolve-fork, --set-bc-top) on a storage that does not start
  at block 0 (pruned, or a backfill in progress) is refused, removing
  nothing (Storage::TruncateNoCheckpoint, exit 3), when no quick-start
  checkpoint is left at or below the new top: such a node cannot
  slow-start. Checkpoints above the top are set aside as <name>.bad
  instead of being deleted.
- The block hashes are removed above the top through the last block
  stored before the cut (Storage::truncated_last), across gaps.
- The fork point search takes a reply only if it holds no more blocks
  than asked for, each of a different height asked for; each block is
  verified once at most, and only one that is not ours. A reply from the
  searched peer that is not for the search is dropped when nothing else
  waits for one.
- A pruned peer (NoSuchBlocks) raises the search floor past the lowest
  height asked for instead of counting a failed attempt, a limited
  number of times.
- Docs: a verified point may lie lower than the refused branch's own
  fork; the cut then removes more blocks than needed (a resync).

Tests: a pruned cut below every checkpoint, the --to-located cut of a
changed block (TruncateMismatch), the hash removal across a gap,
duplicated reply blocks, a pruned peer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant