Skip to content

csnode: bound automatic chain reorganisations - #189

Merged
EnzinoBB merged 15 commits into
masterfrom
fix/bound-reorg
Nov 20, 2026
Merged

EnzinoBB merged 15 commits into
masterfrom
fix/bound-reorg

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Bound automatic chain reorganisations

A node that is behind and syncs a longer competing chain unwinds its own confirmed blocks down to the fork point before adopting the other chain. The official node places no limit on that depth.

This change bounds the depth of every automatic, non-operator rollback and is local and consensus-neutral: it changes only how far this node unwinds on its own, never any block's content or what the node accepts in normal operation. A mixed official/community network behaves exactly as before in normal operation.

What it does

  • The depth check lives inside removeLastBlock() and compromiseLastBlock(), so every automatic removal path is bounded: the sync fork path, the uncertain-block replacement, the antifork cache and the bad-hash reply. The bound (max_auto_rollback, default 100) is measured against a fixed reference captured at the first removal of an episode. The reference is kept in reorg-episode.json next to the database, so neither forward progress nor a restart resets it.
  • A non-bootstrap competing block triggers a rollback only if its round confirmations verify against the confidants of the node's own last block. A bootstrap block never passes that check (the block verifier accepts it without confirmations); it passes only when authenticated locally (own signatures plus the confidants of a round table the node accepted).
  • The start-up integrity repair (blocks of the node's own database that fail the hash check) is a local repair and is not bounded.

FORK HELD

When the node refuses to unwind, it keeps its chain and reports FORK HELD until the episode ends or the operator decides:

  • | FORK HELD on every round line (R-<n> NORMAL | FORK HELD) and a detail line once a minute: since when, the fork point and its hash, our head, the refused block, peer key, address and announced height, and both remedies;
  • fork-held.json next to the database with the same data;
  • the Docker healthcheck reports the container unhealthy, with both remedies.

The state clears when the head climbs back above the reference, or the operator acts. After checking with other nodes or the monitor:

  • This chain is right: create a dismiss-fork file next to the database (docker compose exec node touch /data/dismiss-fork). The running node picks it up within a round. That branch stays refused but is no longer reported (at most an hourly info line); a different branch is reported again. The dismissal survives restarts.
  • The other chain is right: node --resolve-fork (Docker: docker compose stop && docker compose run --rm node resolve-fork && docker compose up -d). It cuts the chain at the recorded fork point through the --set-bc-top path, only if the stored block there still has the recorded hash (otherwise nothing is removed), and clears the record. The node then syncs forward from there. It relaxes nothing else. When the fork point is exact, only the recorded competing block itself (that very hash) may replace a block the node stored again right after the fork point. Unauthenticated offers neither bypass the checks nor cancel the approval. Nothing is ever unwound to or below the fork point. When the fork point is only an upper bound, --set-bc-top <sequence> cuts by hand at the last common block.
  • allow_deep_rollback=true / CS_ALLOW_DEEP_ROLLBACK remains the last resort; it turns itself off after one completed episode.

CI

ci/testnet/run.sh prints REORG DEPTHS and UNWINDS REFUSED. A normal run fails if the bound stops a reorg or any competing block is refused; set DEEP_ROLLBACK_EXPECTED=1 for a scenario that provokes one.

Tests

Unit tests cover:

  • the bound and its fixed reference;
  • refusals at the storeBlock level, which leave the head and the uncertain flag untouched;
  • the antifork abandonment state and the integrity repair;
  • config and environment parsing;
  • the reference and FORK HELD across a simulated restart;
  • dismissal: cleared, still refused, the same branch stays quiet, another branch is raised, the trigger is consumed, and the dismissal survives a restart;
  • the narrow --resolve-fork approval: a forged block pointing at the recorded hash and unauthenticated offers are refused and do not cancel it, the recorded block replaces the re-stored one, nothing goes below the fork point, and the approval clears;
  • the resolve plan, and the storage cut landing exactly at the fork point or refusing a mismatched record;
  • a bootstrap block with confirmations.

The Docker entrypoint test covers the new commands and the healthcheck. The full unit suite passes locally (559 tests in nodetests).

Docs: docs/differences.md ("Consensus fixes"), docs/quick-start.md ("If your node reports FORK HELD"), packaging/docker/README.md, docs/CHANGES.md.

Draft: for maintainer review; do not merge.

🤖 Generated with Claude Code

EnzinoBB and others added 3 commits November 18, 2026 10:31
A node that is behind and syncs a longer competing chain rolls back its own
confirmed blocks down to the fork point before taking the other chain. The
official node puts no limit on that depth, so a competing chain can make a
syncing node unwind arbitrarily deep.

Bound the depth of every automatic, non-operator rollback (this builds on
the earlier local draft b8e5cb59, cherry-picked and extended):

- The depth check now lives inside removeLastBlock() and compromiseLastBlock(),
  so every automatic removal path goes through it: the sync fork path, the
  uncertain-block replacement, the antifork cache and the bad-hash reply. The
  bound is measured against a fixed reference captured at the first removal of
  an episode, so forward progress cannot reset it and a node cannot be made to
  creep past the limit by alternating removals and appends. There is no
  operator-initiated truncation path, so all callers are automatic.

- A competing block that is NOT a bootstrap block triggers a rollback only if
  its round confirmations verify against the confidants of the node's own last
  block -- the authority it already trusts -- exactly as BlockVerifier checks a
  block from the network. A block without confirmations from that authority
  never makes the node unwind.
  An honest micro-fork still works, because the honest competing block at the
  same height was produced by the same deterministically elected trusted group.
  A bootstrap block carries no such confirmations and cannot be checked locally;
  for it the depth bound plus the operator override is the only defence.

Once the bound would be crossed the node keeps its chain, logs a prominent alert
and waits for the operator; it sets no permanent state, so a later legitimate
reorg still works, and it does not crash-loop. The bound ([params]
max_auto_rollback, default 1000) and a deep-switch override ([params]
allow_deep_rollback, or the CS_ALLOW_DEEP_ROLLBACK environment variable) are
configurable. A node that adopted a community bootstrap table may unwind back to
the sequence it held when it took the table (setCommunityBootstrapFloor), which
keeps this coherent with the community bootstrap fallback. Each finished
rollback episode logs its depth; ci/testnet/run.sh reports the deepest one and
fails a normal run if the bound ever stops a reorg.

Local safety bound only: it changes how far this node unwinds on its own and
never changes any block content or what the node accepts in normal operation, so
it stays byte-compatible and invisible to other nodes, and a mixed
official/community network behaves exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… days off master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate in front of an automatic unwind is now one function,
competingBlockMayTriggerUnwind(), checked before the node touches its last
block at all, so a competing block that fails it cannot even mark that block
uncertain (which would let a later replacement remove it).

- A non-bootstrap competing block needs strict round confirmations: a mask of
  more than one entry, of the same size as the node's last confidants, as many
  signatures as it marks, and a majority of those confidants confirming; then
  BlockVerifier::verifyConfirmations. verifyConfirmations itself is unchanged,
  so normal block validation stays as it was.
- A bootstrap flag is a bit any block can carry, so it never opens a weaker
  gate: a bootstrap-flagged competing block passes only when its own signatures
  verify and its confidants are exactly those of a round table this node
  accepted in the last few rounds.
- The operator override (allow_deep_rollback / CS_ALLOW_DEEP_ROLLBACK) lifts
  the gate and the depth bound, for a deliberate switch to a chain confirmed by
  confidants this node does not hold.

Unit tests cover the strict confirmations and the bootstrap gate. run.sh
reports how many competing blocks were refused. The docs describe the effect
after a partition: a node on the shorter side keeps its chain until the
operator decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EnzinoBB added a commit that referenced this pull request Nov 18, 2026
…189), and what CI measured

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EnzinoBB and others added 12 commits November 18, 2026 14:21
Storage keeps a small read cache of pools indexed uniquely by sequence and by hash.
pool_remove_last() did not drop the removed pool from it, and pools_cache_insert()
silently refused a new pool at a sequence still cached. After a node unwound blocks
already written to storage and saved the competing blocks at the same sequences,
pool_load(sequence) kept returning the removed block. The next recordBlock() then
failed its consistency check ("block hash in pool #N doesn't correspond to the last
one") on every attempt, and the node stopped a few blocks above the fork point.

Erase the pool from the cache on every removal path, and let an insert replace any
cached pool with the same sequence or hash. Read-side only: block content and what
the node accepts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BlockChain::removeLastBlock() falls back to pool_remove_last_repair() when the
removed block cannot be read back. Check that this path also forgets the block
in the storage read cache, that a block saved afterwards at the same sequence
is the one read back, and that after a restart (the storage rescan fills the
cache again) the node reads the replacement, not the removed block. Both with
synchronous and asynchronous writes. Tests only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bound on automatic chain reorganisations now defaults to 100 blocks
(Consensus::MaxAutoRollbackDepth and DEFAULT_MAX_AUTO_ROLLBACK, kept in step).

- The episode reference is kept in reorg-episode.json next to the database
  and loaded at start-up, so a restart does not give a fresh full depth.
- When the node refuses to unwind (beyond the bound, or a competing block not
  vouched for by its confidants) it records the fork in fork-held.json (fork
  point and hash, our head, the refused block, peer key, address and height,
  time, reference) and reports FORK HELD: a marker on every round line and a
  detail line with both remedies once a minute. The state clears when the head
  is back above the reference or the operator truncates.
- Keep this chain: a dismiss-fork file next to the database, picked up by the
  running node within a round (also at start-up). The record keeps that branch
  as dismissed: still refused, logged at most hourly, not reported again; a
  different branch is reported again. The dismissal survives restarts.
- Follow the other chain: --resolve-fork cuts the chain at the recorded fork
  point through the --set-bc-top path, only if the stored block there still
  has the recorded hash (otherwise nothing is removed), clears the record and
  approves following the other chain for the one episode that follows.
- allow_deep_rollback / CS_ALLOW_DEEP_ROLLBACK turns itself off after one
  completed episode, loudly.
- The start-up integrity repair (incorrectBlocks_) is not bounded by the guard.
- hasTrustedConfirmations refuses a bootstrap block, which BlockVerifier
  accepts without confirmations.
- The unused community-bootstrap floor hook is removed; no message advises a
  restart any more.

Tests: storeBlock refusals leave the head and the uncertain flag untouched,
the antifork abandonment state, the integrity repair, config and environment
parsing, the reference and FORK HELD across a simulated restart, dismissal
(cleared, still refused, same branch quiet, other branch raised, trigger
consumed, kept across restart), the resolve plan, and the storage cut landing
exactly at the fork point or refusing a mismatched record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…il a normal run

- Docker: resolve-fork, set-bc-top <sequence> and dismiss-fork commands
  (docker compose run --rm node <command>); the healthcheck reports the
  container unhealthy while fork-held.json holds a fork, with both choices.
  Entrypoint test covers the commands and the healthcheck.
- config.ini: max_auto_rollback default 100, --resolve-fork / --set-bc-top as
  the remedy, allow_deep_rollback as the last resort.
- ci/testnet/run.sh: any refused competing block (UNWINDS REFUSED > 0) fails a
  normal run, as a stopped reorg already did, unless DEEP_ROLLBACK_EXPECTED=1;
  FORK HELD nodes are listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- quick-start.md: "If your node reports FORK HELD": what it means, check with
  other nodes or the monitor first, keep this chain (dismiss-fork) or follow
  the other one (--resolve-fork, --set-bc-top), native and Docker, the
  override as the last resort.
- differences.md: the FORK HELD state, the persisted reference, the remedies,
  the default of 100, the lagging node that waits for the operator, the
  integrity repair; the community-bootstrap floor sentence is gone.
- CHANGES.md: the #189 row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--resolve-fork no longer lifts the authority check and the depth bound for
the episode after the cut. The cut alone brings the node to the fork point,
from which it syncs forward. When the recorded fork point is exact, the
recorded competing block right after it, and the block built on it, may
replace a block the node stored above the fork point again
(resolve-approved.json). Nothing is ever unwound to or below the fork point,
and any other competing block meets the usual checks. The approval ends once
the node follows that block, goes past that height with another block, or is
offered another block at that height. When the fork point is only an upper
bound, nothing is approved and the operator is told how to cut by hand.

Tests: another branch is still refused after a resolve, the recorded branch
replaces the re-stored block, nothing goes below the fork point, and the
approval clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The approval no longer lets a block at fork+2 that merely points at the
recorded hash skip the authority check. Only the recorded competing block,
that very hash, can make the node give up its own block, and only when that
block was stored again right after the fork point. The recorded block must
build on the same block as the own one. Every other competing block meets
the usual checks.

An unauthenticated offer no longer cancels the approval or changes any
persisted guard state. The approval ends when the node goes on past that
height with a block of its own, or when a competing block that passes the
usual checks is offered there.

Tests:
- a forged fork+2 block pointing at the recorded hash is refused, and the
  own block and the approval are kept;
- unauthenticated offers do not cancel the approval;
- the recorded block replaces the own one, down to the fork point and never
  below;
- a vouched offer ends the approval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With pipefail, a grep that finds nothing ended the script right after
REORG DEPTHS, so every run failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unce, corrupt files)

- M1: the guard files (reorg-episode.json, fork-held.json,
  resolve-approved.json) are removed when the database is replaced: a new
  empty database at init, and a successful snapshot import. A restored node
  no longer resumes an old episode or FORK HELD.
- M2: --resolve-fork refuses a dismissed record ("dismissed on <date>:
  nothing held"), and also an unreadable one.
- M3: a refusal for a block not vouched for by our confidants is still
  immediate, but becomes FORK HELD (files, banner, healthcheck) only if the
  head stays put, at the second round start after it or a minute later. A
  head that moves drops it, so a peer on a minority branch cannot make
  FORK HELD flap. Depth refusals are reported at once.
- M4: an existing guard file that cannot be read is logged and fails safe:
  - reorg-episode.json starts an episode at the head;
  - an unreadable fork-held.json stays FORK HELD with an "unreadable" record
    that --resolve-fork cannot act on;
  - an unreadable resolve-approved.json approves nothing.
  The files are written to a temporary file, flushed and fsynced, then
  renamed, and the folder is fsynced.
- L1: the dismissed block's hash is never evicted, and the dismissed branch
  grows only through blocks that pass the authority check.
- L2: a 5 s timer on the processor thread polls dismiss-fork and pending
  holds and logs the FORK HELD detail line, also on a node with no rounds.

Tests:
- a dismissed record refused by planResolve;
- empty and garbage files of all three kinds;
- a restart below the floor, and clearAll;
- an authority refusal followed by the next block: no files, no flap;
- dismissed-hash eviction;
- a pruned fork point refused with TruncateMismatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Packaging:
- snapshot-restore.sh, CREDITS_RESET=1 and the cleanup of an interrupted
  restore remove reorg-episode.json, fork-held.json and
  resolve-approved.json with the database.

Docs:
- quick-start and the Docker README say to delete these files with db/.
- FORK HELD is reported only when the head stays put; the trigger file is
  picked up within a few seconds.
- With an encrypted key, resolve-fork and set-bc-top ask the password.
- Swarm, Kubernetes and autoheal would restart-loop on the unhealthy check;
  plain restart: unless-stopped does not.
- differences.md and CHANGES.md are updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB marked this pull request as ready for review November 20, 2026 08:33
@EnzinoBB
EnzinoBB merged commit 4286a0f into master Nov 20, 2026
276 of 282 checks passed
EnzinoBB added a commit that referenced this pull request Nov 20, 2026
…-bootstrap

Conflicts: docs/CHANGES.md (both rows kept), ci/testnet/run.sh (the
reorganisation report and the community bootstrap checks both kept),
robustness.yml (partition files uploaded, 3-day retention off master).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant