Skip to content

csnode, solver: keep stage-1 hashes across a same-round bootstrap (partial hardening; consensus-relevant) - #77

Closed
EnzinoBB wants to merge 3 commits into
masterfrom
fix/bootstrap-hash-race
Closed

EnzinoBB wants to merge 3 commits into
masterfrom
fix/bootstrap-hash-race

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Consensus-relevant: do not merge without the maintainer's explicit OK. Status: partial hardening, does not fix the livelock.

What it does

  • getHash: a node that is not a confidant verifies and caches a hash of the round after its last block. The official node drops it.
  • getBootstrapTable: a table of the current round keeps the hashes already received.
  • SolverCore::gotHash: keeps one entry per sender.

Every check on a hash runs exactly as for a confidant, and the blocks are built from the same stage messages, so a mixed network does not fork. This was confirmed with node_alt_ref=master on contracts-net-loss, net-partition, mixed-power-cut, net-loss, chaos-mixed and bootstrap-livelock: equal state digests.

What it does not do

With #77 on every node, bootstrap-livelock still stalled once (run 39886642406, R-964.1). Fable's analysis of that run:

  • The hash race this PR fixes was not exercised: no node received a hash before its table.

  • The real livelock is a timing problem in official behaviour:

    • the starter re-sends the bootstrap table every round_elapse_time;
    • every same-round table makes each receiver restart the whole consensus (stages cleared);
    • a 7-confidant round with one late stage message needs the consensus fallbacks, which take 12.5 to 16.5 s;
    • so with a 10 s timer the round can never finish.

    Master without csnode, solver: keep stage-1 hashes across a same-round bootstrap (partial hardening; consensus-relevant) #77 shows the same pattern (run 39866725719, R-832.1).

Next

  • Consensus fix (separate PR, needs the maintainer's OK):
    • a re-sent table of the same round with the same confidants must not restart the consensus while the current attempt is younger than the fallback window;
    • the starter must not re-broadcast within that window;
    • reset bootStrapWeight_ on "Bootstrap off".
  • Deterministic reproduction: pause a trusted node for longer than round_elapse_time, with debug logs on chosen nodes (harness PR).

This PR can be kept as hardening or closed. It is not needed for the livelock fix.

🤖 Generated with Claude Code

An official stage-1 race (contracts-net-loss stall at block 831, run 39866725719, analysed by
Fable): each node sends its hash when it takes a bootstrap table; a hash that reaches a node before
that node took the table is dropped (getHash returns when not a confidant, or recv_hash is cleared
by onRoundStart(..., false)), so the last node to take the table never gets MinTrustedNodes-1
hashes, and with round_elapse_time below the ~12.5 s fallback chain the bootstrap repeats for ever.

- getHash: a node that is not a confidant verifies and hands to the solver a hash of the round
  after its last block (the solver caches it, NormalState ignores it); other rounds dropped as before
- getBootstrapTable: a table of the round the node is in keeps the received hashes
  (onRoundStart(..., true), as the starter already does), replayed by gotConveyerSync
- SolverCore::gotHash: one entry per sender, so kept hashes do not pile up across retries
- bootstrap-livelock becomes a regular robustness scenario

Every check of a hash runs as for a confidant; the block is built from the same stage messages, so
a mixed network does not fork. Consensus-relevant: to be merged only with the maintainer's OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB changed the base branch from ci/bootstrap-livelock to master October 23, 2026 01:53
EnzinoBB and others added 2 commits October 23, 2026 03:54
…race

# Conflicts:
#	.github/workflows/robustness.yml
#	docs/CHANGES.md
…00 blocks); CHANGES.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB EnzinoBB changed the title csnode, solver: keep the stage-1 hashes of a same-round bootstrap (consensus-relevant) csnode, solver: keep stage-1 hashes across a same-round bootstrap (partial hardening; consensus-relevant) Oct 23, 2026
@EnzinoBB

Copy link
Copy Markdown
Collaborator Author

Closed without merging. The debug logs of the captured livelock (run 39902923745, scenario bootstrap-livelock-debug) show that the stall starts from a fork of block 1433: two valid blocks that differ only in realTrustedMask, created by a stage-3 retry race. No hash was lost; the stuck nodes rejected matching-sequence hashes because they hold the other block. Keeping hashes across a same-round bootstrap, which is what this PR does, does not address that. The branch is kept. The fix will be a separate PR, after the maintainer's decision.

@EnzinoBB EnzinoBB closed this Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant