Skip to content

ci(testnet): round_elapse_time 20 s, fail a run stuck in a bootstrap, bootstrap-livelock reproduction - #76

Merged
EnzinoBB merged 2 commits into
masterfrom
ci/bootstrap-livelock
Oct 23, 2026
Merged

EnzinoBB merged 2 commits into
masterfrom
ci/bootstrap-livelock

Conversation

@EnzinoBB

Copy link
Copy Markdown
Collaborator

Why

contracts-net-loss stalled at block 831 for over an hour (run 39866725719). Fable's analysis found an official stage-1 hash race in a bootstrap round:

  • each node sends its hash as soon as it takes the bootstrap table;
  • a hash that reaches a node before that node has taken the table is dropped, because the round number is the same;
  • the node that takes the table last never collects 3 hashes;
  • the consensus's fallback for a silent node takes about 12.5 s, which is longer than the 10 s round_elapse_time after which the table is sent again, so the bootstrap never completes.

This is not caused by #70 or #71. With mainnet's 60 s default the round does end, through the fallback, after at least 12.5 s.

Changes

  • Timer. run.sh: new ROUND_ELAPSE_MS, default 20000 (it was a fixed 10000), so slow rounds are no longer pre-empted by a bootstrap.
  • Stuck check. A run fails if n1's last three round headers are bootstrap sub-rounds. A stall after the target used to pass: run 39846198921 ended stuck at R-1'192.1.
  • Reproduction scenario. New robustness scenario bootstrap-livelock: 15 % loss on n3 and n5, 10 s timer, target 3000. It is a manual scenario: it runs only when named in only, until the node fix lands.

The node fix (keeping the hashes of a same-round bootstrap) is consensus-relevant and will be a separate PR.

🤖 Generated with Claude Code

… a bootstrap-livelock reproduction

contracts-net-loss stalled at block 831 for over an hour (run 39866725719). Fable's analysis: a
round slowed by the loss lasted more than round_elapse_time (10 s here), a bootstrap of the full
7-confidant table started, and it can never complete: each node sends its stage-1 hash as soon as
it takes the table, a hash that reaches a node before that node took the table is dropped (same
round number), the last node never gets 3 hashes, and the consensus's fallback for a silent node
(about 12.5 s) is longer than the 10 s after which the table is sent again. Official behaviour,
not caused by #70/#71; with mainnet's 60 s default the round ends after the fallback instead.

- run.sh: ROUND_ELAPSE_MS (default 20000, was a fixed 10000)
- a run fails if n1's last three round headers are bootstrap sub-rounds (a stall after the
  target used to pass: run 39846198921 ended stuck at R-1'192.1)
- robustness: bootstrap-livelock (loss on n3 and n5, 10 s, target 3000) keeps the reproduction; it
  runs only when named in `only` (manual scenarios), until the node keeps those hashes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB merged commit 5f3d5be into master Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant