Repository navigation
Conversation
OFFICIAL_STOP_AT (node_alt_ref=official): the official even nodes, with more stake (OFFICIAL_FUND), are killed in a round they hold the majority of, stay down OFFICIAL_STOP_MINUTES and start again. COMMUNITY_BOOTSTRAP=none expects the network to stay stalled meanwhile; =trusted lists every community key in the community nodes' own trusted.txt and expects them to restart it. Every block is then compared across all nodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[community_bootstrap] confidants= lists community keys; empty (the default) keeps the node as before. After stall_minutes (default 30) without a block, a round package or a regular bootstrap table, and with no neighbour ahead, a listed or initial node may start, and any node with the list may take, a bootstrap table of initial and listed keys that holds more than half of the list, signed by its starter instead of the project. Round window and stall checks still apply; one such table per half round_elapse_time unless heavier; regular tables keep their own weight and always replace it. Official nodes and nodes without the list drop such tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
COMMUNITY_BOOTSTRAP=config turns the fallback on in every node of this build with every community key listed. With OFFICIAL_STOP_AT the network must restart without the official nodes and the fallback must not act before they stop; without it the fallback must never act. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al confidants With n1-n5 as initial confidants, 3 of them community, a bootstrap table that still names the stopped official nodes (stale neighbours) can decide with the 3 community ones, so the network sometimes restarted through the regular path and the fallback was not exercised. INITIAL_NODES picks the initial set: n1, n2, n3, n4, n6, three of them official. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a community table ends with random bytes after the signature, as a regular one: a repeat in the same round is no longer a p2p duplicate (Host::IsDuplicate) that a node whose fallback opens later never receives (cs::makeCommunityTable) - within half a round_elapse_time a heavier community table is taken only from a starter key not above the current one: a higher listed key cannot displace a running table by padding it with keys of stopped nodes - the starter keeps an even set when dropping a key would cost the table its majority of the list (cs::dropForOddTable) - a round package is progress only once its signatures were accepted (or it is the node's own), not when taken unchecked while lagging - a neighbour one block ahead holds the gate only for a further stall_minutes; two or more ahead always - a jump of the steady or wall clock between two checks (suspend/resume, clock set) is progress - gtests: signed bytes as getBootstrapTable reads them, the repeat differs and verifies, a table replayed from another round, even sets, padding, clock jumps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- COMMUNITY_STALL_STARTER: the community starter (lowest key) gets a shorter stall_minutes than the others; each of them must drop a table and take a repeat (official-stop-config-staggered) - OFFICIAL_PARTITION_AT: community nodes cut off from the running official nodes (iptables, UDP by port, TCP by cgroup) for OFFICIAL_PARTITION_MINUTES, then reconnected; partition.py reports the fork, which side was kept and what the other lost; one chain required at the end (official-partition) - logs and node binaries of branch runs kept 3 days Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion behaviour Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a restart variant After a 6 minute cut the two sides did not look each other up again (each node kept only its own side as neighbours). The run now reports the neighbour counts 120 s after the cut is lifted; OFFICIAL_PARTITION_RESTART=1 then restarts the community nodes, which dial every host again (official-partition-restart). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g 60 s after SIGTERM Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…189), and what CI measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-bootstrap Conflicts: docs/CHANGES.md (both rows kept), ci/testnet/run.sh (the reorganisation report and the community bootstrap checks both kept), robustness.yml (partition files uploaded, 3-day retention off master). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…E_FORK_AFTER) RESOLVE_FORK_AFTER=1 waits up to RESOLVE_FORK_WAIT minutes for FORK HELD on the community nodes, stops each one holding a fork, runs it once with --resolve-fork and starts it again: the operator choosing the official chain. New scenarios official-partition-resolve and official-partition-restart-resolve. Refusals are expected in the partition scenarios (DEEP_ROLLBACK_EXPECTED=1), and the reorganisation report is printed before the partition verdict. fork-held.json and reorg-episode.json are uploaded with the logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RESOLVE_FORK_AFTER: after --resolve-fork, a node whose fork point was only an upper bound and that holds the fork again lower is cut with --set-bc-top at the last common block (partition.py fork, from the chains dumped before the reconnection), as docs/quick-start.md tells the operator. A start right after a cut that stops at once is reported and repeated. The no-restart resolve scenario is dropped: without the restart the two sides never meet again, so no node holds a fork. partition.py report: a side that wrote no block while cut off (it stalled) is no fork when every node kept the other side's blocks. REORG DEPTHS and UNWINDS REFUSED are also printed when a partition run times out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… guard differences.md: the community side holds the fork (FORK HELD) and its operator chooses --resolve-fork (then --set-bc-top at the last common block when the fork point is an upper bound) or dismiss-fork; official nodes still follow the longer chain. The fallback's bootstrap blocks pass the guard on the community side; the option does not touch block removal. What the partition scenarios measured with the guard. CHANGES row updated. ci: run.sh prints node lists as n1 n3 ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node holding a fork is offered the other chain's blocks again and again. In CI (official-partition-restart, a community node FORK HELD for 75 min while official peers kept offering their chain) it wrote about 2,000 log lines a minute: 136 refusal lines, 680 sync requests, 680 block replies, 136 "blocks remaining". No file was written per refusal. The refusal line now comes once, then at most once a minute with the count since the previous one (cs::forkguard::RepeatedLine); while FORK HELD the sync request, block reply and "blocks remaining" lines go to the debug log. The FORK HELD round marker and the detail line once a minute are unchanged. run.sh sums the counts for UNWINDS REFUSED. Tests: the throttle (first line at once, then one a minute with the count), and refusals while held leave fork-held.json untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…E_FORK_AFTER) With RESOLVE_FORK_AFTER the chain of node 1, which takes the load, is cut back by --resolve-fork / --set-bc-top: a transfer it took on the cut branch can come back "transaction is expired". run.sh passes --chain-cut and fund.py counts it with the transfers of unknown outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt, one cut only Before --resolve-fork, wait (RESOLVE_LOCATE_WAIT, 6 min) until every node holding the fork reports "fork_point_status" in fork-held.json; an unverified located point is cut with --to-located. The run fails when a cut lands on an upper bound or a start right after the cut stops with an error. Needs the fork locator of #192 on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…% / 50 % The block rate of a window is the median of its one-minute rates, so one slow minute does not decide; both limits are reported (rate and round p95) and configurable (STRESS_RATE_DROP, STRESS_P95_RISE; defaults 10 and 20 as before). snapshot-p2p-stress-loss failed at -10 % against 10 % allowed: a steady fall of 5-16 % across runs on master and branches, not a regression. Its loss reaches the joiner n9 only after the joins, so the second window carries more lossy traffic than the first, and the first window's round p95 alone varies between identical runs (0.57 s or 0.8-0.93 s). It now allows a 20 % rate fall and a 50 % p95 rise. Checked on the logs of three runs: they pass (-9.5 to -11.4 %); the same logs slowed by 30 % give -37 to -40 % and slowed by 20 % give -26 to -31 %: both still fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Option A after a partition (FORK HELD, the operator follows the official chain with the located fork point of #192, --to-located for an unverified one, --set-bc-top otherwise, or keeps the community chain with dismiss-fork); why the option stays out of the set-up; the gate for H_activate_decentralization of BK's plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…key)
The workflow accepts scenario keys of capitals and underscores only, so STRESS_P95_RISE=50 stopped
snapshot-p2p-stress-loss before the network started ("bad scenario entry"). Same meaning: the
allowed rise of the round time's 95th percentile, in percent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A set at the highest block with listed keys that hold no majority of the list no longer blocks the regular restart: the initial confidants in it make the regular table (official odd rule), as on a node without the list (cs::bootstrapTableOf). Before, an initial confidant with the fallback open waited, and the official nodes waited for it. - dropForOddTable drops the last key when the rest is a regular table. - FORK HELD: refusedLine_ is reset in holdFork, so a new episode's first refused block is logged at once. - takeCommunityTable: its own log line for an empty table. - ci: partition_on removes a CSPART chain left by an earlier cut. - docs: known limitations (hostile lowest listed key, replay across a chain cut); CHANGES row. Tests: AMixedSetWithoutAMajorityOfTheListFallsBackToTheRegularTable, AnEvenSetKeepsItsMajorityOfTheList (regular rest), ANewForkHeldEpisodeLogsItsFirstRefusalAtOnce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, consensus-adjacent (bootstrap acceptance): not for merge before an independent review.
Depends on #192 for one CI scenario.
official-partition-restart-resolvenow waits for the fork point that #192's locator finds (fork_point_statusinfork-held.json) and fails on an upper-bound cut. On this branch alone (without #192) that scenario fails by design. #192 is to be merged first; nothing else here needs it. The node code of this PR does not depend on #192.Final state in short (2026-11-21): see "Final changes and CI (2026-11-21)" below. Partitions follow option A: the shorter community side holds the fork (FORK HELD) and its operator chooses.
[community_bootstrap]stays an expert option, not offered by setup. When BK'sH_activate_decentralizationlands, the fallback is to be gated onlastSeq + 1 < H_activate.Why
Only the initial confidants may start a bootstrap, and a table with any other key needs the project's signature. If all official nodes become unavailable at once and fewer than 4 initial confidants are left, the network stays stalled until they come back, whatever the community nodes do.
Part 1: what happens today (CI,
node_alt_ref=official)New robustness scenarios (
OFFICIAL_STOP_ATinci/testnet/run.sh, run by name): 10 nodes, the 5 even ones official (4a95aaa + CI hooks) with more stake, so they hold every trusted seat; the 5 odd ones community. Transfers run throughout. The official nodes are killed together in a round they hold the majority of (5 of 5 seats every time), stay down 5 minutes, and start again. Every block is then compared on every node over the API, and the state digests among community nodes.Final layout: the genesis initial confidants are n1, n2, n3, n4, n6 (
INITIAL_NODES), three of them official. Run https://github.com/CreditsCommunity/node/actions/runs/44122887562:official-stop-stalled(control)trusted.txtofficial-stop-trusted(option a)trusted.txtofficial-stop-config(this PR's option,stall_minutes=2)[community_bootstrap]listing the 5 community keysA first layout with n1-n5 as initial confidants (3 community, 2 official), run https://github.com/CreditsCommunity/node/actions/runs/44107764688, gave the same results for the control (stalled) and option a (restart after 22 s with the table n1, n3, n5, n7, n9; official nodes rejoined in ~20 s, one chain). With the option, though, run https://github.com/CreditsCommunity/node/actions/runs/44114953462 restarted by the regular path 20 s after the stop: the first elapse still saw the stopped nodes as neighbours at the same height, so the starter sent the table n1-n5, and 3 live seats out of 5 decide. That race exists in the official code too, whenever the stopped nodes are a minority of the initial confidants. Hence the official-majority initial set for the final runs.
The official node accepts blocks whose confidants are not its initial confidants: a bootstrap block skips the previous-round confirmation check (
BlockChain::storeBlock,isBootstrap), its signatures are checked against the block's own confidants, and the following blocks are confirmed by those. Only the bootstrap table is restricted, not the blocks.Part 2: an opt-in fallback that leaves consensus alone
[community_bootstrap]inconfig.ini, off unlessconfidants=lists keys:A separate list, not
initialConfidants_, so it survives a plan that clears that set after genesis. With the list set, and only while the fallback is open:stall_minutes(default 30, 1..10080), and no neighbour reports a block above this node's;round_elapse_timeunless larger; its weight is kept apart frombootStrapWeight_, so a regular table always replaces it.Official nodes, and community nodes without the list, drop a community table (a key outside their initial confidants, no project signature), then sync the blocks as in Part 1. In normal operation nothing changes: round packages keep the fallback closed, and the regular path is untouched apart from noting progress.
Exposure: expert option. Not offered by
setup.sh/setup.ps1/Docker env: the list only helps if it is the same on the participating nodes (majority rule), and the option should wait for review and a coordinated list (possibly shipped likeknown_hosts.txt). It is documented indocs/differences.md; Docker users edit~/.credits/config.ini(the entrypoint writes it only when missing).Tests:
tests/csnode/communitybootstrap_tests.cpp: table rules, weight, the stall gate, rate limit, config parsing, and the safety property as a simulation (with blocks at any interval belowstall_minutesfor a day, the gate never opens);official-stop-config: Part 1 with the option instead of the editedtrusted.txt(stall_minutes=2), which must restart and must not act before the stop;community-bootstrap-quiet: option on (stall_minutes=1) with the official nodes running, packet loss on two nodes and the restart test: it must never act.Results: the three runs above;
community-bootstrap-quietpassed (fallback on,stall_minutes=1, official nodes running, loss on two nodes, restart test, 1500 blocks: never used, 7 regular bootstrap lines on all nodes), run https://github.com/CreditsCommunity/node/actions/runs/44114953462; Linux build and unit tests pass.Review fixes (2026-11-18)
Host::IsDuplicatedropped it: a node whose fallback opened after the first send never got one. The table now ends with 4 random bytes after the signature (cs::makeCommunityTable); receivers already ignore what follows the signature. New scenarioofficial-stop-config-staggered: the starter (lowest community key) hasstall_minutes=2, the others 3.docs/differences.md. Threshold stays "more than half of the list".round_elapse_timea heavier community table is taken only if its starter key is not above the current table's starter. The starter of a table is its lowest key, and among the nodes left the lowest key starts anyway, so this grants nothing new to a lower key, while a listed node with a higher key can no longer displace a running table by padding its own with keys of stopped nodes. "First table wins" was not chosen: it would let whichever table arrives first win, including a padded one, and would slow convergence when two starters saw different sets.cs::dropForOddTable); otherwise the official rule applies.stall_minutes.stall_minuteswith no block.round_elapse_time) counts as progress (suspend and resume, clock set).H_activate_decentralizationlands, gate the fallback onlastSeq + 1 < H_activate, and later fold it into the BootstrapSignature protocol.New gtests: the signed bytes as
getBootstrapTablereads them (wrong signer and altered key refused), a repeat in the same round is new bytes and still verifies, a table replayed from another round does not verify, even sets, padding from a higher key, one block ahead, clock jumps.CI (2026-11-18,
node_alt_ref=official)official-stop-stalled: 0 blocks in 5 min; one chain after the restart.official-stop-trusted: restart after 24 s, 516 blocks in 5 min; one chain.official-stop-config: restart after 142 s, 403 blocks in 6 min; the official nodes catch up in about 20 s; one chain.official-stop-config-staggered: starter n5 sends its table at 120, 140 and 160 s, all in round R-422. Each of the 4 others drops 3 tables while its fallback is closed, then takes a repeat. Restart after 203 s; one chain.community-bootstrap-quiet: the fallback is never used.official-partition, community nodes cut off from the running official nodes for 6 min,stall_minutes=2). Runs https://github.com/CreditsCommunity/node/actions/runs/44256513402 and https://github.com/CreditsCommunity/node/actions/runs/44262716917 (job fails, as expected):official-partition-restart: the community nodes are restarted after the reconnection, so they dial every host). Run https://github.com/CreditsCommunity/node/actions/runs/44285502984:Updated to master with #189 (2026-11-20)
Master merged into the branch (no rebase): #189 (bounded automatic reorganisations, FORK HELD,
--resolve-fork,dismiss-fork) and #190. Conflicts indocs/CHANGES.md,ci/testnet/run.shandrobustness.ymlresolved keeping both features. Linux build and unit tests pass.Interplay with #189. This PR does not touch block removal, the rollback bound or the fork records (
removeLastBlock,--set-bc-top,allow_deep_rollbackare unchanged), so nothing here bypasses the guard. A community bootstrap block is signed by the confidants of the community table; a node that took that table holds it as a round table it accepted, which is what #189'sisBootstrapAuthenticatedLocallyasks of a bootstrap block. In CI the fallback's blocks pass the guard on the community side:official-stop-configandofficial-stop-config-staggeredshow 0 refused blocks and 0 reorganisations on every node.After a partition the official nodes still follow the longer chain. The community side holds the fork and reports FORK HELD. Its operator chooses
--resolve-fork(follow the official chain) ordismiss-fork(keep the community chain).docs/differences.mdsays so. (Superseded by the section of 2026-11-21: with #192 the node finds the exact fork point itself and one--resolve-forkcuts there.)Log volume while FORK HELD (commit 019269a, a small change to #189's code with tests). A community node holding a fork for 75 minutes while official peers kept offering their chain wrote about 2,000 log lines a minute: 136 refusal lines, 680 sync requests, 680 block replies and 136 "blocks remaining" lines. No file was written per refusal. The refusal line now comes once, then once a minute with a count ("N refused since the previous such line"). While FORK HELD, the sync request, reply and remaining lines go to the debug log. After the change: about 470 lines a minute, all of them round headers at CI's 0.5 s rounds, and 1 refusal line a minute (about 165 refusals counted).
node.logdropped from 18.9 MB to 3.2 MB. Tests:RefusedBlocksAreCountedNotLoggedOneByOneandRefusalsWhileHeldDoNotTouchTheForkFile.New CI step
RESOLVE_FORK_AFTER=1(scenarioofficial-partition-restart-resolve). After the reconnection and restart, it waits for FORK HELD on the community nodes, stops them and runs--resolve-fork. If a node reports FORK HELD again lower, it runs--set-bc-topat the last common block and starts the node again.partition.pynow treats a side that wrote no block as no fork. The partition scenarios setDEEP_ROLLBACK_EXPECTED=1, and REORG DEPTHS / UNWINDS REFUSED are printed even when a partition run times out.CI (2026-11-20,
node_alt_ref=official)Runs https://github.com/CreditsCommunity/node/actions/runs/44704679196, https://github.com/CreditsCommunity/node/actions/runs/44721479229, https://github.com/CreditsCommunity/node/actions/runs/44738015072 (log throttle) and https://github.com/CreditsCommunity/node/actions/runs/44751374778 (final).
official-stop-stalledofficial-stop-trustedofficial-stop-configofficial-stop-config-staggeredcommunity-bootstrap-quietofficial-partitionofficial-partition-restartofficial-partition-restart(first run)partition.pyverdict bug, now fixedofficial-partition-restart-resolve--resolve-forkcuts 1 block; FORK HELD again at #929 within 20 s;--set-bc-top 532. Then every node holds the official chain: one chain, 179 community blocks and 42 transactions reverted; state digests match. Final run: fork #548, the same path, one chain, 215 blocks / 49 transactions reverted. One run in between failed only because a load transfer came back "transaction is expired" after the cut (fund.py --chain-cutnow counts it as an unknown outcome)Regression: full Robustness suite at 7 nodes on the final commit 77c47d4, https://github.com/CreditsCommunity/node/actions/runs/44758102651. That run failed only on
snapshot-p2p-stress-loss(block rate −10 % against 10 % allowed: noise, see 2026-11-21). Two earlier full runs were cancelled before they finished: one hit a runner cancellation of 3 snapshot-p2p jobs (infrastructure), the other was stopped so that the suite runs only once, on the final commit.Final changes and CI (2026-11-21)
Operator procedure in the scenario (commit da68658, taken from
ci/resolve-followups-partition). After FORK HELD,official-partition-restart-resolvewaits up toRESOLVE_LOCATE_WAIT(6 min) for every holding node to report afork_point_status. Then:exact: plain--resolve-fork;located (unverified):--resolve-fork --to-located;The run also fails if a cut lands on an upper bound or a start right after a cut stops with an error. The
--set-bc-topfallback remains for diagnosis. Needs csnode: FORK HELD follow-ups: clean first start after a cut, exact fork point #192 (see the top).snapshot-p2p-stress-lossstabilised (commits 53333e4 and 01163d6, separate from the feature).STRESS_RATE_DROP,STRESS_ROUND_RISE; defaults 10 and 20 as before).snapshot-p2p-stress-lossonly: it allows a 20 % rate fall and a 50 % p95 rise.Docs (
docs/differences.md): option A with FORK HELD and the operator remedies (the located fork point,--to-located,--set-bc-top,dismiss-fork); why the option stays out of setup; theH_activategate for BK's plan. CHANGES row extended for the CI change.CI on the final commit 01163d6 (PR checks pass):
node_alt_ref=official)official-stop-stalledofficial-stop-trustedofficial-stop-configofficial-stop-config-staggeredcommunity-bootstrap-quietsnapshot-p2p-stress-losssnapshot-p2p-stress-losssnapshot-p2p-stress-lossWith #192:
official-partition-restart-resolveonci/resolve-followups-partition(this branch at 4f83309 merged with #192; the scenario logic is the same as on 01163d6), https://github.com/CreditsCommunity/node/actions/runs/44866942562 (pass):--resolve-forkeach cut there, with 0 failed starts;The same branch's earlier run https://github.com/CreditsCommunity/node/actions/runs/44813507041 found #506 in the same way (275 blocks reverted).
Open points
qs/0above the new head) and--resolve-forkcutting only an upper bound after a partition: both fixed by csnode: FORK HELD follow-ups: clean first start after a cut, exact fork point #192 (checkpoints above the new top removed; fork locator), as the run above shows.H_activate_decentralizationlands: gate onlastSeq + 1 < H_activate, then fold into BootstrapSignature.🤖 Generated with Claude Code