Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8acbd30
ci: official-stop scenarios: every official node stopped at once
EnzinoBB Nov 17, 2026
a5e2700
csnode: community bootstrap fallback, off by default
EnzinoBB Nov 17, 2026
fd5bf96
ci: official-stop-config and community-bootstrap-quiet scenarios
EnzinoBB Nov 17, 2026
b301669
docs: differences.md: the community bootstrap fallback
EnzinoBB Nov 17, 2026
707146c
docs: CHANGES row for #188
EnzinoBB Nov 17, 2026
20fdc17
ci: official-stop scenarios with an official majority among the initi…
EnzinoBB Nov 17, 2026
3a56000
csnode: community bootstrap fallback review fixes
EnzinoBB Nov 18, 2026
03973a5
ci: staggered community stall and official-partition scenarios
EnzinoBB Nov 18, 2026
897953c
docs: differences.md: community bootstrap review fixes and the partit…
EnzinoBB Nov 18, 2026
67c2b8c
ci: official-partition: neighbour counts after the reconnection, and …
EnzinoBB Nov 18, 2026
24903fe
ci: official-partition-restart: SIGKILL a community node still runnin…
EnzinoBB Nov 18, 2026
6d44d6c
docs: differences.md: the partition with the bounded-rollback guard (…
EnzinoBB Nov 18, 2026
775d203
Merge master (#189 bounded reorganisations, #190) into feat/community…
EnzinoBB Nov 20, 2026
6b01fff
ci: official-partition: --resolve-fork after the reconnection (RESOLV…
EnzinoBB Nov 20, 2026
bb32d26
ci: official-partition: the operator path end to end; no-fork verdict
EnzinoBB Nov 20, 2026
947a7a8
docs: community bootstrap after a partition with the bounded-rollback…
EnzinoBB Nov 20, 2026
019269a
csnode: fork guard: count refused blocks instead of a line each
EnzinoBB Nov 20, 2026
77c47d4
ci: fund.py: a transfer expired after a chain cut is no fault (RESOLV…
EnzinoBB Nov 20, 2026
da68658
ci: official-partition-restart-resolve: wait for the located fork poi…
EnzinoBB Nov 21, 2026
53333e4
ci: snapshot stress check: median minute rate; stress-loss allows 20 …
EnzinoBB Nov 21, 2026
4f83309
docs: differences.md: community bootstrap, final state
EnzinoBB Nov 21, 2026
01163d6
ci: snapshot stress check: STRESS_ROUND_RISE (no digit in a scenario …
EnzinoBB Nov 21, 2026
fdf123a
csnode: community bootstrap: final review fixes
EnzinoBB Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 67 additions & 2 deletions .github/workflows/robustness.yml
Original file line number Diff line number Diff line change
Expand Up @@ -392,14 +392,20 @@ jobs:
contracts: true
# stress on the confidants that serve (n1, n3, n5, n7): four nodes join at once, the first killed
# during its fetch and its import; n3 and n9 on a 2 Mbit/s link. The block rate and the round
# times while serving must stay within 10 % / 20 % of the same window before the joins
# times while serving must stay within 10 % / 20 % of the same window before the joins (the rate
# is the median of the one-minute rates)
- name: snapshot-p2p-stress
official: 'no' # four joining nodes: the even ones would be official
env: NODES=8 TRUSTED_NODES=7 NODE_STORAGE=lmdb,berkeleydb,lmdb:pruned,berkeleydb,rocksdb,berkeleydb,lmdb,berkeleydb,lmdb:pruned SNAPSHOT_SERVE_NODES=1,3,5,7 JOIN_FETCH=p2p JOIN_PRUNED_AT=2300 JOIN_MIN_SOURCES=3 STRESS_WINDOW=300 TARGET_SEQ=3300 JOIN_EXTRA=3 JOIN_FETCH_RATE=200 JOIN_KILL_DURING_FETCH=1 NET_FAULT=rate NET_FAULT_RATE=2mbit NET_FAULT_NODES=3,9
fixed: true
# 5 % loss on n3 and n9: 20 % / 50 % allowed. The loss reaches the joiner n9 only once it has
# joined, so the second window has more lossy traffic than the first, whatever serving costs;
# and the first window's round p95 alone varies between identical runs (0.57 s or 0.8-0.93 s).
# Measured: a steady fall of 5-16 % (-10 % failed the former 10 % limit); the same logs slowed
# by 30 % (or 20 %) give -37 to -40 % (-26 to -31 %), which still fails
- name: snapshot-p2p-stress-loss
official: 'no' # four joining nodes: the even ones would be official
env: NODES=8 TRUSTED_NODES=7 NODE_STORAGE=lmdb,berkeleydb,lmdb:pruned,berkeleydb,rocksdb,berkeleydb,lmdb,berkeleydb,lmdb:pruned SNAPSHOT_SERVE_NODES=1,3,5,7 JOIN_FETCH=p2p JOIN_PRUNED_AT=2300 JOIN_MIN_SOURCES=3 STRESS_WINDOW=300 TARGET_SEQ=3300 JOIN_EXTRA=3 NET_FAULT=loss NET_FAULT_PCT=5 NET_FAULT_NODES=3,9
env: NODES=8 TRUSTED_NODES=7 NODE_STORAGE=lmdb,berkeleydb,lmdb:pruned,berkeleydb,rocksdb,berkeleydb,lmdb,berkeleydb,lmdb:pruned SNAPSHOT_SERVE_NODES=1,3,5,7 JOIN_FETCH=p2p JOIN_PRUNED_AT=2300 JOIN_MIN_SOURCES=3 STRESS_WINDOW=300 STRESS_RATE_DROP=20 STRESS_ROUND_RISE=50 TARGET_SEQ=3300 JOIN_EXTRA=3 NET_FAULT=loss NET_FAULT_PCT=5 NET_FAULT_NODES=3,9
fixed: true
# with contracts running before and during the joins
- name: snapshot-p2p-stress-contracts
Expand Down Expand Up @@ -497,6 +503,61 @@ jobs:
- name: bootstrap-livelock-rewards
env: NET_FAULT=loss NET_FAULT_PCT=15 NET_FAULT_NODES=3,5 ROUND_ELAPSE_MS=10000 REWARD_ROUND=100 LOAD_SECONDS=2400 TARGET_SEQ=3000
manual: true
# if all official nodes become unavailable at once (node_alt_ref=official; run by name): 5
# official even nodes with more stake hold the trusted seats, 5 community odd nodes; n1, n2,
# n3, n4 and n6 are the genesis initial confidants (official majority: a table that still
# names the stopped nodes cannot decide). They are killed in a round they hold the majority of, stay
# down 5 minutes and start again. Without extra keys the network must stay stalled until
# they are back; with every community key in the community nodes' own trusted.txt the
# community nodes must restart it, and the official nodes must then take the same blocks
- name: official-stop-stalled
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_STOP_AT=400 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=none RESTART_AT=0 LOAD_SECONDS=1500
fixed: true
manual: true
- name: official-stop-trusted
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_STOP_AT=400 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=trusted RESTART_AT=0 LOAD_SECONDS=1500
fixed: true
manual: true
# the same with the community bootstrap fallback ([community_bootstrap], stall_minutes=2) in
# place of the hand-edited trusted.txt: it must restart the network, and not act before
- name: official-stop-config
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_STOP_AT=400 OFFICIAL_STOP_MINUTES=6 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=config COMMUNITY_STALL_MINUTES=2 RESTART_AT=0 LOAD_SECONDS=1500
fixed: true
manual: true
# the same with staggered stall_minutes: the starter (the community node with the lowest key)
# 2, the others 3. Its first tables reach nodes whose fallback is still closed: the restart
# needs a repeat of the table in the same round to get through (not dropped as a p2p duplicate)
- name: official-stop-config-staggered
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_STOP_AT=400 OFFICIAL_STOP_MINUTES=7 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=config COMMUNITY_STALL_MINUTES=3 COMMUNITY_STALL_STARTER=2 RESTART_AT=0 LOAD_SECONDS=1500
fixed: true
manual: true
# the community nodes cut off from the RUNNING official nodes for 6 minutes (fallback on,
# stall_minutes=2): both sides write blocks; after the reconnection the run reports which
# side the nodes kept, how many blocks and transactions the other lost, and needs one chain
# (refusals and FORK HELD on the community side are expected: DEEP_ROLLBACK_EXPECTED=1)
- name: official-partition
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_PARTITION_AT=400 OFFICIAL_PARTITION_MINUTES=6 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=config COMMUNITY_STALL_MINUTES=2 RESTART_AT=0 LOAD_SECONDS=1500 TARGET_SEQ=2000 DEEP_ROLLBACK_EXPECTED=1
fixed: true
manual: true
# the same, the community nodes restarted after the reconnection (they dial every host again)
- name: official-partition-restart
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_PARTITION_AT=400 OFFICIAL_PARTITION_MINUTES=6 OFFICIAL_PARTITION_RESTART=1 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=config COMMUNITY_STALL_MINUTES=2 RESTART_AT=0 LOAD_SECONDS=1500 TARGET_SEQ=2000 DEEP_ROLLBACK_EXPECTED=1
fixed: true
manual: true
# the operator path after a partition: the community nodes, restarted after the
# reconnection, hold the fork (FORK HELD: the longer chain is not vouched for by their
# confidants); each one is stopped and run once with --resolve-fork, then, if the fork
# point was only an upper bound and the node holds the fork again lower, with --set-bc-top
# at the last common block; every node must end on the official chain
- name: official-partition-restart-resolve
env: NODES=10 TRUSTED_NODES=5 INITIAL_NODES=1,2,3,4,6 OFFICIAL_PARTITION_AT=400 OFFICIAL_PARTITION_MINUTES=6 OFFICIAL_PARTITION_RESTART=1 RESOLVE_FORK_AFTER=1 OFFICIAL_FUND=200000 COMMUNITY_BOOTSTRAP=config COMMUNITY_STALL_MINUTES=2 RESTART_AT=0 LOAD_SECONDS=1500 TARGET_SEQ=2000 DEEP_ROLLBACK_EXPECTED=1
fixed: true
manual: true
# the fallback on (stall_minutes=1) in a network whose official nodes keep running, under
# packet loss and the restart test: it must never act
- name: community-bootstrap-quiet
env: COMMUNITY_BOOTSTRAP=config COMMUNITY_STALL_MINUTES=1 NET_FAULT=loss NET_FAULT_PCT=15 NET_FAULT_NODES=3,5 TARGET_SEQ=1500
manual: true
# a pruned node quick-starting from checkpoints that hold the contract states
- name: contracts-pruned-term-kill
official: 'no' # needs a feature of this fork on an even node
Expand Down Expand Up @@ -608,4 +669,8 @@ jobs:
testnet-run/resources.tsv
testnet-run/resources-summary.txt
testnet-run/sampler.log
testnet-run/partition-*.txt
testnet-run/n*/fork-held.json
testnet-run/n*/reorg-episode.json
# branch runs (test variants) are kept briefly: Actions storage is shared
retention-days: ${{ github.ref_name == 'master' && 14 || 3 }}
12 changes: 11 additions & 1 deletion ci/testnet/fund.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,12 @@ def main():
parser.add_argument("--amount", type=int, default=60000)
parser.add_argument("--low-stake-nodes", type=int, default=0)
parser.add_argument("--low-amount", type=int, default=1000)
# the even nodes' amount (0 = --amount): with official even nodes, more stake takes them the seats
parser.add_argument("--even-amount", type=int, default=0)
parser.add_argument("--load-seconds", type=int, default=0)
# the chain of node 1 is cut back during the load (run.sh RESOLVE_FORK_AFTER): a transfer it took on
# the branch that was cut may come back expired, an answer and not a fault
parser.add_argument("--chain-cut", action="store_true")
parser.add_argument("--dpos-at", type=int, default=0)
parser.add_argument("--reward-round", type=int, default=0)
parser.add_argument("--min-stake", type=int, default=0)
Expand All @@ -228,7 +233,9 @@ def main():

failures = 0
def funded(i):
return args.low_amount if i > len(nodes) - args.low_stake_nodes else args.amount
if i > len(nodes) - args.low_stake_nodes:
return args.low_amount
return args.even_amount if args.even_amount and i % 2 == 0 else args.amount

for i, node in enumerate(nodes, 1):
ok, message = send(client, master, node.pk, funded(i))
Expand Down Expand Up @@ -320,6 +327,9 @@ def delegate(name, src, dst, amount, value):
# the inner id of a transfer whose outcome was unknown: an answer, not a fault
pending += 1
print(f"load transfer rejected after an unknown outcome: {message}", flush=True)
elif not ok and args.chain_cut and "transaction is expired" in message:
pending += 1
print(f"load transfer expired after the chain was cut back: {message}", flush=True)
elif not ok and "not syncronized" in message:
# the node refuses transfers while it catches up after a restart (power-cut scenario)
unsynced += 1
Expand Down
136 changes: 136 additions & 0 deletions ci/testnet/partition.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
#!/usr/bin/env python3
"""The two chains of a partition (run.sh, OFFICIAL_PARTITION_AT) and the chain left after it.

partition.py dump --from 395 --port 9090 > community.txt # "seq hash transactions" per block
partition.py report --community community.txt --official official.txt --port 9090 --port 9102
partition.py fork --community community.txt --official official.txt # the first block that differs

dump: the blocks of one node from a sequence to its head, taken just before the reconnection.
report: where the two sides forked, and which side's blocks the nodes hold after the reconnection
(each port given, every block of both sides compared); the side whose blocks are gone was unwound,
and the transactions of its blocks above the fork with them (unless written again on the other side).
"""
import argparse
import sys

from thriftpy2.protocol import TBinaryProtocolFactory
from thriftpy2.rpc import make_client
from thriftpy2.transport import TBufferedTransportFactory

from fund import api


def client_of(port):
return make_client(api.API, host="127.0.0.1", port=int(port), proto_factory=TBinaryProtocolFactory(),
trans_factory=TBufferedTransportFactory(), timeout=60000)


def blocks(client, first, last):
"""{seq: (hash, transactions)} of the blocks first..last a node holds"""
result = {}
seq = first
while seq <= last:
count = min(100, last - seq + 1)
pools = client.PoolListGetStable(seq + count - 1, count).pools or []
for pool in pools:
if first <= pool.poolNumber <= last:
result[pool.poolNumber] = (bytes(pool.hash).hex(), pool.transactionsCount)
seq += count
return result


def head(client):
return client.SyncStateGet().lastBlock


def read(path):
result = {}
with open(path) as f:
for line in f:
seq, h, txs = line.split()
result[int(seq)] = (h, int(txs))
return result


def dump(args):
client = client_of(args.port)
last = head(client)
for seq, (h, txs) in sorted(blocks(client, args.first, last).items()):
print(seq, h, txs)


def fork_of(community, official):
"""the first sequence at which the two sides hold different blocks"""
common = sorted(set(community) & set(official))
fork = next((s for s in common if community[s][0] != official[s][0]), None)
if fork is None:
# one side may only be shorter: the fork is right above the shorter head
fork = min(max(community, default=0), max(official, default=0)) + 1
return fork


def report(args):
community = read(args.community)
official = read(args.official)
c_head = max(community) if community else 0
o_head = max(official) if official else 0
fork = fork_of(community, official)
c_side = {s: v for s, v in community.items() if s >= fork}
o_side = {s: v for s, v in official.items() if s >= fork}
print(f"PARTITION RESULT: fork at #{fork}; before the reconnection the community side held #{fork}..#{c_head} "
f"({len(c_side)} blocks, {sum(t for _, t in c_side.values())} transactions), the official side #{fork}..#{o_head} "
f"({len(o_side)} blocks, {sum(t for _, t in o_side.values())} transactions)")
failed = False
verdicts = set()
for port in args.port:
client = client_of(port)
last = max(c_head, o_head)
now = blocks(client, fork, last)
kept_c = sum(1 for s, v in c_side.items() if now.get(s, (None,))[0] == v[0])
kept_o = sum(1 for s, v in o_side.items() if now.get(s, (None,))[0] == v[0])
lost_c_txs = sum(v[1] for s, v in c_side.items() if now.get(s, (None,))[0] != v[0])
lost_o_txs = sum(v[1] for s, v in o_side.items() if now.get(s, (None,))[0] != v[0])
if not o_side and c_side and kept_c == len(c_side):
# the official side wrote nothing while cut off (it stalled): no fork, one chain
verdict = f"no fork: the official side wrote no block, the community blocks were kept ({len(c_side)} blocks)"
elif not c_side and o_side and kept_o == len(o_side):
verdict = f"no fork: the community side wrote no block, the official blocks were kept ({len(o_side)} blocks)"
elif c_side and kept_c == 0 and kept_o == len(o_side):
verdict = f"the community side was unwound ({len(c_side)} blocks, {lost_c_txs} transactions reverted)"
elif o_side and kept_o == 0 and kept_c == len(c_side):
verdict = f"the official side was unwound ({len(o_side)} blocks, {lost_o_txs} transactions reverted)"
else:
verdict = f"mixed: {kept_c}/{len(c_side)} community and {kept_o}/{len(o_side)} official blocks kept"
failed = True
verdicts.add(verdict)
print(f"PARTITION RESULT: port {port} (head #{head(client)}): {verdict}")
if len(verdicts) > 1:
print("PARTITION RESULT: the nodes compared did not keep the same side")
failed = True
sys.exit(1 if failed else 0)


def main():
parser = argparse.ArgumentParser()
sub = parser.add_subparsers(dest="command", required=True)
d = sub.add_parser("dump")
d.add_argument("--from", dest="first", type=int, required=True)
d.add_argument("--port", required=True)
r = sub.add_parser("report")
r.add_argument("--community", required=True)
r.add_argument("--official", required=True)
r.add_argument("--port", action="append", required=True)
f = sub.add_parser("fork")
f.add_argument("--community", required=True)
f.add_argument("--official", required=True)
args = parser.parse_args()
if args.command == "dump":
dump(args)
elif args.command == "fork":
print(fork_of(read(args.community), read(args.official)))
else:
report(args)


if __name__ == "__main__":
main()
Loading
Loading