From e435775305311302eeee9bcc703c3425e9298a68 Mon Sep 17 00:00:00 2001 From: EnzinoBB Date: Thu, 29 Oct 2026 23:30:43 +0100 Subject: [PATCH 1/4] p2p: the node runs the EnzinoBB/p2p fork (frame size limit, host data refresh, ordered shutdown) The p2p submodule now comes from EnzinoBB/p2p, branched from the commit the official node pins (8fb3c516): - a packet header announcing more than 256 MiB of data closes the connection before any allocation, with one error line a minute per address; - a peer's host data (version, snapshot capability) is refreshed from its own datagrams, and the full-discovery peer list shows that value; - a received datagram is passed with its own length; - ~Host stops the routing table (threads joined, UDP socket closed) after its io thread and clears the Network singleton's reference to it. ~Transport no longer clears that reference itself: ~Host does it once the io thread has ended; doing it before ~Host raced with Network::OnConnected. Co-Authored-By: Claude Opus 5.5 --- .gitmodules | 2 +- docs/CHANGES.md | 1 + docs/differences.md | 14 +++++++++++++- net/src/transport.cpp | 11 ++++------- p2p | 2 +- 5 files changed, 20 insertions(+), 10 deletions(-) diff --git a/.gitmodules b/.gitmodules index 940aa7f4c..ec80a6a89 100644 --- a/.gitmodules +++ b/.gitmodules @@ -24,7 +24,7 @@ url = https://github.com/Neargye/nameof.git [submodule "p2p"] path = p2p - url = https://github.com/SychevS/p2p.git + url = https://github.com/EnzinoBB/p2p.git [submodule "third-party/rocksdb"] path = third-party/rocksdb url = https://github.com/facebook/rocksdb.git diff --git a/docs/CHANGES.md b/docs/CHANGES.md index 667f3db0c..2085ef787 100644 --- a/docs/CHANGES.md +++ b/docs/CHANGES.md @@ -41,6 +41,7 @@ lines under the new release. |---|---| | #119 | ci: the release checks the Windows and macOS packages on mainnet before publishing them (a node of each system restores the published snapshot, syncs to the head, logs the same state digests as Linux and stops cleanly), and the same check runs weekly on the latest published release | | #120 | ci: a private network of 7 nodes (with the restart of one at block 300) runs on macOS and on Windows every night and on demand (`testnet-platforms.yml`): the nodes reach the same state digests as on Linux and a stopped node quick-starts again; Linux-only scenarios (network faults, slow nodes, full disks) stay on Linux | +| #PR | p2p: the node runs its fork of the p2p library ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the official commit `8fb3c516`): a packet header announcing more than 256 MiB closes the connection before any allocation (one error line a minute per address); peers' host data (version, snapshot capability) is refreshed from their own datagrams, so a node restarted with `serve=on` is found without leaving the routing tables first; a received datagram is passed with its own length; `~Host` stops the routing table (threads joined, UDP socket closed) after its io thread and clears the `Network` reference, which `~Transport` no longer does. ci: robustness `net-huge-frame`; `snapshot-p2p-wait` back in the suite; mixed runs build the official node with its own p2p commit | ## community.14 — 2026-10-27 diff --git a/docs/differences.md b/docs/differences.md index 98323512d..e80e13396 100644 --- a/docs/differences.md +++ b/docs/differences.md @@ -73,7 +73,10 @@ What an operator sees and sets is in [snapshots.md](snapshots.md). requests of its p2p neighbours while it is synced with the network, within `serve_rate_kbps` (KB/s, all peers together; half for one peer), `serve_max_peers` peers within 30 s, and half of that while it is a confidant or the writer. It announces it in its p2p host data at start, with - its community version. Official nodes neither see nor answer these messages. + its community version; community peers refresh that value from each datagram the node sends + them, so a restart with another `serve` value is seen within seconds (official peers keep the + value they first saw until the node leaves their routing table). Official nodes neither see nor + answer these messages. - **A first start fetches a snapshot from the network.** A node whose `db/` holds no database, with no checkpoint in `qs/`, `db_backend` `lmdb` or `rocksdb` and `fetch=auto` (the default), starts its p2p transport without opening the chain and looks for neighbours that serve @@ -180,6 +183,15 @@ consensus rules. - **Remote crashes and out-of-bounds reads.** Fixed when a peer sends a malformed compressed region, an empty packet, or a pool whose lengths point past its data. Also fixed: a data race in `LockFreeChanger`. +- **A size limit on p2p packets.** The node runs its own fork of the p2p library + ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the commit the official node uses). A + packet header may announce at most 256 MiB of data, far above any packet the node sends; + above that the connection is closed before anything is allocated, with an error line at most + once a minute per address (`Packet header from
announces bytes, above the limit + of 268435456: connection dropped (...)`). Official nodes accept any announced size. The fork + also shows a peer's latest host data (version, snapshot capability) instead of the first one + seen, passes each received datagram with its own length, and stops its routing table (threads, + UDP socket) with the p2p host at exit. - **Smart-contract solver.** Fixed: - stale contract locks; - executor connections lost between calls; diff --git a/net/src/transport.cpp b/net/src/transport.cpp index 85aa443d1..8479e59a8 100644 --- a/net/src/transport.cpp +++ b/net/src/transport.cpp @@ -121,13 +121,10 @@ Transport::~Transport() { if (processorThread_.joinable()) { processorThread_.join(); } - // p2p's Network singleton keeps a reference to the host's routing table (Host::Run). Left there, - // the routing table (its UDP socket, pinger and explorer threads) would outlive host_ and its - // io_context and be destroyed at static destruction, against the io_context's freed reactor: - // SIGSEGV in kqueue_reactor::deregister_descriptor at exit on macOS, undefined behaviour on the - // other systems. Without it the routing table goes with host_ (just below, as host_ is the last - // member), after ~Host has stopped its io_context and before the io_context is destroyed. - net::Network::Instance().SetRoutingTable(nullptr); + // host_ (the last member) goes next: ~Host stops its io_context and joins its thread, then stops + // the routing table and clears the Network singleton's reference to it (p2p fork), so the routing + // table, its UDP socket and its threads end while the io_context exists. Clearing the reference + // here, before ~Host, raced with Network::OnConnected on the p2p io thread. } void Transport::run() { diff --git a/p2p b/p2p index 8fb3c5160..c10b2b203 160000 --- a/p2p +++ b/p2p @@ -1 +1 @@ -Subproject commit 8fb3c51603386e26084e6f2f0bc851e327964840 +Subproject commit c10b2b2032152c76c94d21af15c5e921f2604745 From 7690ffab05a1b19c9c6e11f671e33589f82edb26 Mon Sep 17 00:00:00 2001 From: EnzinoBB Date: Thu, 29 Oct 2026 23:30:43 +0100 Subject: [PATCH 2/4] ci: net-huge-frame robustness scenario; snapshot-p2p-wait back in the suite - run.sh HUGE_FRAME_AT: huge_frame.py opens connections to the p2p TCP port of every node of this build and sends packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes; each node must close them at once, log the limit, keep running with its memory flat and stay in consensus. Row net-huge-frame. - snapshot-p2p-wait is no longer manual, and the capability check now expects the late serving nodes to be seen as serving (peers refresh host data). - robustness/testnet/scale: the node under comparison (node_alt_ref, the official node) is built with the p2p commit it points to, since this repository now points to its fork. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/robustness.yml | 30 ++++++-- .github/workflows/scale.yml | 17 ++++- .github/workflows/testnet.yml | 17 ++++- ci/testnet/huge_frame.py | 118 +++++++++++++++++++++++++++++++ ci/testnet/run.sh | 35 ++++++++- 5 files changed, 203 insertions(+), 14 deletions(-) create mode 100755 ci/testnet/huge_frame.py diff --git a/.github/workflows/robustness.yml b/.github/workflows/robustness.yml index 2ae54d67b..48135c7aa 100644 --- a/.github/workflows/robustness.yml +++ b/.github/workflows/robustness.yml @@ -11,7 +11,7 @@ name: Robustness # - a pruned node killed while it prunes; # - every block database, full and pruned, in one network, also with crashes and a power cut; # - packet loss, corruption, delay, reordering and a partition on some nodes, slow nodes, random -# datagrams, and all of them at once; +# datagrams, packet headers announcing huge sizes, and all of them at once; # - smart contracts across crashes, a power cut, packet loss, a pruned node killed and a node joining, # and a database conversion. # All scenarios run funded transfers and the usual restart test. @@ -140,14 +140,27 @@ jobs: git -C node-alt apply --whitespace=nowarn "$PWD/ci/official/testnet-hooks.patch" git -C node-alt diff --stat fi - # the submodules are copied from this run's checkout, which must point to the same commits + # the submodules are copied from this run's checkout, which must point to the same commits, + # except p2p: this repository runs its fork (EnzinoBB/p2p, which holds the upstream history), + # and the node under comparison gets the files of the p2p commit it points to (same submodules) for m in cscrypto p2p third-party/berkeleydb third-party/thrift third-party/thrift-interface-definitions \ third-party/googletest third-party/lmdb third-party/lmdbxx third-party/nameof third-party/rocksdb; do - if [ "$(git ls-tree HEAD -- "$m")" != "$(git -C node-alt ls-tree HEAD -- "$m")" ]; then + alt=$(git -C node-alt ls-tree HEAD -- "$m" | awk '{print $3}') + if [ "$(git ls-tree HEAD -- "$m" | awk '{print $3}')" != "$alt" ] && [ "$m" != p2p ]; then echo "$NODE_ALT_REF points $m to another commit than this run"; exit 1 fi rm -rf "node-alt/$m" cp -a "$m" "node-alt/$m" + if [ "$m" = p2p ] && [ "$(git -C p2p rev-parse HEAD)" != "$alt" ]; then + git -C p2p cat-file -e "$alt^{commit}" 2>/dev/null || git -C p2p fetch -q origin "$alt" + for s in $(git -C p2p ls-tree -r HEAD | awk '$2 == "commit" {print $4}'); do + [ "$(git -C p2p ls-tree HEAD -- "$s")" = "$(git -C p2p ls-tree "$alt" -- "$s")" ] \ + || { echo "p2p $alt points $s to another commit than this run"; exit 1; } + done + git -C p2p ls-files -s | awk '$1 != "160000" { sub(/^[^\t]*\t/, ""); print }' | (cd node-alt/p2p && xargs rm -f) + git -C p2p archive "$alt" | tar -x -C node-alt/p2p + echo "$NODE_ALT_REF: p2p at $alt (this run: $(git -C p2p rev-parse --short HEAD))" + fi done cmake -S node-alt -B build-alt -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ @@ -273,6 +286,11 @@ jobs: # random datagrams to every node's p2p port - name: garbage-datagrams env: GARBAGE_AT=150 GARBAGE_RATE=300 + # p2p packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes to every node of this build + # (the odd ones with node_alt_ref): each closes the connections at once (frame limit of the p2p + # fork), keeps running with its memory flat and stays in consensus + - name: net-huge-frame + env: HUGE_FRAME_AT=150 # all at once, on a mixed-backend network - name: chaos-mixed env: DB_BACKEND=mixed NET_FAULT=corrupt NET_FAULT_PCT=3 NET_FAULT_NODES=4,5 SLOW_NODES=6 GARBAGE_AT=200 @@ -339,11 +357,9 @@ jobs: - name: snapshot-p2p-fallback env: NODES=8 TRUSTED_NODES=7 NODE_STORAGE=lmdb,berkeleydb,lmdb:pruned,berkeleydb,rocksdb,berkeleydb,lmdb,berkeleydb,lmdb SNAPSHOT_SERVE_NODES=3 JOIN_FETCH=p2p JOIN_PRUNED_AT=2300 JOIN_FULL=1 JOIN_MIN_SOURCES=2 JOIN_WAIT=60 JOIN_FALLBACK=sync JOIN_EXPECT=fallback TARGET_SEQ=3300 fixed: true - # two serving nodes, a third one (n7) starts serving later: the joiner waits (fetch_fallback=wait). - # Manual until plan P9: peers keep the capability bit they first saw for n7 (the p2p routing - # table is not refreshed), so the joiner finds n7 only through snapshot_sources.txt (-wait-file) + # two serving nodes, a third one (n7) starts serving later: the joiner waits (fetch_fallback=wait) + # and finds n7 by the capability bit its peers refresh from n7's datagrams (p2p fork, plan P9) - name: snapshot-p2p-wait - manual: true env: NODES=8 TRUSTED_NODES=7 NODE_STORAGE=lmdb,berkeleydb,lmdb:pruned,berkeleydb,rocksdb,berkeleydb,lmdb,berkeleydb,lmdb:pruned JOIN_FETCH=p2p JOIN_PRUNED_AT=2300 SNAPSHOT_SERVE_NODES=3,5 SNAPSHOT_LATE_SERVE_NODES=7 JOIN_MIN_SOURCES=3 JOIN_WAIT=60 JOIN_FALLBACK=wait TARGET_SEQ=3300 fixed: true # the same with the serving nodes listed in the joiner's snapshot_sources.txt diff --git a/.github/workflows/scale.yml b/.github/workflows/scale.yml index 55a4260cf..07b209bae 100644 --- a/.github/workflows/scale.yml +++ b/.github/workflows/scale.yml @@ -183,14 +183,27 @@ jobs: git worktree add -q node-alt FETCH_HEAD git -C node-alt apply --whitespace=nowarn "$PWD/ci/official/testnet-hooks.patch" git -C node-alt diff --stat - # the submodules are copied from this run's checkout, which must point to the same commits + # the submodules are copied from this run's checkout, which must point to the same commits, + # except p2p: this repository runs its fork (EnzinoBB/p2p, which holds the upstream history), + # and the node under comparison gets the files of the p2p commit it points to (same submodules) for m in cscrypto p2p third-party/berkeleydb third-party/thrift third-party/thrift-interface-definitions \ third-party/googletest third-party/lmdb third-party/lmdbxx third-party/nameof third-party/rocksdb; do - if [ "$(git ls-tree HEAD -- "$m")" != "$(git -C node-alt ls-tree HEAD -- "$m")" ]; then + alt=$(git -C node-alt ls-tree HEAD -- "$m" | awk '{print $3}') + if [ "$(git ls-tree HEAD -- "$m" | awk '{print $3}')" != "$alt" ] && [ "$m" != p2p ]; then echo "the official node points $m to another commit than this run"; exit 1 fi rm -rf "node-alt/$m" cp -a "$m" "node-alt/$m" + if [ "$m" = p2p ] && [ "$(git -C p2p rev-parse HEAD)" != "$alt" ]; then + git -C p2p cat-file -e "$alt^{commit}" 2>/dev/null || git -C p2p fetch -q origin "$alt" + for s in $(git -C p2p ls-tree -r HEAD | awk '$2 == "commit" {print $4}'); do + [ "$(git -C p2p ls-tree HEAD -- "$s")" = "$(git -C p2p ls-tree "$alt" -- "$s")" ] \ + || { echo "p2p $alt points $s to another commit than this run"; exit 1; } + done + git -C p2p ls-files -s | awk '$1 != "160000" { sub(/^[^\t]*\t/, ""); print }' | (cd node-alt/p2p && xargs rm -f) + git -C p2p archive "$alt" | tar -x -C node-alt/p2p + echo "the official node: p2p at $alt (this run: $(git -C p2p rev-parse --short HEAD))" + fi done cmake -S node-alt -B build-alt -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ diff --git a/.github/workflows/testnet.yml b/.github/workflows/testnet.yml index 9d283a829..cbabecb82 100644 --- a/.github/workflows/testnet.yml +++ b/.github/workflows/testnet.yml @@ -178,14 +178,27 @@ jobs: git -C node-alt apply --whitespace=nowarn "$PWD/ci/official/testnet-hooks.patch" git -C node-alt diff --stat fi - # the submodules are copied from this run's checkout, which must point to the same commits + # the submodules are copied from this run's checkout, which must point to the same commits, + # except p2p: this repository runs its fork (EnzinoBB/p2p, which holds the upstream history), + # and the node under comparison gets the files of the p2p commit it points to (same submodules) for m in cscrypto p2p third-party/berkeleydb third-party/thrift third-party/thrift-interface-definitions \ third-party/googletest third-party/lmdb third-party/lmdbxx third-party/nameof third-party/rocksdb; do - if [ "$(git ls-tree HEAD -- "$m")" != "$(git -C node-alt ls-tree HEAD -- "$m")" ]; then + alt=$(git -C node-alt ls-tree HEAD -- "$m" | awk '{print $3}') + if [ "$(git ls-tree HEAD -- "$m" | awk '{print $3}')" != "$alt" ] && [ "$m" != p2p ]; then echo "$NODE_ALT_REF points $m to another commit than this run"; exit 1 fi rm -rf "node-alt/$m" cp -a "$m" "node-alt/$m" + if [ "$m" = p2p ] && [ "$(git -C p2p rev-parse HEAD)" != "$alt" ]; then + git -C p2p cat-file -e "$alt^{commit}" 2>/dev/null || git -C p2p fetch -q origin "$alt" + for s in $(git -C p2p ls-tree -r HEAD | awk '$2 == "commit" {print $4}'); do + [ "$(git -C p2p ls-tree HEAD -- "$s")" = "$(git -C p2p ls-tree "$alt" -- "$s")" ] \ + || { echo "p2p $alt points $s to another commit than this run"; exit 1; } + done + git -C p2p ls-files -s | awk '$1 != "160000" { sub(/^[^\t]*\t/, ""); print }' | (cd node-alt/p2p && xargs rm -f) + git -C p2p archive "$alt" | tar -x -C node-alt/p2p + echo "$NODE_ALT_REF: p2p at $alt (this run: $(git -C p2p rev-parse --short HEAD))" + fi done cmake -S node-alt -B build-alt -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ diff --git a/ci/testnet/huge_frame.py b/ci/testnet/huge_frame.py new file mode 100755 index 000000000..f2d7b69ac --- /dev/null +++ b/ci/testnet/huge_frame.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +"""Robustness test client (scenario net-huge-frame): opens connections to a node's p2p TCP port and +sends on each one packet header announcing a data size above the p2p frame limit, then checks that +the node closes every connection and that its memory stays flat. + +The p2p packet header is 77 bytes, little-endian: type (1 byte), data_size (8 bytes), sender id +(32 bytes), receiver id (32 bytes), reserved (4 bytes). A node with the frame limit (256 MiB) +closes the connection at once, before reading or allocating anything for the data. + +Usage: huge_frame.py HOST PORT [--size N ...] [--connections N] [--timeout S] [--pid PID] + --size announced data size, repeatable (default 2^40) + --connections connections per size, all held open together (default 4) + --timeout seconds the node has to close them (default 5; the p2p idle timeout is 10) + --pid the node's pid: its resident memory (Linux /proc) is sampled during the test and + may not grow by more than --max-growth MiB (default 64) +Exit status: 0 when the node closed every connection in time (and its memory stayed flat), 1 otherwise. +""" +import argparse +import os +import select +import socket +import struct +import sys +import time + +REGISTRATION = 2 # the first packet of a connection must be a registration + + +def header(data_size): + return struct.pack(" max_rss): + max_rss = r + still_open = len(socks) + for s in socks: + s.close() + return closed, still_open, errors, max_rss + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("host") + ap.add_argument("port", type=int) + ap.add_argument("--size", type=int, action="append") + ap.add_argument("--connections", type=int, default=4) + ap.add_argument("--timeout", type=float, default=5.0) + ap.add_argument("--pid", type=int) + ap.add_argument("--max-growth", type=int, default=64, help="MiB") + a = ap.parse_args() + sizes = a.size or [1 << 40] + + before = rss_kib(a.pid) if a.pid else None + ok = True + peak = before + for size in sizes: + closed, still_open, errors, max_rss = run(a.host, a.port, size, a.connections, a.timeout, a.pid) + print(f"huge-frame: {a.host}:{a.port}: {closed} of {a.connections} connection(s) announcing " + f"{size} bytes closed by the node, {still_open} still open after {a.timeout:g} s, {errors} error(s)") + if closed != a.connections: + ok = False + if max_rss is not None and (peak is None or max_rss > peak): + peak = max_rss + if a.pid: + if before is None or peak is None: + print(f"huge-frame: no memory reading for pid {a.pid}") + ok = False + else: + growth = (peak - before) / 1024 + print(f"huge-frame: node memory {before / 1024:.0f} MiB before, peak {peak / 1024:.0f} MiB " + f"during the test (+{growth:.0f} MiB, limit {a.max_growth} MiB)") + if growth > a.max_growth: + ok = False + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ci/testnet/run.sh b/ci/testnet/run.sh index 578b592f2..a048cecc0 100755 --- a/ci/testnet/run.sh +++ b/ci/testnet/run.sh @@ -71,6 +71,11 @@ # SLOW_NODES (comma separated, default none): those nodes get SLOW_CPU percent of one CPU (default 20) # GARBAGE_AT (default 0 = off): from that sequence garbage.py sends GARBAGE_RATE (default 200) # random datagrams a second to the nodes' p2p ports for GARBAGE_SECONDS (default 300) +# HUGE_FRAME_AT (default 0 = off): from that sequence huge_frame.py opens connections to the p2p +# TCP port of every node of this build (the odd ones when the even nodes run BIN_ALT, which may +# have no frame limit) and sends packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes: each +# node must close every connection at once, log the limit, keep running with its memory flat +# (Linux: at most +64 MiB) and stay in consensus # PRUNE_NODE (default 0 = off): that node runs with pruned storage on LMDB (a checkpoint every # 1'000 blocks, prune_keep_blocks=300, block_db=lmdb); it must remove old blocks, survive the restart test if it is # the restarted node, and end with the same state digest @@ -352,6 +357,7 @@ SLOW_CPU=${SLOW_CPU:-20} GARBAGE_AT=${GARBAGE_AT:-0} GARBAGE_SECONDS=${GARBAGE_SECONDS:-300} GARBAGE_RATE=${GARBAGE_RATE:-200} +HUGE_FRAME_AT=${HUGE_FRAME_AT:-0} for v in "$NET_FAULT_NODES" "$SLOW_NODES"; do echo "$v" | grep -qE '^([0-9]+(,[0-9]+)*)?$' || { echo "bad node list '$v'"; exit 1; } done @@ -840,6 +846,7 @@ net_fault_done=0 snapshot_done=0 storage_switched=0 GARBAGE_PID= +huge_frame_done=0 if [ -n "$SLOW_NODES" ]; then echo "+cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control > /dev/null 2>&1 || true fi @@ -1028,6 +1035,26 @@ while :; do python3 "$HERE/garbage.py" "$BASE_PORT" "$ALL_NODES" "$GARBAGE_SECONDS" "$GARBAGE_RATE" > "$WORK/garbage.log" 2>&1 & GARBAGE_PID=$! fi + if [ "$HUGE_FRAME_AT" -gt 0 ] && [ "$huge_frame_done" -eq 0 ] && [ "$min" -ge "$HUGE_FRAME_AT" ]; then + huge_frame_done=1 + echo "huge frame: packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes to the p2p port of every node of this build, from sequence $min" + for i in $(seq 1 "$NODES"); do + # the even nodes of BIN_ALT (the official node, another ref) may have no frame limit + [ -n "$BIN_ALT" ] && [ $((i % 2)) -eq 0 ] && continue + pid_arg="" + [ "$OS_KIND" != linux ] || pid_arg="--pid ${PIDS[$i]}" + hf_status=0 + python3 "$HERE/huge_frame.py" "$(node_ip "$i")" $((BASE_PORT + i)) --size $((1 << 40)) --size $((1 << 30)) \ + --size $(( (256 << 20) + 1 )) --connections 4 $pid_arg > "$WORK/huge-frame-n$i.log" 2>&1 || hf_status=$? + sed "s/^/huge frame: n$i: /" "$WORK/huge-frame-n$i.log" + [ "$hf_status" -eq 0 ] || { echo "FAIL: n$i did not close every connection announcing a huge frame, or its memory grew"; exit 1; } + sleep 1 + kill -0 "${PIDS[$i]}" 2>/dev/null || { echo "FAIL: n$i stopped after the huge frame headers"; exit 1; } + grep -aq 'announces .* bytes, above the limit of 268435456' "$WORK/n$i"/*.log \ + || { echo "FAIL: n$i logged no frame limit line"; exit 1; } + echo "huge frame: n$i: $(grep -ah 'announces .* bytes, above the limit' "$WORK/n$i"/*.log | awk 'NR == 1' | sed 's/.*ERROR [^]]*\] //')" + done + fi if [ "$SNAPSHOT_AT" -gt 0 ] && [ "$snapshot_done" -eq 0 ] && [ "$min" -ge "$SNAPSHOT_AT" ]; then snapshot_done=1 @@ -1895,8 +1922,8 @@ if [ -n "$SNAPSHOT_SERVE_NODES" ]; then serving="" late="" for i in ${SNAPSHOT_SERVE_NODES//,/ }; do - # a late serving node announced the capability only at its restart, and peers keep the value - # they first saw (the routing table is not refreshed, design 4.1, plan P9): either value is fine + # a late serving node announced the capability only at its restart: peers refresh the value from + # its own datagrams (p2p fork, plan P9), so it must be seen too (listed apart in the output) if [[ ",$SNAPSHOT_LATE_SERVE_NODES," == *",$i,"* ]]; then late="$late,${KEYS[$((i - 1))]}"; else serving="$serving ${KEYS[$((i - 1))]}"; fi done python3 - "$WORK/n$observer/known_peers.json" "$late" $serving <<'PY' || { echo "FAIL: snapshot capability not announced as expected"; exit 1; } @@ -1905,11 +1932,13 @@ peers = {p["id"]: p for p in json.load(open(sys.argv[1]))} late = set(k for k in sys.argv[2].split(",") if k) serving = set(sys.argv[3:]) bad = [k for k in serving if peers.get(k, {}).get("capabilities") != 1] +late_bad = [k for k in late if peers.get(k, {}).get("capabilities") != 1] extra = [k for k, p in peers.items() if k not in serving and k not in late and p["capabilities"] != 0] community = sum(1 for p in peers.values() if p["minor"] > 0) print(f"capabilities: {len(peers)} known peers, {len(serving) - len(bad)} of {len(serving)} serving nodes announce snapshots, " + f"{len(late) - len(late_bad)} of {len(late)} late serving nodes do since their restart, " f"{len(extra)} others do, {community} announce a community version") -sys.exit(1 if bad or extra else 0) +sys.exit(1 if bad or late_bad or extra else 0) PY PIDS[$observer]="" fi From a9cce1cc2c15cb8532d5ad50fe87bf6ab36c9f38 Mon Sep 17 00:00:00 2001 From: EnzinoBB Date: Thu, 29 Oct 2026 23:32:22 +0100 Subject: [PATCH 3/4] docs: CHANGES line of #121 Co-Authored-By: Claude Opus 5.5 --- docs/CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CHANGES.md b/docs/CHANGES.md index 2085ef787..2f6b8ff77 100644 --- a/docs/CHANGES.md +++ b/docs/CHANGES.md @@ -41,7 +41,7 @@ lines under the new release. |---|---| | #119 | ci: the release checks the Windows and macOS packages on mainnet before publishing them (a node of each system restores the published snapshot, syncs to the head, logs the same state digests as Linux and stops cleanly), and the same check runs weekly on the latest published release | | #120 | ci: a private network of 7 nodes (with the restart of one at block 300) runs on macOS and on Windows every night and on demand (`testnet-platforms.yml`): the nodes reach the same state digests as on Linux and a stopped node quick-starts again; Linux-only scenarios (network faults, slow nodes, full disks) stay on Linux | -| #PR | p2p: the node runs its fork of the p2p library ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the official commit `8fb3c516`): a packet header announcing more than 256 MiB closes the connection before any allocation (one error line a minute per address); peers' host data (version, snapshot capability) is refreshed from their own datagrams, so a node restarted with `serve=on` is found without leaving the routing tables first; a received datagram is passed with its own length; `~Host` stops the routing table (threads joined, UDP socket closed) after its io thread and clears the `Network` reference, which `~Transport` no longer does. ci: robustness `net-huge-frame`; `snapshot-p2p-wait` back in the suite; mixed runs build the official node with its own p2p commit | +| #121 | p2p: the node runs its fork of the p2p library ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the official commit `8fb3c516`): a packet header announcing more than 256 MiB closes the connection before any allocation (one error line a minute per address); peers' host data (version, snapshot capability) is refreshed from their own datagrams, so a node restarted with `serve=on` is found without leaving the routing tables first; a received datagram is passed with its own length; `~Host` stops the routing table (threads joined, UDP socket closed) after its io thread and clears the `Network` reference, which `~Transport` no longer does. ci: robustness `net-huge-frame`; `snapshot-p2p-wait` back in the suite; mixed runs build the official node with its own p2p commit | ## community.14 — 2026-10-27 From 02a3dd24d37252c65a60b4bc07cf4668015572e8 Mon Sep 17 00:00:00 2001 From: EnzinoBB Date: Fri, 30 Oct 2026 01:20:16 +0100 Subject: [PATCH 4/4] p2p fork review fixes: registration limit, chunked reads, refresh from ping answers - p2p at 09252ec: a registration may announce at most 64 bytes, later packets 256 MiB, read in 1 MiB chunks so memory follows the bytes received; host data refreshed only from answers to our pings; a zero k-bucket value does not hide the explorer's; RoutingTable::Stop waits for the routing-table notification threads (at most 5 s). - net-huge-frame: oversized headers before and after a registration (closed at once, both limit lines logged) and just-below-limit headers without data (64 bytes, 256 MiB - 1: held until the idle timeout), memory checked each step. - docs: differences.md and CHANGES.md follow. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/robustness.yml | 7 +- ci/testnet/huge_frame.py | 139 +++++++++++++++++++------------ ci/testnet/run.sh | 32 ++++--- docs/CHANGES.md | 2 +- docs/differences.md | 23 ++--- p2p | 2 +- 6 files changed, 128 insertions(+), 77 deletions(-) diff --git a/.github/workflows/robustness.yml b/.github/workflows/robustness.yml index 48135c7aa..14891eb60 100644 --- a/.github/workflows/robustness.yml +++ b/.github/workflows/robustness.yml @@ -286,9 +286,10 @@ jobs: # random datagrams to every node's p2p port - name: garbage-datagrams env: GARBAGE_AT=150 GARBAGE_RATE=300 - # p2p packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes to every node of this build - # (the odd ones with node_alt_ref): each closes the connections at once (frame limit of the p2p - # fork), keeps running with its memory flat and stays in consensus + # p2p packet headers above the limits of the p2p fork (64 bytes before registration, 256 MiB + # after) and just below them, to every node of this build (the odd ones with node_alt_ref): + # each closes the oversized ones at once, holds almost no memory for the others, keeps running + # and stays in consensus - name: net-huge-frame env: HUGE_FRAME_AT=150 # all at once, on a mixed-backend network diff --git a/ci/testnet/huge_frame.py b/ci/testnet/huge_frame.py index f2d7b69ac..718d07387 100755 --- a/ci/testnet/huge_frame.py +++ b/ci/testnet/huge_frame.py @@ -1,19 +1,26 @@ #!/usr/bin/env python3 """Robustness test client (scenario net-huge-frame): opens connections to a node's p2p TCP port and -sends on each one packet header announcing a data size above the p2p frame limit, then checks that -the node closes every connection and that its memory stays flat. +sends on each one a packet header announcing a large data size, then checks how the node treats them +and that its memory stays flat. The p2p packet header is 77 bytes, little-endian: type (1 byte), data_size (8 bytes), sender id -(32 bytes), receiver id (32 bytes), reserved (4 bytes). A node with the frame limit (256 MiB) -closes the connection at once, before reading or allocating anything for the data. - -Usage: huge_frame.py HOST PORT [--size N ...] [--connections N] [--timeout S] [--pid PID] - --size announced data size, repeatable (default 2^40) - --connections connections per size, all held open together (default 4) - --timeout seconds the node has to close them (default 5; the p2p idle timeout is 10) - --pid the node's pid: its resident memory (Linux /proc) is sampled during the test and - may not grow by more than --max-growth MiB (default 64) -Exit status: 0 when the node closed every connection in time (and its memory stayed flat), 1 otherwise. +(32 bytes), receiver id (32 bytes), reserved (4 bytes). The first packet of a connection must be a +registration (its data: address type, an IPv4 address and a port, 7 bytes). + +Limits of the p2p fork: a registration may announce at most 64 bytes, any later packet at most +256 MiB; above that the node closes the connection at once. Below the limit it reads the data in +chunks of 1 MiB, so the memory it holds follows the bytes actually received, not the size announced. + +Usage: huge_frame.py HOST PORT [options] + --size N announced data size, repeatable (default 2^40) + --registered send a valid registration first, then the header (a direct packet) + --expect close the node must close each connection within --timeout seconds (default) + --expect hold the node must keep each connection open for --timeout seconds (it waits for the + data), then close it on its idle timeout (10 s) within --idle seconds + --connections N connections per size, all held open together (default 4) + --pid PID the node's pid: its resident memory (Linux /proc) is sampled during the test and + may not grow by more than --max-growth MiB (default 64) +Exit status: 0 when the node behaved as expected (and its memory stayed flat), 1 otherwise. """ import argparse import os @@ -23,11 +30,17 @@ import sys import time -REGISTRATION = 2 # the first packet of a connection must be a registration +DIRECT = 0 +REGISTRATION = 2 + +def header(packet_type, data_size, receiver=None): + return struct.pack(" self.value): + self.value = r + + +def wait_closed(socks, seconds, peak): + """Reads from socks until each is closed by the node or `seconds` pass; returns the closed count.""" + closed = 0 + deadline = time.monotonic() + seconds while socks and time.monotonic() < deadline: readable, _, _ = select.select(socks, [], [], 0.1) for s in readable: try: - done = s.recv(4096) == b"" + done = s.recv(65536) == b"" except (ConnectionResetError, BrokenPipeError): done = True except BlockingIOError: @@ -68,14 +83,39 @@ def run(host, port, size, count, timeout, pid): closed += 1 socks.remove(s) s.close() - if pid: - r = rss_kib(pid) - if r is not None and (max_rss is None or r > max_rss): - max_rss = r - still_open = len(socks) + peak.sample() + return closed + + +def run(a, size, peak): + socks, errors = [], 0 + for _ in range(a.connections): + try: + s = socket.create_connection((a.host, a.port), timeout=5) + payload = header(REGISTRATION, size) + if a.registered: + payload = registration() + header(DIRECT, size) + s.sendall(payload) + s.setblocking(False) + socks.append(s) + except OSError as e: + print(f"huge-frame: connect/send to {a.host}:{a.port}: {e}") + errors += 1 + what = f"{len(socks)} connection(s) announcing {size} bytes{' after a registration' if a.registered else ''}" + ok = errors == 0 + if a.expect == "close": + closed = wait_closed(socks, a.timeout, peak) + print(f"huge-frame: {a.host}:{a.port}: {what}: {closed} closed by the node within {a.timeout:g} s") + ok = ok and closed == a.connections + else: + early = wait_closed(socks, a.timeout, peak) + late = wait_closed(socks, a.idle, peak) + print(f"huge-frame: {a.host}:{a.port}: {what}: {early} closed within {a.timeout:g} s (expected 0, " + f"the node waits for the data), {late} closed on the idle timeout within {a.idle:g} s more") + ok = ok and early == 0 and late == a.connections for s in socks: s.close() - return closed, still_open, errors, max_rss + return ok def main(): @@ -83,34 +123,29 @@ def main(): ap.add_argument("host") ap.add_argument("port", type=int) ap.add_argument("--size", type=int, action="append") + ap.add_argument("--registered", action="store_true") + ap.add_argument("--expect", choices=("close", "hold"), default="close") ap.add_argument("--connections", type=int, default=4) ap.add_argument("--timeout", type=float, default=5.0) + ap.add_argument("--idle", type=float, default=15.0) ap.add_argument("--pid", type=int) ap.add_argument("--max-growth", type=int, default=64, help="MiB") a = ap.parse_args() - sizes = a.size or [1 << 40] - before = rss_kib(a.pid) if a.pid else None + peak = Peak(a.pid) + before = peak.value ok = True - peak = before - for size in sizes: - closed, still_open, errors, max_rss = run(a.host, a.port, size, a.connections, a.timeout, a.pid) - print(f"huge-frame: {a.host}:{a.port}: {closed} of {a.connections} connection(s) announcing " - f"{size} bytes closed by the node, {still_open} still open after {a.timeout:g} s, {errors} error(s)") - if closed != a.connections: - ok = False - if max_rss is not None and (peak is None or max_rss > peak): - peak = max_rss + for size in a.size or [1 << 40]: + ok = run(a, size, peak) and ok if a.pid: - if before is None or peak is None: + if before is None or peak.value is None: print(f"huge-frame: no memory reading for pid {a.pid}") ok = False else: - growth = (peak - before) / 1024 - print(f"huge-frame: node memory {before / 1024:.0f} MiB before, peak {peak / 1024:.0f} MiB " + growth = (peak.value - before) / 1024 + print(f"huge-frame: node memory {before / 1024:.0f} MiB before, peak {peak.value / 1024:.0f} MiB " f"during the test (+{growth:.0f} MiB, limit {a.max_growth} MiB)") - if growth > a.max_growth: - ok = False + ok = ok and growth <= a.max_growth return 0 if ok else 1 diff --git a/ci/testnet/run.sh b/ci/testnet/run.sh index a048cecc0..29c91b844 100755 --- a/ci/testnet/run.sh +++ b/ci/testnet/run.sh @@ -73,9 +73,10 @@ # random datagrams a second to the nodes' p2p ports for GARBAGE_SECONDS (default 300) # HUGE_FRAME_AT (default 0 = off): from that sequence huge_frame.py opens connections to the p2p # TCP port of every node of this build (the odd ones when the even nodes run BIN_ALT, which may -# have no frame limit) and sends packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes: each -# node must close every connection at once, log the limit, keep running with its memory flat -# (Linux: at most +64 MiB) and stay in consensus +# have no frame limit) and sends packet headers above the limits (2^40 and 65 bytes before a +# registration, 2^40 and 256 MiB + 1 after one: closed at once, a limit line logged) and just +# below them without data (64 bytes, 256 MiB - 1: held until the 10 s idle timeout); each node +# must keep running with its memory flat (Linux: at most +64 MiB per step) and stay in consensus # PRUNE_NODE (default 0 = off): that node runs with pruned storage on LMDB (a checkpoint every # 1'000 blocks, prune_keep_blocks=300, block_db=lmdb); it must remove old blocks, survive the restart test if it is # the restarted node, and end with the same state digest @@ -1037,22 +1038,33 @@ while :; do fi if [ "$HUGE_FRAME_AT" -gt 0 ] && [ "$huge_frame_done" -eq 0 ] && [ "$min" -ge "$HUGE_FRAME_AT" ]; then huge_frame_done=1 - echo "huge frame: packet headers announcing 2^40, 2^30 and 256 MiB + 1 bytes to the p2p port of every node of this build, from sequence $min" + echo "huge frame: oversized and just-below-limit packet headers to the p2p port of every node of this build, from sequence $min" for i in $(seq 1 "$NODES"); do # the even nodes of BIN_ALT (the official node, another ref) may have no frame limit [ -n "$BIN_ALT" ] && [ $((i % 2)) -eq 0 ] && continue pid_arg="" [ "$OS_KIND" != linux ] || pid_arg="--pid ${PIDS[$i]}" + hf="python3 $HERE/huge_frame.py $(node_ip "$i") $((BASE_PORT + i)) --connections 4 $pid_arg" hf_status=0 - python3 "$HERE/huge_frame.py" "$(node_ip "$i")" $((BASE_PORT + i)) --size $((1 << 40)) --size $((1 << 30)) \ - --size $(( (256 << 20) + 1 )) --connections 4 $pid_arg > "$WORK/huge-frame-n$i.log" 2>&1 || hf_status=$? + { + # before registration: at most 64 bytes, closed at once + $hf --size $((1 << 40)) --size 65 || hf_status=$? + # after a registration: at most 256 MiB, closed at once + $hf --registered --size $((1 << 40)) --size $(( (256 << 20) + 1 )) || hf_status=$? + # just below the limits, no data sent: the node waits for the data (reading it in 1 MiB + # chunks, so it holds almost nothing) and closes on its 10 s idle timeout + $hf --expect hold --size 64 || hf_status=$? + $hf --registered --expect hold --size $(( (256 << 20) - 1 )) || hf_status=$? + } > "$WORK/huge-frame-n$i.log" 2>&1 sed "s/^/huge frame: n$i: /" "$WORK/huge-frame-n$i.log" - [ "$hf_status" -eq 0 ] || { echo "FAIL: n$i did not close every connection announcing a huge frame, or its memory grew"; exit 1; } + [ "$hf_status" -eq 0 ] || { echo "FAIL: n$i did not treat the oversized or just-below-limit headers as expected, or its memory grew"; exit 1; } sleep 1 kill -0 "${PIDS[$i]}" 2>/dev/null || { echo "FAIL: n$i stopped after the huge frame headers"; exit 1; } - grep -aq 'announces .* bytes, above the limit of 268435456' "$WORK/n$i"/*.log \ - || { echo "FAIL: n$i logged no frame limit line"; exit 1; } - echo "huge frame: n$i: $(grep -ah 'announces .* bytes, above the limit' "$WORK/n$i"/*.log | awk 'NR == 1' | sed 's/.*ERROR [^]]*\] //')" + for lim in 64 268435456; do + grep -aq "announces .* bytes, above the limit of $lim:" "$WORK/n$i"/*.log \ + || { echo "FAIL: n$i logged no limit line for $lim bytes"; exit 1; } + done + grep -ah 'announces .* bytes, above the limit' "$WORK/n$i"/*.log | awk 'NR <= 2' | sed "s/.*ERROR [^]]*\] /huge frame: n$i: /" done fi diff --git a/docs/CHANGES.md b/docs/CHANGES.md index 2f6b8ff77..7f4927087 100644 --- a/docs/CHANGES.md +++ b/docs/CHANGES.md @@ -41,7 +41,7 @@ lines under the new release. |---|---| | #119 | ci: the release checks the Windows and macOS packages on mainnet before publishing them (a node of each system restores the published snapshot, syncs to the head, logs the same state digests as Linux and stops cleanly), and the same check runs weekly on the latest published release | | #120 | ci: a private network of 7 nodes (with the restart of one at block 300) runs on macOS and on Windows every night and on demand (`testnet-platforms.yml`): the nodes reach the same state digests as on Linux and a stopped node quick-starts again; Linux-only scenarios (network faults, slow nodes, full disks) stay on Linux | -| #121 | p2p: the node runs its fork of the p2p library ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the official commit `8fb3c516`): a packet header announcing more than 256 MiB closes the connection before any allocation (one error line a minute per address); peers' host data (version, snapshot capability) is refreshed from their own datagrams, so a node restarted with `serve=on` is found without leaving the routing tables first; a received datagram is passed with its own length; `~Host` stops the routing table (threads joined, UDP socket closed) after its io thread and clears the `Network` reference, which `~Transport` no longer does. ci: robustness `net-huge-frame`; `snapshot-p2p-wait` back in the suite; mixed runs build the official node with its own p2p commit | +| #121 | p2p: the node runs its fork of the p2p library ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the official commit `8fb3c516`): a registration announcing more than 64 bytes or a later packet header announcing more than 256 MiB closes the connection before any allocation (one error line a minute per address and limit), and packet data is read in 1 MiB chunks so memory follows the bytes received; peers' host data (version, snapshot capability) is refreshed from their answers to our pings, so a node restarted with `serve=on` is found without leaving the routing tables first; a received datagram is passed with its own length; `~Host` stops the routing table (threads joined, UDP socket closed, routing-table notifications waited for) after its io thread and clears the `Network` reference, which `~Transport` no longer does. ci: robustness `net-huge-frame`; `snapshot-p2p-wait` back in the suite; mixed runs build the official node with its own p2p commit | ## community.14 — 2026-10-27 diff --git a/docs/differences.md b/docs/differences.md index e80e13396..f09d649d8 100644 --- a/docs/differences.md +++ b/docs/differences.md @@ -73,8 +73,8 @@ What an operator sees and sets is in [snapshots.md](snapshots.md). requests of its p2p neighbours while it is synced with the network, within `serve_rate_kbps` (KB/s, all peers together; half for one peer), `serve_max_peers` peers within 30 s, and half of that while it is a confidant or the writer. It announces it in its p2p host data at start, with - its community version; community peers refresh that value from each datagram the node sends - them, so a restart with another `serve` value is seen within seconds (official peers keep the + its community version; community peers refresh that value from the node's answers to their + pings, so a restart with another `serve` value is seen within about a minute (official peers keep the value they first saw until the node leaves their routing table). Official nodes neither see nor answer these messages. - **A first start fetches a snapshot from the network.** A node whose `db/` holds no database, @@ -184,14 +184,17 @@ consensus rules. region, an empty packet, or a pool whose lengths point past its data. Also fixed: a data race in `LockFreeChanger`. - **A size limit on p2p packets.** The node runs its own fork of the p2p library - ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the commit the official node uses). A - packet header may announce at most 256 MiB of data, far above any packet the node sends; - above that the connection is closed before anything is allocated, with an error line at most - once a minute per address (`Packet header from
announces bytes, above the limit - of 268435456: connection dropped (...)`). Official nodes accept any announced size. The fork - also shows a peer's latest host data (version, snapshot capability) instead of the first one - seen, passes each received datagram with its own length, and stops its routing table (threads, - UDP socket) with the p2p host at exit. + ([EnzinoBB/p2p](https://github.com/EnzinoBB/p2p), from the commit the official node uses). The + first packet of a connection (the registration) may announce at most 64 bytes of data, any + later packet at most 256 MiB, far above any packet the node sends; above that the connection is + closed before anything is allocated, with an error line at most once a minute per address + (`Packet header from
announces bytes, above the limit of : connection + dropped (...)`). Below the limit the data is read in chunks of 1 MiB, so the memory held follows + the bytes actually received, not the size announced. Official nodes allocate whatever size is + announced. The fork also shows a peer's latest host data (version, snapshot capability), + refreshed from its answers to our pings, instead of the first one seen; passes each received + datagram with its own length; and stops its routing table (threads, UDP socket) with the p2p + host at exit. - **Smart-contract solver.** Fixed: - stale contract locks; - executor connections lost between calls; diff --git a/p2p b/p2p index c10b2b203..09252ec9a 160000 --- a/p2p +++ b/p2p @@ -1 +1 @@ -Subproject commit c10b2b2032152c76c94d21af15c5e921f2604745 +Subproject commit 09252ec9a5db814016f7b83b6ac54279d4a7fe57