Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/robustness.yml
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,11 @@ jobs:
official: 'no' # needs a feature of this fork on an even node
env: NODE_STORAGE=lmdb,lmdb,lmdb:pruned,lmdb:pruned,lmdb:pruned,lmdb:pruned,lmdb:pruned,lmdb JOIN_PRUNED_AT=2300 JOIN_FULL=1 SYNCRO_ROUND=100 TARGET_SEQ=3000
fixed: true
# the same with node_alt_ref=official: the even nodes, which answer the joiner (n9, this build)
# with the low blocks, are official responders with their anti-duplicate log (special order 35)
- name: pruned-peers-join-official
env: NODES=8 TRUSTED_NODES=7 NODE_STORAGE=lmdb,berkeleydb,lmdb:pruned,berkeleydb,lmdb:pruned,berkeleydb,lmdb:pruned,berkeleydb,lmdb JOIN_PRUNED_AT=2300 JOIN_FULL=1 SYNCRO_ROUND=100 TARGET_SEQ=3000
fixed: true
# kills that land while the node prunes, or right after a checkpoint it prunes from
- name: rocksdb-pruned-crash
official: 'no' # needs a feature of this fork on an even node
Expand Down
2 changes: 2 additions & 0 deletions csnode/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ add_library(csnode
include/csnode/walletsids_serializer.hpp
include/csnode/blockhashes.hpp
include/csnode/poolsynchronizer.hpp
include/csnode/syncropacing.hpp
include/csnode/fee.hpp
include/csnode/transactionsvalidator.hpp
include/csnode/walletsstate.hpp
Expand Down Expand Up @@ -104,6 +105,7 @@ add_library(csnode
src/walletsids_serializer.cpp
src/blockhashes.cpp
src/poolsynchronizer.cpp
src/syncropacing.cpp
src/fee.cpp
src/transactionsvalidator.cpp
src/transactionsindex.cpp
Expand Down
5 changes: 5 additions & 0 deletions csnode/include/csnode/poolsynchronizer.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

#include <csnode/blockchain.hpp>
#include <csnode/nodecore.hpp>
#include <csnode/syncropacing.hpp>

#include <lib/system/common.hpp>
#include <lib/system/timer.hpp>
Expand Down Expand Up @@ -60,6 +61,9 @@ private slots:
void updateSynchroLog();
bool removeSynchroLog(const cs::PublicKey& sender);
bool checkSynchroLog(const cs::PublicKey& sender);
// the request log is kept for this peer: special order 35 applied here, or the peer sent a
// SyncroMsg (its own anti-duplicate log is active)
bool synchroLogFor(const cs::PublicKey& sender) const;

bool showSyncronizationProgress(Sequence lastWrittenSequence) const;
void manageSyncBlocks(cs::PoolsBlock&& poolsBlock);
Expand All @@ -81,6 +85,7 @@ private slots:
std::unordered_map<PublicKey, Sequence> neighbours_;

std::map<cs::PublicKey, std::tuple<cs::PoolsRequestedSequences, SyncroMessage, uint64_t>> synchroLog_;
SyncroPacing pacing_;
Timer timer_;
};
} // namespace cs
Expand Down
62 changes: 62 additions & 0 deletions csnode/include/csnode/syncropacing.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#ifndef SYNCROPACING_HPP
#define SYNCROPACING_HPP

#include <cstdint>
#include <map>

#include <csnode/nodecore.hpp>
#include <lib/system/common.hpp>

namespace cs {

// Forward sync against responders whose anti-duplicate log is active (special order 35): such a
// responder refuses a request whose last sequence is not above the last one it logged for the same
// requester, while that entry lives (5 s after its reply), and for ever when it accepted a request
// (AwaitAnswer) and then had nothing to send (NoSuchBlocks). A requester that has not applied
// order 35 itself (a node syncing from block 0 before that block) knows it only from the SyncroMsg
// the responder sends: each peer that sent one is active, and requests to it are paced.
//
// Every forward request is recorded, also to a peer not known as active yet, so that the answer to
// it counts. A SyncroMsg that arrives while no forward request to that peer is in flight answers
// another request (the backfill's): it marks the peer active and changes nothing else.
class SyncroPacing {
public:
static constexpr uint64_t kSentLifeMs = 6'000; // the responder keeps a Sent entry 5 s
static constexpr uint64_t kInFlightMs = 30'000; // a request without an answer this long is over
static constexpr uint64_t kRefusedForMs = 3'600'000; // "for ever" on the responder: an hour here

void onSyncroMessage(const PublicKey& peer, SyncroMessage msg, uint64_t nowMs);
void onRequest(const PublicKey& peer, Sequence highest, uint64_t nowMs);
void onReply(const PublicKey& peer, uint64_t nowMs);
void forget(const PublicKey& peer);

// the peer sent a SyncroMsg: its anti-duplicate log is active
bool active(const PublicKey& peer) const;
bool anyActive() const;
// a forward request to that peer waits for its answer
bool inFlight(const PublicKey& peer, uint64_t nowMs) const;
// a request to that peer ending at `highest` would not be refused as a duplicate
bool allows(const PublicKey& peer, Sequence highest, uint64_t nowMs) const;

private:
struct State {
bool active = false;
bool requested = false;
Sequence lastHighest = 0; // the last sequence of our latest request it logged
Sequence lastHighestBefore = 0; // ... and before that request, if it refuses to log it
uint64_t loggedUntilBefore = 0;
uint64_t requestedAt = 0;
bool inFlight = false;
uint64_t loggedUntil = 0; // its entry for that request lives until then
bool awaiting = false; // AwaitAnswer for the request in flight
Sequence refusedUpTo = 0; // not again up to this (NoSuchBlocks after AwaitAnswer) ...
uint64_t refusedUntil = 0; // ... until then
};
bool inFlight(const State& s, uint64_t nowMs) const;

std::map<PublicKey, State> peers_;
};

} // namespace cs

#endif // SYNCROPACING_HPP
14 changes: 8 additions & 6 deletions csnode/src/node.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2246,14 +2246,16 @@ void Node::getSyncroMessage(const uint8_t* data, const size_t size, const cs::Pu
cs::Byte msg = cs::SyncroMessage::NoAnswer;
cs::IDataStream stream(data, size);
stream >> msg;
// a responder sends these only once special order 35 is active on its side: the backfill reads
// them whatever this node knows of that round, to pace its requests to that peer
if (backwardSync_ && stream.isValid()) {
backwardSync_->onSyncroMessage(sender, static_cast<cs::SyncroMessage>(msg));
}
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (!stream.isValid() || msg > cs::SyncroMessage::Sent) {
return;
}
// a responder sends these only once special order 35 is active on its side, which a node that
// syncs from block 0 does not know yet: the backfill and the synchronizer read them whatever
// this node knows of that round, and pace their requests to that responder; each acts only on
// the requests of its own
if (backwardSync_) {
backwardSync_->onSyncroMessage(sender, static_cast<cs::SyncroMessage>(msg));
}
poolSynchronizer_->getSyncroMessage(sender, static_cast<cs::SyncroMessage>(msg));
}

Expand Down
56 changes: 41 additions & 15 deletions csnode/src/poolsynchronizer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ void PoolSynchronizer::getBlockReply(cs::PoolsBlock&& poolsBlock, const cs::Publ
<< ", seqs: [" << poolsBlock.front().sequence()
<< ", " << poolsBlock.back().sequence() << "]";
removeSynchroLog(sender);
pacing_.onReply(sender, cs::Utils::currentTimestamp());
//std::thread sThread(&PoolSynchronizer::manageSyncBlocks, this, std::move(poolsBlock));;
//sThread.detach();
manageSyncBlocks(std::move(poolsBlock));
Expand Down Expand Up @@ -296,6 +297,7 @@ void PoolSynchronizer::onPingReceived(cs::Sequence sequence, const cs::PublicKey

void PoolSynchronizer::onNeighbourRemoved(const cs::PublicKey& publicKey) {
neighbours_.erase(publicKey);
pacing_.forget(publicKey);
}

//
Expand Down Expand Up @@ -356,15 +358,22 @@ void PoolSynchronizer::sendBlockRequest() {
csdetails() << "SYNC: All sequences already requested";
break;
}
const auto now = cs::Utils::currentTimestamp();
if (!pacing_.allows(neighbour->first, neededSequences.back(), now)) {
// its anti-duplicate log would refuse a request ending there: the range goes to the next
csdebug() << "SYNC: " << cs::Utils::byteStreamToHex(neighbour->first) << " would refuse blocks up to "
<< neededSequences.back() << " as a duplicate, asking another neighbour";
}
else {
maxRequestedSequence_ = neededSequences.back();
}
if (checkSynchroLog(neighbour->first)) {
cslog() << "SYNC: requesting for " << neededSequences.size()
<< " blocks [" << neededSequences.front() << ", " << neededSequences.back()
<< "] from " << cs::Utils::byteStreamToHex(neighbour->first);
emit sendRequest(neighbour->first, neededSequences);
addSynchroLog(neighbour->first, neededSequences, SyncroMessage::NoAnswer);
if (checkSynchroLog(neighbour->first)) {
cslog() << "SYNC: requesting for " << neededSequences.size()
<< " blocks [" << neededSequences.front() << ", " << neededSequences.back()
<< "] from " << cs::Utils::byteStreamToHex(neighbour->first);
emit sendRequest(neighbour->first, neededSequences);
pacing_.onRequest(neighbour->first, neededSequences.back(), now);
addSynchroLog(neighbour->first, neededSequences, SyncroMessage::NoAnswer);
}
}

++neighbour;
Expand Down Expand Up @@ -444,11 +453,22 @@ void PoolSynchronizer::stop() {
cslog() << "SYNC: shutdown — pool synchronizer stopped";
}

// A responder sends SyncroMsg only once special order 35 is active on its side: read whatever this
// node knows of that round, so that a node syncing from block 0 (before the order) paces its requests
// to that responder as the responder expects
void PoolSynchronizer::getSyncroMessage(const cs::PublicKey& sender, SyncroMessage msg) {
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (!pacing_.active(sender)) {
cslog() << "SYNC: " << cs::Utils::byteStreamToHex(sender)
<< " keeps an anti-duplicate log of requests (special order 35): requests to it are paced";
}
const auto now = cs::Utils::currentTimestamp();
// an answer to a forward request in flight; any other (the backfill's) only marks the peer active
const bool forward = pacing_.inFlight(sender, now);
pacing_.onSyncroMessage(sender, msg, now);
csdebug() << __func__;
if (!forward) {
return;
}
csdebug() << __func__;
if (msg == SyncroMessage::AwaitAnswer) {
changeSynchroLog(sender, msg);
}
Expand All @@ -458,7 +478,7 @@ void PoolSynchronizer::getSyncroMessage(const cs::PublicKey& sender, SyncroMessa
}

void PoolSynchronizer::addSynchroLog(const cs::PublicKey& sender, cs::PoolsRequestedSequences& sequences, SyncroMessage msg) {
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (!synchroLogFor(sender)) {
return;
}
//csdebug() << __func__;
Expand All @@ -467,7 +487,7 @@ void PoolSynchronizer::addSynchroLog(const cs::PublicKey& sender, cs::PoolsReque
}

bool PoolSynchronizer::changeSynchroLog(const cs::PublicKey& sender, SyncroMessage msg) {
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (!synchroLogFor(sender)) {
return true;
}
//csdebug() << __func__;
Expand All @@ -480,15 +500,17 @@ bool PoolSynchronizer::changeSynchroLog(const cs::PublicKey& sender, SyncroMessa
}

void PoolSynchronizer::updateSynchroLog() {
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound && !pacing_.anyActive()) {
return;
}
auto timeNow = cs::Utils::currentTimestamp();
auto it = synchroLog_.begin();
while (it != synchroLog_.end()) {
auto msg = std::get<1>(it->second);
auto timeEvent = std::get<2>(it->second);
if ((msg != SyncroMessage::AwaitAnswer && timeNow > timeEvent + 50000) || ((std::get<1>(it->second) == SyncroMessage::NoAnswer && timeNow > timeEvent + 2000))) {
// an AwaitAnswer whose reply was lost: asked again after 30 s
if ((msg != SyncroMessage::AwaitAnswer && timeNow > timeEvent + 50000) || ((std::get<1>(it->second) == SyncroMessage::NoAnswer && timeNow > timeEvent + 2000))
|| (msg == SyncroMessage::AwaitAnswer && timeNow > timeEvent + SyncroPacing::kInFlightMs)) {
it = synchroLog_.erase(it);
}
else {
Expand All @@ -498,7 +520,7 @@ void PoolSynchronizer::updateSynchroLog() {
}

bool PoolSynchronizer::removeSynchroLog(const cs::PublicKey& sender) {
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (!synchroLogFor(sender)) {
return true;
}
//csdebug() << __func__;
Expand All @@ -511,8 +533,12 @@ bool PoolSynchronizer::removeSynchroLog(const cs::PublicKey& sender) {
return true;
}

bool PoolSynchronizer::synchroLogFor(const cs::PublicKey& sender) const {
return cs::Conveyer::instance().currentRoundNumber() >= Consensus::syncroChangeRound || pacing_.active(sender);
}

bool PoolSynchronizer::checkSynchroLog(const cs::PublicKey& sender) { //is true when we can request blocks's set
if (cs::Conveyer::instance().currentRoundNumber() < Consensus::syncroChangeRound) {
if (!synchroLogFor(sender)) {
return true;
}
auto it = synchroLog_.find(sender);
Expand Down
115 changes: 115 additions & 0 deletions csnode/src/syncropacing.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
#include <csnode/syncropacing.hpp>

#include <algorithm>

namespace cs {

bool SyncroPacing::inFlight(const State& s, uint64_t nowMs) const {
return s.inFlight && nowMs < s.requestedAt + kInFlightMs;
}

void SyncroPacing::onSyncroMessage(const PublicKey& peer, SyncroMessage msg, uint64_t nowMs) {
auto& s = peers_[peer];
s.active = true;
if (!inFlight(s, nowMs)) {
// not an answer to a forward request (the backfill's, or a late one): nothing else changes
s.inFlight = false;
s.awaiting = false;
return;
}
switch (msg) {
case SyncroMessage::AwaitAnswer:
s.awaiting = true;
break;
case SyncroMessage::NoSuchBlocks:
if (s.awaiting) {
// the entry stays AwaitAnswer in its log: every request up to it is refused
s.refusedUpTo = std::max(s.refusedUpTo, s.lastHighest);
s.refusedUntil = nowMs + kRefusedForMs;
}
else {
// refused before it logged the request: its log holds what it held before
s.lastHighest = s.lastHighestBefore;
s.loggedUntil = s.loggedUntilBefore;
}
s.awaiting = false;
s.inFlight = false;
break;
case SyncroMessage::DuplicatedRequest:
// not logged: its entry from an earlier request still lives
s.lastHighest = std::max(s.lastHighestBefore, s.lastHighest);
s.loggedUntil = std::max(s.loggedUntilBefore, nowMs + kSentLifeMs);
s.awaiting = false;
s.inFlight = false;
break;
case SyncroMessage::IncorrectRequest:
s.lastHighest = s.lastHighestBefore;
s.loggedUntil = s.loggedUntilBefore;
s.awaiting = false;
s.inFlight = false;
break;
default:
break;
}
}

void SyncroPacing::onRequest(const PublicKey& peer, Sequence highest, uint64_t nowMs) {
auto& s = peers_[peer];
s.lastHighestBefore = s.lastHighest;
s.loggedUntilBefore = s.loggedUntil;
// a request it accepts replaces the last logged sequence, a lower one keeps the old entry
if (!s.requested || highest > s.lastHighest || nowMs >= s.loggedUntil) {
s.lastHighest = highest;
}
s.requested = true;
s.requestedAt = nowMs;
s.inFlight = true;
s.awaiting = false;
// until the reply comes the entry lives anyway; the reply sets the 5 s
s.loggedUntil = std::max(s.loggedUntil, nowMs + kSentLifeMs);
}

void SyncroPacing::onReply(const PublicKey& peer, uint64_t nowMs) {
const auto it = peers_.find(peer);
if (it == peers_.end()) {
return;
}
it->second.awaiting = false;
it->second.inFlight = false;
it->second.loggedUntil = nowMs + kSentLifeMs;
}

void SyncroPacing::forget(const PublicKey& peer) {
peers_.erase(peer);
}

bool SyncroPacing::active(const PublicKey& peer) const {
const auto it = peers_.find(peer);
return it != peers_.end() && it->second.active;
}

bool SyncroPacing::anyActive() const {
return std::any_of(peers_.begin(), peers_.end(), [](const auto& p) { return p.second.active; });
}

bool SyncroPacing::inFlight(const PublicKey& peer, uint64_t nowMs) const {
const auto it = peers_.find(peer);
return it != peers_.end() && inFlight(it->second, nowMs);
}

bool SyncroPacing::allows(const PublicKey& peer, Sequence highest, uint64_t nowMs) const {
const auto it = peers_.find(peer);
if (it == peers_.end() || !it->second.active) {
return true;
}
const auto& s = it->second;
if (nowMs < s.refusedUntil && highest <= s.refusedUpTo) {
return false;
}
if (s.requested && nowMs < s.loggedUntil && highest <= s.lastHighest) {
return false;
}
return true;
}

} // namespace cs
1 change: 1 addition & 0 deletions docs/CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ lines under the new release.
| #133 | ci: backward sync on BerkeleyDB: a pruned BerkeleyDB node switched to full fetches the blocks it pruned into its deleted Recno records (`backfill-berkeleydb`, `STORAGE_SWITCH_NODE`); unit test on `DatabaseBerkeleyDB` |
| #134 | api: token API leftovers: a failed transfer no longer takes a place in the pages of `TokenTransfersGet`/`TokenWalletTransfersGet`; `TokenTransferGet` answers only for a transfer of that token (it listed any transaction and added entries to the operations map); token counters above 2^31 report the largest i32 instead of wrapping; the token cache is saved, loaded and hashed under its lock (the background refresh could change it during a checkpoint) |
| #135 | samples: a token template (`samples/contracts/Token.java`: checked amounts and decimals, balances at the token's scale, ERC-20 `approve()`, sale through `payable()`), run by the executor harness, and a Python client that calls a contract with the argument types it declares (`samples/python/token_client.py`) |
| #136 | csnode: forward sync reads the responders' SyncroMsg before it has applied special order 35 itself, and does not ask a responder with an anti-duplicate log for a range that responder would refuse as a duplicate; it asks another neighbour instead (a node syncing from block 0 among mostly pruned peers could otherwise wait for ever for its first blocks) |
| #138 | ci: macOS jobs run on `macos-15` (GitHub retires the macOS 14 images on 2026-11-02 and fails them during brownouts until then), with binaries built for macOS 14 and later, and time out after their measured duration plus a margin (build 60, platform sync 40, private network 60 minutes instead of 240, 90, 150) |
| #139 | ci: the backfill crash steps (kill, power cut...) wait until the node stored a block since its last start, so that every restart resumes lower; in `fund.py` a transfer rejected by consensus right after an earlier transfer of the same sender ended with an unknown outcome (a partition longer than the 60 s wait) counts with the pending answers, not as a fault (net-partition) |

Expand Down
Loading
Loading