From 88300f9db89e71025edf5f9e6e007d3194c5738f Mon Sep 17 00:00:00 2001 From: EnzinoBB Date: Mon, 9 Nov 2026 08:09:33 +0100 Subject: [PATCH 1/2] solver: stage-3 diagnostics show how the timestamp was chosen and the block the network stored Logging only, on the failure path (info level only when a peer's stage 3 fails the final check, debug level otherwise); nothing signed, sent or stored changes. - "stage3 timestamp": alongside SolverCore::chooseTimeStamp(), without touching its arithmetic or order, the last block timestamp used, every stage-1 timestamp held for the round with its fate (in, outlier, masked, not-after-last, empty, invalid), N, mean, dispersion, the chosen timestamp and the fallback taken, so two nodes can compare the set of stamps they averaged. - "stage3 inputs" gains the hashed fields the line did not show: the confirmation mask bits, a hash of the round-confirmation signatures (with the number of zero slots), the real-trusted mask written, the block version, the round cost and the bootstrap flag. - "stage3 network block": when the stage 3 of this node failed, the block the network stores for that sequence is logged once with the same fields and the names of the fields that differ, so a failing node's log alone shows which mask-independent input differed. - docs/consensus-diagnostics.md: how to read and compare the new lines, and why ts and reward of the stored block differ anyway when the writer retried stage 3 with a smaller mask. Co-Authored-By: Claude Opus 5.5 --- docs/consensus-diagnostics.md | 72 +++++++- solver/include/solver/solvercore.hpp | 17 ++ solver/src/solvercore.cpp | 253 ++++++++++++++++++++++----- solver/src/solverinterface.cpp | 27 ++- 4 files changed, 317 insertions(+), 52 deletions(-) diff --git a/docs/consensus-diagnostics.md b/docs/consensus-diagnostics.md index 54936040c..4a485ef7a 100644 --- a/docs/consensus-diagnostics.md +++ b/docs/consensus-diagnostics.md @@ -27,24 +27,32 @@ at debug level it is logged for every stage 3: ``` info SolverCore: stage3 inputs: R-37 iter=0 seq=37 prev=7C839893 ts=1793783441147 reward=- tx=1 - txIds=40E79CA6 fees=69F07B9F charMask=FC87F4F1/1 newWallets=0 smartSigs=0 conf=5/5 writer=2 - block=BC24F874 round=44C9C6F6 trusted=D7B1BD25 mask=[3] [4] [0] [1] [2] + txIds=40E79CA6 fees=69F07B9F newWallets=0 smartSigs=0 conf=5/5 confMask=1F confSigs=9A0B7C21 + realTrusted=[0] [1] [2] [3] [4] version=0 roundCost=0.0 bootstrap=0 metaTs=1793783441147 + charMask=FC87F4F1/1 writer=2 block=BC24F874 round=44C9C6F6 trusted=D7B1BD25 mask=[3] [4] [0] [1] [2] ``` (one line in the log; wrapped here.) Hashes are shortened to their first 8 hex characters; `-` means -empty (no transactions, no reward field). +empty (no transactions, no reward field). The fields from `seq` to `bootstrap` are the hashed +fields of the block, in the order of the block binary; the `stage3 network block` line of +section 1b prints the same fields for the block the network stores. | Field | Meaning | Goes into | |---|---|---| | `seq`, `prev` | sequence and previous block hash of the new block | block hash | -| `ts` | block timestamp chosen from the stage-1 timestamps | block hash and round table | +| `ts` | block timestamp chosen from the stage-1 timestamps (section 1a shows how) | block hash and round table | | `reward` | block reward field (hex; `-` before DPoS or with mining off), computed from the confidants' cached wallet data | block hash and round table | | `tx`, `txIds` | number of transactions and a hash of their signatures in block order | block hash | | `fees` | hash of the counted fee of every transaction, in order | block hash | -| `charMask` | hash and length of the characteristic (the mask that chose the transactions) | block hash | | `newWallets` | number of new wallet ids the block assigns | block hash | | `smartSigs` | number of contract-consensus signatures carried in the block | block hash | | `conf` | round confirmations in the block: mask size / signature count | block hash | +| `confMask` | the confirmation mask bits (hex): which of the previous round's trusted signed this round's table | block hash | +| `confSigs` | hash of the confirmation signatures in order; `(n zero)` counts empty signature slots | block hash | +| `realTrusted` | the real-trusted mask written into the block | block hash | +| `version`, `roundCost`, `bootstrap` | block version, round cost and the bootstrap flag of the block | block hash | +| `metaTs` | the timestamp of the round package being built (equal to `ts`) | round table | +| `charMask` | hash and length of the characteristic (the mask that chose the transactions) | block hash | | `writer`, `mask` | writer index and real-trusted mask of this iteration (`X` = excluded) | compared directly | | `block`, `round`, `trusted` | the three hashes this node signed | signatures | @@ -74,7 +82,59 @@ nodes that differ, the first field that differs is the cause: The round table also signs the timestamp, reward, characteristic, sequence and previous hash, so a difference in one of them makes both `block` and `round` fail (`failed=block+round` below); only `block` failing points to the block-only inputs (`txIds`, `fees`, `newWallets`, `conf`, -`smartSigs`). +`confMask`, `confSigs`, `smartSigs`). + +### 1a. How the timestamp was chosen: `stage3 timestamp` + +Logged right after `stage3 inputs` (info level on a failure, debug level otherwise). The block +timestamp is the mean of the stage-1 timestamps of the confidants in the mask, taken as offsets +from the last block's timestamp, with a 3-sigma outlier drop (`SolverCore::chooseTimeStamp`). +The line shows every value that went into it: + +``` +info R-37 SolverCore> stage3 timestamp: R-37 last=1793783440011 chosen=1793783441147 N=5/5 mean=1136.200 disp=18.560 + [T0:KasmLeQs:1793783441140:in:1129] [T1:7gZ4BWYh:1793783441152:in:1141] [T2:B9XbBZUD:1793783441143:in:1132] + [T3:Br6U154N:1793783441150:in:1139] [T4:EJGNJj2S:1793783441146:in:1135] +``` + +| Field | Meaning | +|---|---| +| `last` | the last block's timestamp this node used (`BlockChain::getLastTimeStamp()`: the deferred block or the last stored one) | +| `chosen` | the timestamp written into the block (`ts` of the inputs line) | +| `N=a/b` | stamps left after the outlier drop / stamps that entered the mean | +| `mean`, `disp` | mean and dispersion of the offsets (in ms) at the end of the computation | +| `fallback` | present when no stamp was usable: `plain-mean-of-all` (integer mean of every stage-1 timestamp, masked or not), `last+1`, or `no-last-timestamp` (no block timestamp could be read: `0` is chosen) | +| `[Ti:key:stamp:fate[:offset]]` | one entry per stage 1 held for the round: confidant index, key, the raw stage-1 timestamp and its fate: `in` (used, with its offset from `last`), `outlier` (dropped by the 3-sigma test), `masked` (confidant excluded in the mask), `not-after-last` (not later than `last`), `empty`, `invalid` (not a number), `sender-out-of-range` | + +**How to compare it between nodes.** Stage-1 messages are signed, so two nodes hold the same +stamp for the same confidant: entries with a different stamp for the same index mean one node holds +another stage-1 message of that sender (a stage 1 built twice in the round). The set of entries +and their fates must agree too: a different `last` changes which stamps pass `not-after-last`, and +a stamp present on one node and missing on the other changes the mean. Note that the `mean` +alone, with the same `last`, gives `chosen`: with the same entries and the same `last`, two nodes +cannot choose different timestamps. + +### 1b. The block the network stored: `stage3 network block` + +When the stage 3 of this node failed against its peers, the block the network writes for that +sequence arrives in the next round package. If it differs from the block this node signed, it is +logged once, with the same fields as `stage3 inputs`, and the names of the fields that differ: + +``` +info R-38 SolverCore> stage3 network block: seq=37 hash=6AE2F8B5 (we signed C566481F) seq=37 prev=7C839893 + ts=1793783441150 reward=... tx=1 txIds=40E79CA6 fees=69F07B9F newWallets=0 smartSigs=0 conf=5/5 + confMask=1F confSigs=9A0B7C21 realTrusted=[0] [1] [X] [3] [4] version=0 roundCost=0.0 bootstrap=0 + differs=ts,reward,realTrusted (ts and reward also change with the real trusted mask when the writer retried stage 3) +``` + +Read `differs` with care: when the majority excluded this node (`realTrusted` has an `X` at its +index), the writer rebuilt the block in a second stage-3 iteration with the smaller mask, so `ts` +(the mean of fewer stage-1 stamps) and `reward` (shared among fewer confidants) differ for that +reason alone. Fields that do not depend on the mask (`prev`, `txIds`, `fees`, `newWallets`, +`conf`, `confMask`, `confSigs`, `version`, `roundCost`, `bootstrap`) must be equal on a healthy +node; one of them differing is the cause of the failure. When only `ts`, `reward` and +`realTrusted` differ, the first-iteration block cannot be compared this way: collect the +`stage3 inputs` and `stage3 timestamp` lines of the same round from a peer of the majority. ## 2. Why a peer failed: `stage3 check failed` diff --git a/solver/include/solver/solvercore.hpp b/solver/include/solver/solvercore.hpp index e888e5050..cff674365 100644 --- a/solver/include/solver/solvercore.hpp +++ b/solver/include/solver/solvercore.hpp @@ -103,6 +103,11 @@ class SolverCore { void gotRoundInfoRequest(const cs::PublicKey& requester, cs::RoundNumber requester_round); void gotRoundInfoReply(bool next_round_started, const cs::PublicKey& respondent); + // diagnostics only: a block stored (or a deferred block replaced) by the blockchain; when a + // stage-3 failure was logged for the same sequence and the block differs from the one this + // node signed, logs the network's block with the same fields as the "stage3 inputs" line + void onBlockStored(const csdb::Pool& pool); + // Solver3 "public" extension void gotStageOne(const cs::StageOne& stage); void gotStageTwo(const cs::StageTwo& stage); @@ -336,6 +341,18 @@ class SolverCore { // diagnostics only: the "stage3 inputs" line of our last stage 3, logged at info level only when // a peer's stage 3 fails the final check (at debug level every round) std::string stage3InputsLine_; + // diagnostics only: how chooseTimeStamp() chose the block timestamp of our last stage 3 + // (last block timestamp, every stage-1 timestamp with its fate, N, mean, dispersion), logged + // with the inputs line; the fields of our block as (name, value) pairs and the hash we signed, + // to compare with the block the network stores for that sequence + std::string stage3TimeStampLine_; + std::vector> stage3InputsFields_; + cs::Bytes stage3InputsBlockHash_; + // diagnostics only: armed by the first stage-3 failure of a round with the fields above, read + // and cleared by onBlockStored() when the network's block of that sequence arrives + cs::Sequence stage3DiagSeq_ = 0; + cs::Bytes stage3DiagBlockHash_; + std::vector> stage3DiagFields_; bool realTrustedChanged_; cs::Bytes tempRealTrusted_; diff --git a/solver/src/solvercore.cpp b/solver/src/solvercore.cpp index 4084eb2ec..07acb8408 100644 --- a/solver/src/solvercore.cpp +++ b/solver/src/solvercore.cpp @@ -13,6 +13,10 @@ #include #include +#include + +#include +#include #include #include #include @@ -23,6 +27,95 @@ namespace { const char* log_prefix = "SolverCore: "; + + // diagnostics only (stage-3 lines): short forms of keys and hashes + std::string diagShortKey(const cs::PublicKey& key) { + return EncodeBase58(key.data(), key.data() + key.size()).substr(0, 8); + } + + std::string diagShort8(const cs::Byte* data, size_t size) { + if (size == 0) { + return "-"; + } + return cs::Utils::byteStreamToHex(data, std::min(size, 4)); + } + + std::string diagHash8(const cs::Bytes& bytes) { + if (bytes.empty()) { + return "-"; + } + const cs::Hash h = cscrypto::calculateHash(bytes.data(), bytes.size()); + return diagShort8(h.data(), h.size()); + } + + // The hashed fields of a block as (name, value) pairs, in the order of the "stage3 inputs" + // line, so that the block this node signed and the block the network stores for the same + // sequence print alike and can be compared field by field. Read-only. + std::vector> diagBlockFields(const csdb::Pool& blk) { + std::vector> fields; + const auto& trxs = blk.transactions(); + cs::Bytes txSigs; + cs::Bytes txFees; + txSigs.reserve(trxs.size() * sizeof(cs::Signature)); + txFees.reserve(trxs.size() * 2); + for (const auto& t : trxs) { + const cs::Signature& sig = t.signature(); + txSigs.insert(txSigs.end(), sig.cbegin(), sig.cend()); + auto fee = t.counted_fee(); + const uint16_t rawFee = fee.get_raw(); + txFees.push_back(static_cast(rawFee & 0xFF)); + txFees.push_back(static_cast(rawFee >> 8)); + } + const cs::Bytes prevHash = blk.previous_hash().to_binary(); + std::string ts; + if (blk.user_field_ids().count(BlockChain::kFieldTimestamp) > 0) { + ts = blk.user_field(BlockChain::kFieldTimestamp).value(); + } + std::string reward; + if (blk.user_field_ids().count(BlockChain::kFieldBlockReward) > 0) { + reward = blk.user_field(BlockChain::kFieldBlockReward).value(); + } + // the round confirmations: the signatures of the previous round's trusted on this round's + // table, as the node holds them; zero signatures are counted apart + cs::Bytes confSigs; + size_t zeroSigs = 0; + for (const auto& s : blk.roundConfirmations()) { + confSigs.insert(confSigs.end(), s.cbegin(), s.cend()); + if (s == cs::Zero::signature) { + ++zeroSigs; + } + } + std::ostringstream confMask; + confMask << std::hex << std::uppercase << blk.roundConfirmationMask(); + fields.emplace_back("seq", std::to_string(blk.sequence())); + fields.emplace_back("prev", diagShort8(prevHash.data(), prevHash.size())); + fields.emplace_back("ts", ts.empty() ? std::string("-") : ts); + fields.emplace_back("reward", reward.empty() ? std::string("-") : cs::Utils::byteStreamToHex(reinterpret_cast(reward.data()), reward.size())); + fields.emplace_back("tx", std::to_string(trxs.size())); + fields.emplace_back("txIds", diagHash8(txSigs)); + fields.emplace_back("fees", diagHash8(txFees)); + fields.emplace_back("newWallets", std::to_string(blk.newWallets().size())); + fields.emplace_back("smartSigs", std::to_string(blk.smartSignatures().size())); + fields.emplace_back("conf", std::to_string(static_cast(blk.numberConfirmations())) + '/' + std::to_string(blk.roundConfirmations().size())); + fields.emplace_back("confMask", confMask.str()); + fields.emplace_back("confSigs", diagHash8(confSigs) + (zeroSigs ? "(" + std::to_string(zeroSigs) + " zero)" : std::string())); + fields.emplace_back("realTrusted", cs::TrustedMask::toString(cs::Utils::bitsToMask(blk.numberTrusted(), blk.realTrusted()))); + fields.emplace_back("version", std::to_string(static_cast(blk.version()))); + fields.emplace_back("roundCost", blk.roundCost().to_string()); + fields.emplace_back("bootstrap", BlockChain::isBootstrap(blk) ? "1" : "0"); + return fields; + } + + std::string diagJoin(const std::vector>& fields) { + std::string s; + for (const auto& [name, value] : fields) { + s += ' '; + s += name; + s += '='; + s += value; + } + return s; + } } namespace cs { @@ -230,12 +323,53 @@ uint64_t SolverCore::lastTimeStamp() { } std::string SolverCore::chooseTimeStamp(cs::Bytes mask) { + // Diagnostics only (the "stage3 timestamp" line): every stage-1 timestamp with its fate in + // this computation. The arithmetic below is the original one: the diagnostics read the same + // values in the same order and change nothing. + struct TsDiag { + uint8_t sender; + std::string raw; + const char* fate; + double x; + }; + std::vector diag; + diag.reserve(stageOneStorage.size()); + const char* fallback = nullptr; + double diagDisp = 0.0; + int diagN0 = 0; + std::string chosen; + const auto finish = [&](int64_t lastTimeStamp, double mean, int N) { + // the surviving stamps are those still in the list: an included stamp whose value is no + // longer there was dropped as an outlier (equal values share the same fate) + const cs::Conveyer& conveyer = cs::Conveyer::instance(); + std::ostringstream os; + os << "stage3 timestamp: R-" << conveyer.currentRoundNumber() << " last=" << lastTimeStamp + << " chosen=" << chosen << " N=" << N << '/' << diagN0 << std::fixed << std::setprecision(3) + << " mean=" << mean << " disp=" << diagDisp; + if (fallback != nullptr) { + os << " fallback=" << fallback; + } + for (const auto& d : diag) { + os << " [T" << static_cast(d.sender) << ':' + << (conveyer.isConfidantExists(d.sender) ? diagShortKey(conveyer.confidantByIndex(d.sender)) : std::string("?")) + << ':' << (d.raw.empty() ? std::string("-") : d.raw) << ':' << d.fate; + if (std::strcmp(d.fate, "in") == 0 || std::strcmp(d.fate, "outlier") == 0) { + os << ':' << std::setprecision(0) << d.x; + } + os << ']'; + } + stage3TimeStampLine_ = os.str(); + return chosen; + }; + int64_t lastTimeStamp; try { lastTimeStamp = std::stoll(pnode->getBlockChain().getLastTimeStamp()); } catch(...) { csdebug() << "ChooseTimeStamp - Timestamp was announced as zero"; - return std::to_string(0); + chosen = std::to_string(0); + fallback = "no-last-timestamp"; + return finish(0, 0.0, 0); } std::list stamps; @@ -243,20 +377,26 @@ std::string SolverCore::chooseTimeStamp(cs::Bytes mask) { double mean = 0.0; int N = 0; for (auto& it : stageOneStorage) { + diag.push_back({it.sender, it.roundTimeStamp, "sender-out-of-range", 0.0}); if (it.sender >= cs::Conveyer::instance().confidantsCount()) { continue; } + diag.back().fate = it.roundTimeStamp.empty() ? "empty" : "masked"; if (!it.roundTimeStamp.empty() && mask[it.sender] != cs::ConfidantConsts::InvalidConfidantIndex) { int64_t tStamp; + diag.back().fate = "invalid"; try { tStamp = std::stoll(it.roundTimeStamp); } catch(...) { cswarning() << log_prefix << "incompatible timestamp received from [" << (int)it.sender << "]"; continue; } + diag.back().fate = "not-after-last"; if (tStamp > lastTimeStamp) { double x = static_cast(tStamp - lastTimeStamp); + diag.back().x = x; if (x > DBL_EPSILON) { + diag.back().fate = "in"; stamps.push_back(x); sx += x; sx2 += x * x; @@ -266,6 +406,7 @@ std::string SolverCore::chooseTimeStamp(cs::Bytes mask) { } } + diagN0 = N; bool isDrop = true; while (isDrop) { @@ -286,17 +427,27 @@ std::string SolverCore::chooseTimeStamp(cs::Bytes mask) { sx0 += static_cast(tStamp); } + for (auto& d : diag) { + if (std::strcmp(d.fate, "in") == 0) { + d.fate = "outlier"; + } + } if (N0 > 0) { - return std::to_string(sx0/N0); + chosen = std::to_string(sx0/N0); + fallback = "plain-mean-of-all"; + return finish(lastTimeStamp, mean, 0); } else { - return std::to_string(lastTimeStamp + 1); + chosen = std::to_string(lastTimeStamp + 1); + fallback = "last+1"; + return finish(lastTimeStamp, mean, 0); } } double N_1 = 1. / N; mean = sx * N_1; double disp = sx2 * N_1 - mean * mean; + diagDisp = disp; isDrop = false; for (auto it = stamps.begin(), end = stamps.end(); it != end;) { double x = *it; @@ -312,7 +463,13 @@ std::string SolverCore::chooseTimeStamp(cs::Bytes mask) { auto meanTimeStamp = static_cast(lastTimeStamp + mean + 0.5); //csdebug() << "finish: " << meanTimeStamp; - return std::to_string(meanTimeStamp); + chosen = std::to_string(meanTimeStamp); + for (auto& d : diag) { + if (std::strcmp(d.fate, "in") == 0 && std::find(stamps.cbegin(), stamps.cend(), d.x) == stamps.cend()) { + d.fate = "outlier"; + } + } + return finish(lastTimeStamp, mean, N); } @@ -635,57 +792,63 @@ void SolverCore::spawn_next_round(const cs::PublicKeys& nodes, const cs::Packets // signed, sent or stored. { const csdb::Pool& blk = deferredBlock_; - const auto& trxs = blk.transactions(); - cs::Bytes txSigs; - cs::Bytes txFees; - txSigs.reserve(trxs.size() * sizeof(cs::Signature)); - txFees.reserve(trxs.size() * 2); - for (const auto& t : trxs) { - const cs::Signature& sig = t.signature(); - txSigs.insert(txSigs.end(), sig.cbegin(), sig.cend()); - auto fee = t.counted_fee(); - const uint16_t rawFee = fee.get_raw(); - txFees.push_back(static_cast(rawFee & 0xFF)); - txFees.push_back(static_cast(rawFee >> 8)); - } - auto short8 = [](const cs::Byte* data, size_t size) -> std::string { - if (size == 0) { - return "-"; - } - return cs::Utils::byteStreamToHex(data, std::min(size, 4)); - }; - auto hash8 = [&short8](const cs::Bytes& bytes) -> std::string { - if (bytes.empty()) { - return "-"; - } - const cs::Hash h = cscrypto::calculateHash(bytes.data(), bytes.size()); - return short8(h.data(), h.size()); - }; - const cs::Bytes prevHash = blk.previous_hash().to_binary(); + // the hashed fields of the block, in the order of diagBlockFields(); the "stage3 network + // block" line prints the same fields for the block the network stores for this sequence + stage3InputsFields_ = diagBlockFields(blk); + stage3InputsBlockHash_ = lastHashBin; const cs::PoolMetaInfo& meta = justCreatedRoundPackage_.poolMetaInfo(); std::ostringstream line; line << "stage3 inputs: R-" << conveyer.currentRoundNumber() << " iter=" << static_cast(stage3.iteration) - << " seq=" << seq - << " prev=" << short8(prevHash.data(), prevHash.size()) - << " ts=" << meta.timestamp - << " reward=" << (meta.reward.empty() ? std::string("-") : cs::Utils::byteStreamToHex(meta.reward.data(), meta.reward.size())) - << " tx=" << trxs.size() - << " txIds=" << hash8(txSigs) - << " fees=" << hash8(txFees) - << " charMask=" << hash8(block_characteristic->mask) << '/' << block_characteristic->mask.size() - << " newWallets=" << blk.newWallets().size() - << " smartSigs=" << blk.smartSignatures().size() - << " conf=" << static_cast(blk.numberConfirmations()) << '/' << blk.roundConfirmations().size() + << diagJoin(stage3InputsFields_) + << " metaTs=" << meta.timestamp + << " charMask=" << diagHash8(block_characteristic->mask) << '/' << block_characteristic->mask.size() << " writer=" << static_cast(stage3.writer) - << " block=" << short8(stage3.blockHash.data(), stage3.blockHash.size()) - << " round=" << short8(stage3.roundHash.data(), stage3.roundHash.size()) - << " trusted=" << short8(stage3.trustedHash.data(), stage3.trustedHash.size()) + << " block=" << diagShort8(stage3.blockHash.data(), stage3.blockHash.size()) + << " round=" << diagShort8(stage3.roundHash.data(), stage3.roundHash.size()) + << " trusted=" << diagShort8(stage3.trustedHash.data(), stage3.trustedHash.size()) << " mask=" << cs::TrustedMask::toString(stage3.realTrustedMask); stage3InputsLine_ = line.str(); csdebug() << log_prefix << stage3InputsLine_; + if (!stage3TimeStampLine_.empty()) { + csdebug() << log_prefix << stage3TimeStampLine_; + } + } +} + +void SolverCore::onBlockStored(const csdb::Pool& pool) { + // Diagnostics only. Armed by logStage3Failure() with the fields of the block this node signed + // in the round whose stage 3 failed; when the blockchain stores (or takes over) another block + // of that sequence, the network's block is logged with the same fields and the names of the + // fields that differ. Read-only: the block is not changed. Disarmed in any case, so at most one + // line per diagnosed round; a block of another sequence does not touch the armed state. + if (stage3DiagSeq_ == 0 || !pool.is_valid() || pool.sequence() != stage3DiagSeq_) { + return; + } + const cs::Bytes hash = pool.hash().to_binary(); + const cs::Sequence seq = stage3DiagSeq_; + const auto ours = std::move(stage3DiagFields_); + const cs::Bytes signedHash = std::move(stage3DiagBlockHash_); + stage3DiagSeq_ = 0; + stage3DiagFields_.clear(); + stage3DiagBlockHash_.clear(); + if (hash == signedHash) { + return; // the block we signed: nothing to compare + } + const auto theirs = diagBlockFields(pool); + std::string differs; + for (const auto& [name, value] : theirs) { + for (const auto& [ourName, ourValue] : ours) { + if (ourName == name && ourValue != value) { + differs += (differs.empty() ? "" : ",") + name; + } + } } + cslog() << kLogPrefix_ << "stage3 network block: seq=" << seq << " hash=" << diagShort8(hash.data(), hash.size()) + << " (we signed " << diagShort8(signedHash.data(), signedHash.size()) << ")" << diagJoin(theirs) + << " differs=" << (differs.empty() ? std::string("none") : differs) + << " (ts and reward also change with the real trusted mask when the writer retried stage 3)"; } void SolverCore::uploadNewStates([[maybe_unused]] std::vector newStates) { diff --git a/solver/src/solverinterface.cpp b/solver/src/solverinterface.cpp index 409e0acea..676508837 100644 --- a/solver/src/solverinterface.cpp +++ b/solver/src/solverinterface.cpp @@ -71,6 +71,9 @@ void SolverCore::init( void SolverCore::subscribeToSignals() { psmarts->subscribeToSignals(pnode); + // diagnostics only: the network's block of a sequence whose stage 3 failed on this node + cs::Connector::connect(&pnode->getBlockChain().storeBlockEvent, this, &SolverCore::onBlockStored); + cs::Connector::connect(&pnode->getBlockChain().lastBlockReplacedEvent, this, &SolverCore::onBlockStored); } void SolverCore::gotConveyerSync(cs::RoundNumber rNum) { @@ -203,6 +206,9 @@ void SolverCore::nextRound(bool updateRound) { trueStageThreeStorage.clear(); stage3Failures_.clear(); stage3InputsLine_.clear(); + stage3TimeStampLine_.clear(); + stage3InputsFields_.clear(); + stage3InputsBlockHash_.clear(); trusted_candidates.clear(); realTrustedChanged_ = false; tempRealTrusted_.clear(); @@ -513,8 +519,27 @@ void SolverCore::logStage3Failure(const cs::StageThree& stageFrom, uint8_t patte } } if (stage3Failures_.empty() && !stage3InputsLine_.empty()) { - // the first failure of the iteration: what we signed, to compare with the other side's line + // the first failure of the iteration: what we signed, to compare with the other side's line, + // how the timestamp was chosen, and arm the comparison with the block the network stores cslog() << kLogPrefix_ << stage3InputsLine_; + if (!stage3TimeStampLine_.empty()) { + cslog() << kLogPrefix_ << stage3TimeStampLine_; + } + if (!stage3InputsFields_.empty() && !stage3InputsBlockHash_.empty()) { + stage3DiagSeq_ = 0; + for (const auto& [name, value] : stage3InputsFields_) { + if (name == "seq") { + try { + stage3DiagSeq_ = static_cast(std::stoull(value)); + } + catch (...) { + stage3DiagSeq_ = 0; + } + } + } + stage3DiagFields_ = stage3InputsFields_; + stage3DiagBlockHash_ = stage3InputsBlockHash_; + } } stage3Failures_.emplace_back(stageFrom.sender, pattern); if (stage3Failures_.size() > kS3SplitMaxFailures) { From e79d7faadf8ab3b5ea0d4d4ff5a5ab4cbc24cdf3 Mon Sep 17 00:00:00 2001 From: EnzinoBB Date: Mon, 9 Nov 2026 08:09:52 +0100 Subject: [PATCH 2/2] docs: CHANGES row for #176 Co-Authored-By: Claude Opus 5.5 --- docs/CHANGES.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/CHANGES.md b/docs/CHANGES.md index ff0547bc8..2a57e799d 100644 --- a/docs/CHANGES.md +++ b/docs/CHANGES.md @@ -48,6 +48,7 @@ lines under the new release. | #173 | packaging: a new full node defaults to rocksdb (it starts from a snapshot) in Docker and in the guided set-up (setup.sh, setup.ps1); pruned stays on lmdb; existing installs keep their backend (the database in `db/`, the `db_backend` of `config.ini`, berkeleydb for a full `[storage]` without one; the entrypoint also detects the database of a data folder whose `config.ini` was removed); `docker-compose.yml` takes its limits from `CREDITS_CPUS` (1.5) and `CREDITS_MEMORY` (4g) in `.env` (a full node: 4 and 8g); texts aligned: a full node on rocksdb about 250 to 300 GB, backfill about 2 to 3 days, glibc 2.38 for `linux-x64` (2.31 for `-glibc2.31`) in the README, the image `ghcr.io/creditscommunity/credits-node`, the `config.pruned.ini` comment on the snapshot start; quick-start, storage.md and the Docker README updated; ci: setup-test and the Docker entrypoint test check the rocksdb default and that a full berkeleydb install is kept | | #174 | docs: quick-start: using an existing node key, plain or encrypted, in Docker or native (a section with copy-paste steps; an encrypted key converted once to plain with the node's own `--dumpkeys`, for Docker and systemd) | | #175 | node: `--decryptkey` turns a key file encrypted with a password by the node into the plain key after asking the old password (the encrypted file kept as `NodePrivate.txt.encrypted-`, the public key checked, no configuration, network or database); Docker: `docker compose run --rm node decrypt-key`, and a normal start with an encrypted key stops with that hint instead of waiting for a password; quick-start section 10 uses it instead of the `--dumpkeys` procedure of #174 | +| #176 | solver: stage-3 diagnostics (logging only, failure path): a `stage3 timestamp` line with the last block timestamp, every stage-1 timestamp and its fate (in, outlier, masked, not after last, empty, invalid), N, mean, dispersion and the chosen value; `stage3 inputs` also shows the confirmation mask bits, a hash of the round-confirmation signatures, the real-trusted mask, version, round cost and bootstrap flag; a `stage3 network block` line prints the block the network stored for a sequence whose stage 3 failed here, with the fields that differ; docs/consensus-diagnostics.md sections 1a and 1b | ## community.17 — 2026-11-05