Skip to content

solver: info-level stage-3 diagnostics (logging only) - #167

Merged
EnzinoBB merged 5 commits into
masterfrom
diag/stage3-logging
Nov 4, 2026
Merged

EnzinoBB merged 5 commits into
masterfrom
diag/stage3-logging

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Info-level diagnostics for stage-3 disagreements, following the operator report where a trusted node failed "Block/Round Signatures are not valid" for most peers (trusted signature and mask agreeing), waited for bootstrap, and saw its own key gray-listed for 1000 rounds.

Logging only. Nothing that is signed, hashed, compared, sent or stored changes; no new exceptions; no serialized structure changes.

What is logged

Where Line Level
solvercore.cpp (after "StageThree prepared") stage3 inputs: round, iteration, seq, prev hash, timestamp, reward (hex), tx count, hash of the tx signatures in block order, hash of the counted fees, characteristic mask hash/size, newWallets, smart signatures, confirmations (mask size / signatures), writer, our block/round/trusted hashes (8 hex), realTrustedMask. One line per stage 3. info
solverinterface.cpp (final check) stage3 check failed T[i] <key8> ... failed=<checks> -> <verdict>: (a) signed another block/round hash (key ok, trusted hash agreed), (b) key or message mismatch (trusted signature fails too), plus already marked untrusted / mask or writer differs info
same stage3 split: who agreed with our hashes (own marked) and the failing senders grouped by failure pattern; bounded to 10 failures per iteration info
node.cpp performRoundPackage round package R-n from <sender8> gray-lists k of N confidants of R-(n-1) for 1000 rounds: [T<i>:<key8>...], with (OWN KEY) / - OWN KEY gray-listed info
trustedstage3state.cpp "not enough active confidants" now has active a of N, need m, mask ... warning (unchanged)
blockchain.cpp "signatures of the block are insufficient or incorrect" raised to warning, with #seq hash <hex> prev <hex> debug → warning

Keys are the first 8 base58 characters of public keys. Private keys are never logged.

About point (c) of the request (verify a failing signature against other peers' hashes): stage 3 carries no hashes on the wire, and every accepted stage 3 matched our hashes, so there is no other hash to verify against; re-deriving an alternative round hash (RoundPackage::bytesToSign(!showVersion)) would rewrite the package's cached binary, so it is not done. Failing senders are grouped by pattern instead, which shows the majority/minority split.

docs/consensus-diagnostics.md explains how an operator reads the lines to find the input that differed; docs/differences.md links it; docs/CHANGES.md row added.

Tests

  • Linux build (Release, GCC 13) and ctest: 3/3 passed.
  • Testnet, mixed with official nodes, contracts — run 41803547222 (node_alt_ref=official, contracts=1): success, all 7 nodes at 1101, equal STATE DIGESTs; 0 check failures, 0 gray-listings. Our nodes (odd) print stage3 inputs, and all of them print identical hashes per round, e.g. R-600 on three nodes:
    info SolverCore: stage3 inputs: R-600 iter=0 seq=600 prev=DE8B1C9D ts=1793783737499 reward=- tx=0 txIds=- fees=- charMask=-/0 newWallets=0 smartSigs=0 conf=5/5 writer=0 block=B278C18D round=97AABEF7 trusted=3983BD64 mask=[0] [1] [2] [3] [4]
    info SolverCore: stage3 inputs: R-37 iter=0 seq=37 prev=7C839893 ts=1793783441147 reward=- tx=1 txIds=40E79CA6 fees=69F07B9F charMask=FC87F4F1/1 newWallets=0 smartSigs=0 conf=5/5 writer=2 block=BC24F874 round=44C9C6F6 trusted=D7B1BD25 mask=[3] [4] [0] [1] [2]
    
  • Robustness chaos-mixed, net-partition, slow-nodes — run 41803551514 dispatched with only=chaos-mixed,net-partition,slow-nodes (comma-separated, no spaces: the filter is contains(',<only>,', ',<name>,')); only those three ran, all success. Gray-list lines appeared (no stage-3 signature failures in these scenarios: exclusions there come from missing stages):
    info NODE> round package R-383 from 14bxQA13 gray-lists 2 of 5 confidants of R-382 for 1000 rounds: [T1:Gc85w6MG T2:Ex9XyKoB(OWN KEY)] - OWN KEY gray-listed
    info NODE> round package R-265 from EyGZsFmd gray-lists 3 of 7 confidants of R-264 for 1000 rounds: [T1:6Ja9p5Fp T2:8zpQtvcU T3:9UjMt31j]
    warning Trusted-3: not enough active confidants to make a decision, Bootstrap required (active 2 of 4, need 3, mask [X] [0] [X] [0] )
    
  • Reproduction of the reported sequence — run 41807016444 on the separate branch test/stage3-skew (NOT part of this PR, never to be merged): n3 and n5 build their block with the timestamp + 1 ms. The run reached 508 blocks and failed only on its own STATE DIGEST #1000 check (target was 500). It reproduces the operator's report and the new lines pinpoint the timestamp:
    stage3 inputs: R-1 iter=0 seq=1 prev=C0541961 ts=1793784561704 ... block=2E6353D9 round=A5256CC5 trusted=CA0A93D1   (majority)
    stage3 inputs: R-1 iter=0 seq=1 prev=C0541961 ts=1793784561705 ... block=9876C3C4 round=7B1677FB trusted=CA0A93D1   (skewed)
    info R-1 SolverCore> stage3 check failed T[4] EJGNJj2S iter=0 msg=ok failed=block+round -> signed another block/round hash (key ok, trusted hash agreed)
    info R-1 SolverCore> stage3 split iter=0: agree(5)=[T0:KasmLeQs T1:7gZ4BWYh T2:B9XbBZUD(own) T3:Br6U154N T6:HhovRb4b] | block+round(2)=[T4:EJGNJj2S T5:GqamRj6f] of 7
    info R-1 SolverCore> stage3 split iter=0: agree(1)=[T4:EJGNJj2S(own)] | block+round(5)=[T0:KasmLeQs T1:7gZ4BWYh T2:B9XbBZUD T3:Br6U154N T6:HhovRb4b] of 7
    warning Trusted-3: not enough active confidants to make a decision, Bootstrap required (active 2 of 7, need 4, mask [X] [X] [X] [X] [1] [2] [X] )
    info NODE> round package R-2 from Br6U154N gray-lists 2 of 7 confidants of R-1 for 1000 rounds: [T4:EJGNJj2S T5:GqamRj6f(OWN KEY)] - OWN KEY gray-listed
    

Consensus-neutrality notes for review

  • solvercore.cpp: reads deferredBlock_ through a const reference (newWallets() const overload, so no copy-on-write detach), the round package's poolMetaInfo(), and hashes copies of tx signatures/fees with cscrypto::calculateHash. deferredBlock_.hash() was already computed above. One allocation of n×64 bytes per stage 3.
  • blockchain.cpp: pool.hash() lazily computes and caches a hash; it is only called when pool.is_valid(), i.e. after compose() already set it, so nothing new is cached.
  • solverinterface.cpp: adds a member stage3Failures_ (diagnostic state, cleared with the stage storages in nextRound/adjustStageThreeStorage); never read by consensus code. No extra signature verifications.
  • node.cpp: the sender parameter of performRoundPackage is now named (was /*sender*/); the gray-list loop calls addToGraylist exactly as before.

🤖 Generated with Claude Code

EnzinoBB and others added 5 commits November 4, 2026 09:57
- solvercore: one "stage3 inputs" line per stage 3 with the inputs of the
  signed hashes (seq, prev hash, timestamp, reward, tx/fee hashes,
  characteristic, new wallets, confirmations, writer, mask, our 3 hashes)
- solverinterface: classify each stage-3 final-check failure (another
  block/round hash vs key or message mismatch) and log the agree/disagree
  split, grouped by failure pattern, bounded to 10 failures per iteration
- node: performRoundPackage logs the keys a round package gray-lists,
  "OWN KEY" when ours is one of them
- trustedstage3state: active/total/needed counts and mask on the
  "not enough active confidants" warning
- blockchain: "signatures of the block are insufficient or incorrect" is a
  warning with the block's sequence, hash and previous hash
- docs/consensus-diagnostics.md: how to read the lines

Nothing that is signed, hashed, compared, sent or stored changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB merged commit f566f2c into master Nov 4, 2026
4 checks passed
EnzinoBB added a commit that referenced this pull request Nov 4, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant