Repository navigation
csnode: backfill signatures verified on worker threads - #152
Merged
Merged
Conversation
The backfill held the processor thread 97-98 % of the time on mainnet, about 2.5 ms a block: 1 ms for the block's signatures, 1 ms for the round confirmations of the block stored before, 0.5 ms to store it. A pool of verifier threads (Settings::verifyThreads, [storage] backfill_verify_threads, 0..16, default min(4, cores - 1); 0 keeps everything on the processor thread) now checks the signatures of each block as it enters the buffer, highest blocks first, on a copy of the block whose hash was computed before it was shared. drain() keeps the sequence and hash-chain checks, takes the result when it is ready and ends the turn otherwise, never waiting. The result lives in a shared_ptr owned by the buffer entry: a block refused, replaced or dropped leaves its result to nobody, and the verifier skips a check nobody waits for. The round confirmations of the block stored before are checked on the verifier threads after the store, with the same predicate; a mismatch is counted and logged as before, never refused. stop() and the destructor join the threads (a check under way ends first, the queued ones are dropped); after the end of the backfill the threads finish the queued confirmations and return. The per-phase averages of #151 add the verifier threads' time atomically; busyUs and the busy share count only the processor thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- signature checks are served before the confirmation checks (only the former gate the storage), each highest block first - a failure to grow the thread vector falls back like a thread that cannot start (reserve, catch std::exception) - the slow-turn line no longer reports a confirmations time when the signatures run on the verifier threads - test: wait for the confirmation line itself, not only the counter that goes up before it is logged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmation jobs are served after the signature jobs; on a single slow verifier they could pile up, each holding two blocks. Past 2000 queued, the processor thread checks the confirmations of the next stored blocks itself (the existing fallback when submit() refuses), which also slows the storing down until the verifiers catch up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Measured on the mainnet test node (master b703913, 1.4 M blocks), per backfilled block on the processor thread:
verify(signatures plus sequence/hash checks) ≈ 1000 µs,confirmations(the round confirmations of the block stored before against this block's confidants) ≈ 1000 µs,store≈ 500 µs. The backfill held the processor thread 97–98 % of the time (turn budgets of #140/#146 plus the drains inonBlocks). Moving the two signature checks to worker threads should leave about 0.5 ms a block on the processor thread.Design
BackwardSync::Settings::verifyThreads(config[storage] backfill_verify_threads,0..16, defaultmin(4, cores − 1), at least 1;0= everything on the processor thread as before). The threads start lazily at the first job and are joined bystop()and the destructor, like the flush thread of csnode: the periodic backfill flush runs on its own thread #148.buffer_(onBlocks), its hash is computed (and cached in the sharedcsdb::Pooldata) on the processor thread, and a job with a copy of the pool plus ashared_ptr<SignatureCheck>(atomic<bool> ready+why) is queued. The queue is ordered highest sequence first, since the storage always waits for the highest buffered block.sequence,hash, thensignatures). If the block's check is ready its result is used — a failure goes throughreject()exactly as today; if not, the turn ends (no waiting) and the next tick/reply goes on.SignatureCheckis owned by theBufferedentry. A block that is rejected, replaced by a re-sent copy, or dropped atfinish()leaves its result to nobody; the verifier skips a job whose check nobody holds any more (use_count() == 1). No keying by sequence/hash is needed.onBlocks): they are queued right after the block is stored, as a job with copies oflastStored_(the block above) and the stored block, i.e. exactly the pairs checked today. They decide nothing (a mismatch is counted and logged, never a rejection), so nothing needs to wait for them. Pairing at buffer time would have needed results keyed by the upper block's hash, plus a fallback when the upper block changed. One consequence:confirmationErrorsand its log line can come slightly after the block is stored, and astop()drops the checks still queued. After the end of the backfill, the threads finish the queued confirmations and return.phaseVerifyUs_/phaseConfirmUs_are atomics that the verifier threads add to.busyUs/ "processor thread busy %" count only processor-thread time, including the hashing done inonBlocks. The progress line says when verify and confirmations ran on the verifier threads. The slow-turn log still reports processor time only.onBlocksafterstop()now drops the reply (the verifiers are gone, and the node is closing its storage).std::system_error), it logs and falls back toverifyThreads = 0.Unchanged: the verification predicates (
BlockVerifier::verifySignatures/verifyConfirmations), the order of storing, block 0 (genesis without signatures:verifySignaturesaccepts sequence 0), floor/finish, retries and peer marking.Thread safety
csdb::Poolcopies, the job'sSignatureCheck, the atomic counters (confirmationErrors_,phaseVerifyUs_,phaseConfirmUs_), andlog()/hooks_.onLog/hooks_.onVerify, which are set before any thread starts and only read afterwards. It never readsbuffer_,next_,lastStored_or any other processor-thread state.csdb::Poolis a ref-counted copy-on-write handle with an atomic count. Its const getters used byBlockVerifierare reads, excepthash(), which computes the hash lazily through a non-const access (detach). The hash is therefore computed on the processor thread before the copy is shared, so no thread writes the shared data. A later non-const access on either side detaches (copies) while the count is > 1, which is a read of the shared data.BlockChain::isBootstrap,NodeUtils::checkGroupSignatureand cscrypto/libsodium verification are pure.whyis written beforeready.store(release)and read afterready.load(acquire).stop(): it setsverifyStopping_underverifyMutex_, takes the queue out (the queued jobs are dropped), notifies, and joins outside the lock. A verifier in the middle of a check holds no lock, so it ends that check and returns.-fsanitize=thread: 27/27 pass with no ThreadSanitizer report, also 30 repetitions of the threaded tests pinned to one core.Tests
settingsOf()setsverifyThreads = 0, so all existing tests run on the processor thread as before.TearDown()destroys the sync while the fixture it logs to still exists. The sharedGate/Releaseof the flush test moved to the namespace so the new tests can reuse it.VerifierThreadsStoreTheSameBlocks(3 threads): out-of-order replies are stored strictly downwards to block 0; the block 350 confirmation mismatch is counted exactly once.VerifierThreadsRefuseBadSignatures(2 threads): a bad signature on block 400 is rejected, the peer is marked, the batch is re-requested from another peer, and the backfill completes.ATurnDoesNotWaitForTheVerifiers: with the verifiers held at a gate,onBlocksand ticks return without storing anything (100 buffered); once the gate opens the blocks are stored.StopEndsTheVerifiers:stop()while a verifier is mid-check returns once that check ends, and nothing more is stored.BackfillConfig.VerifyThreads: the default (unset), 0, 3, clamped to 16, and bad values leave it unset.For the reviewer
backfill_store_per_turn(default 50) becomes the cap at 10 ticks/s plus the drains in replies: about 500+ blocks/s. It may be worth raising after a mainnet measurement.🤖 Generated with Claude Code