Repository navigation
csnode: a backfill turn ends after 50 ms; slow turns are logged - #140
Merged
Merged
Conversation
On mainnet a full node from the snapshot (rocksdb) had backfill turns of up to 1.7 s for 50 blocks, about 85 ms on average, while a round lasts about 1.2 s: a confidant would hold the processor thread through a round. A turn now also ends once 50 ms have passed (after at least one block); a turn over 200 ms is logged with its blocks, transactions and the time spent verifying, storing and checking confirmations, so that the cause shows, and the progress line counts them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a rejected block's peer key is copied before reject() erases the buffer entry it lives in (it was read after being freed) - the budget and the slow-turn decision use the hooks' clock, so the test drives them without real sleeps - the storage flush is timed in its log line and in the stats - [storage] backfill_turn_budget_ms (50, 0 = no limit, at most 1000) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The mainnet test (full rocksdb node from the snapshot, master e0d2d31) backfills at ~500-600 blocks/s, but its longest processor-thread turn grew to 1744 ms for at most 50 blocks (~85 ms on average), while a mainnet round lasts ~1.2 s. On a confidant that holds consensus. Only the maximum was logged, so neither the cause nor the frequency was visible. RocksDB logged no write stalls; the log gaps at whole minutes are network-wide (the pruned node shows them too).
Change
Settings::turnBudgetMs(50 ms): a turn ends once it has used its time, after at least one block, besidesbackfill_store_per_turn.Stats::slowTurns; the progress line adds the count.ASlowTurnStopsAtItsTimeBudget(a store hook that sleeps); the other tests run without a time limit.Test plan
🤖 Generated with Claude Code