Repository navigation
csnode: the periodic backfill flush runs on its own thread - #148
Merged
Merged
Conversation
On the mainnet test node the backfill flush (transaction index, block hashes, storage sync) took up to 5.6 s on the processor thread. It now runs on a thread of BackwardSync: one at a time (the blocks stored meanwhile go with the next one), the final flush at the end of the backfill waits for the one under way and runs on the calling thread, and stop() / the destructor join the thread; Node::destroy stops the backfill before it flushes and closes the storage. The engines' flushes are thread-safe (LMDB syncs under the map lock shared with transactions, RocksDB SyncWAL, BerkeleyDB with DB_THREAD; error state is per thread). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the test's gate is shared with the flush thread by value and waits up to 5 s for the thread to start (a failed assertion no longer leaves it on a dead mutex) - stopFlusher detaches instead of terminating when called on the flush thread itself; a thread that cannot be created falls back to the synchronous flush; an exception from the flush is caught and logged - BlockHashes::failures_ is atomic (the flush thread may report an LMDB failure); comment on the flush skipped after stop(); docs: the one remaining wait (LMDB map growth during a sync) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On the mainnet test node (full rocksdb from the snapshot, master e422d2f) the periodic backfill flush ran on the processor thread for up to 5.6 s (median ~360 ms, every ~10,000 blocks). #147 defers it while the node is a confidant; this moves it off the processor thread altogether.
Change
Settings::asyncFlush(default true): a due periodic flush is handed to a thread ofBackwardSync(started at the first one). One at a time: blocks stored meanwhile go with the next flush. The log line ends withon the flush thread; its duration no longer counts as processor-thread time.stop()and the destructor stop and join the thread;Node::destroystops the backfill before its lastflushBackfill()andclose().BlockChain::flushBackfill):cs::Lmdb::flushsyncs under the map mutex shared with transactions (a map growth waits for it);DatabaseRocksDB::flushisSyncWAL; BerkeleyDB is opened withDB_THREAD;Storage::flushwith async writes waits on its own queue lock; csdb error state is thread_local. No dbLock_ (as before).ThePeriodicFlushRunsOnItsOwnThread(a flush held by the test: storing goes on, no second flush, the final flush waits); the other tests flush synchronously.Test plan
flushedstep, power cut)🤖 Generated with Claude Code