Repository navigation
csdb: compaction of pruned storage can no longer terminate the node - #25
Merged
Merged
Conversation
On mainnet the first pruning cycle of a node syncing from block 0 (blocks before 99'000 removed at the 200'000 checkpoint) terminated the node: compacting the 600 MB blockchain.db in one Db::compact call needed more of the environment's buffer cache than it holds, and BerkeleyDB threw (BDB3017 unable to allocate space from the buffer cache; terminate called after throwing an instance of 'DbException', what(): Db::compact: Cannot allocate memory). DatabaseBerkeleyDB::compact now works through each database 64 pages per call, resuming from the key the previous call stopped at, and turns any exception into an error result: compaction only returns space to the file system (BerkeleyDB reuses freed pages anyway), so pruneStep logs the failure and the node carries on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BerkeleyDB's default buffer cache (about 64 pages of 4 KB) is usually full of freshly written pages while a node syncs, and Db::compact then cannot allocate pages even in small steps (seen on mainnet at the 300'000 checkpoint). Nothing is wrong with the storage: compaction only returns space, and the pages freed by pruning are reused for new blocks (blockchain.db grew from 207 to 232 MB while 100'000 blocks were added and 100'000 removed). Log it as a warning instead of setting the storage error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On mainnet a pruned node syncing fast terminated in Db::del: BerkeleyDB's default buffer cache was exhausted (BDB3017) and the exception escaped DatabaseBerkeleyDB::remove. remove now turns any exception into an error result, and pruneStep keeps a block it could not remove and retries it on the next step instead of counting it as pruned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A pruned node (#22) was started on mainnet from block 0, as an observer in a container. Its first pruning cycle terminated it: at the 200,000 checkpoint it removed the blocks before 99,000 (fine), then compacted the ~600 MB
blockchain.dbin a singleDb::compactcall:The test network never showed this, because its databases are a few MB.
Fix
DatabaseBerkeleyDB::compact: works through each database 64 pages per call, resuming from the key where the previous call stopped. Any exception becomes an error result; it can no longer terminate the node.Storage::compact: a failure is logged as a warning. It is not a storage fault: compaction only returns space to the file system.What mainnet shows with the fix
The node restarted from its pruned storage ("oldest block stored is 99'000") and kept syncing.
blockchain.dbgrew from 207 MB to 232 MB while 100,000 blocks were added and 100,000 removed.Not changed here: making compaction succeed during sync would need a larger BerkeleyDB cache (
set_cachesize), which changes every node's memory use. That needs measuring first.🤖 Generated with Claude Code