Repository navigation
csnode, csdb: pruned storage - #22
Merged
Merged
Conversation
A node can now keep only the blocks it needs. With [storage] prune_keep_blocks=N (0, the default, keeps every block), after each periodic checkpoint C the blocks before C - checkpoint_every - N are removed: the previous checkpoint stays usable, and N keeps a margin above the 100 rounds contracts refer back to. The quick-start checkpoint already holds the whole state, so these blocks are no longer needed to validate or apply new ones. - csdb: Storage::pool_prune removes an old block (block record and hash index), Storage::first_sequence reports the oldest block stored, and Storage::compact gives the freed pages back to the file system (DB->compact with DB_FREE_SPACE; a no-op default in Database). - BlockChain schedules the pruning after a periodic checkpoint and carries it out 64 blocks at a time after each block applied live, then compacts. While the storage is read at start-up it only schedules, as a cursor is walking the file. - A pruned storage starts after block 0, so Storage::rescan refuses a slow start on it (it would replay the remaining blocks onto empty caches) and asks for a quick-start checkpoint instead. - After a quick start, a contract state whose block was pruned is kept as unverifiable instead of being invalidated. - A pruned node does not take the Trusted role for now. The config needs checkpoint_every, raises checkpoint_keep to 2 and prune_keep_blocks to 200 at least. Nodes without prune_keep_blocks behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refusing the role once the network has picked the node does not make it an observer: the round waits for its stages. On the test network, with the pruned node in the initial trusted set, round 1 never completed and no block was written. A pruned node has the whole state (its checkpoint) and the recent blocks, which is what validating and building blocks needs, so it now takes the role as any other node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LMDB transaction index (one entry per address per block) and the sequence/hash tables of BlockHashes still grew with the whole history on a pruned node. pruneStep now loads each block before removing it, drops its transaction index entries (TransactionsIndex::onPruneBlock) and its BlockHashes entries. Neither is needed for blocks before the pruning point: - the transaction index serves the API's walk back through an address's history, which now ends at the oldest block stored: getPrevTransBlock answers kWrongSequence for a missing entry, as for an address's first transaction, without flagging the index as inconsistent; - consensus reads block hashes of recent blocks only (the contract seed of the current block, the previous round in stage 3, checkpoint heads); the random old block GetSeed picks for getters, outside consensus, now hashes an empty value. LMDB reuses the freed pages, so both stop growing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Update: the transaction index and the BlockHashes tables are now pruned too (commit The test network found a crash: the pruned node segfaulted as soon as a joining node asked it for blocks it had pruned. The cause is not in this PR. The With #24, run 38104614549 passed. It combined pruned n7, restarted at block ~2,500, with n8 joining from block 0 at block 3,526 and requesting pruned blocks from its peers. All 8 nodes report identical digests to block 4,600.
Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A node can keep only the blocks it needs. The option is
[storage] prune_keep_blocks=N, off by default (0); nodes without it behave exactly as before.The quick-start checkpoint already holds the whole state: balances, delegations and contracts. Validating and applying a new block then needs only the recent blocks; contracts refer back at most 100 rounds. So after each periodic checkpoint
C, the node removes the blocks beforeC - checkpoint_every - N. This keeps the previous checkpoint usable, plus a margin ofNblocks.Storage::pool_pruneremoves an old block: the block record and its hash index.Storage::first_sequencereports the oldest block still stored.Storage::compactgives the freed pages back to the file system (DB->compactwithDB_FREE_SPACE). The newDatabase::compactdefaults to a no-op.BlockChainschedules the pruning after a periodic checkpoint. It carries it out 64 blocks at a time after each block applied live, then compacts. While the storage is read at start-up it only schedules, because a cursor is walking the file.Storage::rescanrefuses it and asks for a quick-start checkpoint instead.checkpoint_every. It raisescheckpoint_keepto at least 2 andprune_keep_blocksto at least 200.A pruned node takes the Trusted role like any other node. An earlier version refused it, and the test network showed why that cannot work: the pruned node was in the initial trusted set, round 1 waited for its stages, and no block was ever written.
Limits (follow-ups)
Verification
nodetests: 134/134 pass.Private 7-node network (
testnet/prune-check= this branch + theprune_nodescenario). n7 runs pruned, with a checkpoint every 1,000 blocks andprune_keep_blocks=300, and is restarted at block ~2,520.blockchain.dbis 1,608 KB, against 4,420 KB on a full node;Test-network blocks are nearly empty, so the BerkeleyDB log files weigh as much as the data there.
🤖 Generated with Claude Code