Repository navigation
csnode, client: snapshot importer and --import-snapshot - #92
Merged
Merged
Conversation
A format-2 snapshot is restored into an empty node folder: every file must match the manifest (size, chunk and whole-file SHA-256); qs/<S> is written from the checkpoint files, whose head and state digest must be the manifest's; every block passes the BlockVerifier and the hash chain ending at the manifest's head before it is written to a new database of the configured backend (lmdb or rocksdb) with caches/hashdb and seqdb; then the contract data and snapshot.json. Any failure removes what the import created. The node then quick-starts from qs/<S> and rebuilds its transaction index. --import-snapshot <snapshot.json> restores from a local folder and exits 0, or 2 with the reason. CI: the snapshot test uses --export-snapshot and --import-snapshot (SNAPSHOT_FORMAT=1 keeps the scripts); new scenarios export from a pruned BerkeleyDB node into a pruned LMDB one and from a pruned RocksDB node into a full RocksDB one. Step 3c of snapshots served by community nodes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Security review of the importer: - every block's round confirmations must verify against the previous block's confidants (the node's own rule for a stored block), so a forged range needs the keys of every round's confidants; - blocks.bin and contracts.bin are capped at their declared raw_size and a contract key or value at 64 MB before anything is buffered; - the header states what the import proves and what anchors it (the agreement of sources, then the network, and the quick-start check). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…works can run the snapshot test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng the engine A pruned RocksDB node whose blocks were still in the write-ahead log was not recognised: detection opened the engine a second time in the process to look for blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gest run.sh's digest_offline runs in a subshell, where wait cannot wait for the node (not its child): the node was still stopping while the offline start and the export opened its database, which RocksDB's lock refuses. Both scripts now wait for the start-up line, not the digest a quick start logs for the checkpoint it loaded, before stopping the node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The checkpoint is written to qs/<S>.tmp, which the node's checkpoint manager removes at its start: qs/<S>.import was read as version S. - The database and the hash databases are closed before a failed import removes their files (RocksDB writes on close); a hash database failure fails the import and its rebuild marker is removed too. - The decompressed checkpoint files, the qs folder, snapshot.json and the node folder are synced to disk before the import reports success. - --import-snapshot creates db/ first, so that it holds db/.lock while it writes. - Test: a failed RocksDB import leaves no table or CURRENT file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 3c of snapshots served by community nodes.
Importer (
cs::importSnapshot) restores a format-2 snapshot into a node folder whose database is empty:qs/<S>is written from the checkpoint files, with sizes capped at the declaredraw_size(at most 4 GB).head.binmust carry the manifest's block and hash, andstate_digest.binits state and settings digests;BlockVerifier(signatures against its confidants) and the hash chain that ends at the manifest's head. Then it is written to a new database of the configured backend (lmdb or rocksdb; berkeleydb is refused because it cannot start above block 0), together withcaches/hashdbandcaches/seqdb;snapshot.json.On any failure, everything the import created is removed: only its own database files, not
db/with the node's lock. The node then quick-starts fromqs/<S>and rebuilds the transaction index, as after a format-1 restore.--import-snapshot <snapshot.json>restores from a local folder. It exits 0, or 2 with the reason, assnapshot-restore.shdoes.Not checked by the importer:
quick_start_hashes.dat. It hashes the loaded objects, not the files, so the node checks it at quick start as it does today, together with the checkpoint digest of #87.Tests: round trip builder → importer on LMDB and RocksDB with signed blocks (blocks byte for byte, contract data, hash databases, checkpoint files); BerkeleyDB refused; altered signature in block 120 (rejected, nothing left); a validly signed manifest that lies about the state digest or the head; truncated or altered blocks file; oversized
raw_size; an existing database refused and left intact.CI:
snapshot-join/snapshot-corruptnow use export + import (format 2). New:snapshot-join-from-berkeleydb(pruned BerkeleyDB → pruned LMDB),snapshot-join-rocksdb-full(pruned RocksDB → full RocksDB),snapshot-join-format1(old scripts, kept for the previous image).🤖 Generated with Claude Code