Skip to content

csnode, client: snapshot importer and --import-snapshot - #92

Merged
EnzinoBB merged 11 commits into
masterfrom
feat/snapshot-importer
Oct 24, 2026
Merged

EnzinoBB merged 11 commits into
masterfrom
feat/snapshot-importer

Conversation

@EnzinoBB

Copy link
Copy Markdown
Collaborator

Step 3c of snapshots served by community nodes.

Importer (cs::importSnapshot) restores a format-2 snapshot into a node folder whose database is empty:

  1. every file must match the manifest: size, the SHA-256 of every chunk and of the whole file;
  2. qs/<S> is written from the checkpoint files, with sizes capped at the declared raw_size (at most 4 GB). head.bin must carry the manifest's block and hash, and state_digest.bin its state and settings digests;
  3. each block must pass BlockVerifier (signatures against its confidants) and the hash chain that ends at the manifest's head. Then it is written to a new database of the configured backend (lmdb or rocksdb; berkeleydb is refused because it cannot start above block 0), together with caches/hashdb and caches/seqdb;
  4. the contract data is written, then snapshot.json.

On any failure, everything the import created is removed: only its own database files, not db/ with the node's lock. The node then quick-starts from qs/<S> and rebuilds the transaction index, as after a format-1 restore.

--import-snapshot <snapshot.json> restores from a local folder. It exits 0, or 2 with the reason, as snapshot-restore.sh does.

Not checked by the importer: quick_start_hashes.dat. It hashes the loaded objects, not the files, so the node checks it at quick start as it does today, together with the checkpoint digest of #87.

Tests: round trip builder → importer on LMDB and RocksDB with signed blocks (blocks byte for byte, contract data, hash databases, checkpoint files); BerkeleyDB refused; altered signature in block 120 (rejected, nothing left); a validly signed manifest that lies about the state digest or the head; truncated or altered blocks file; oversized raw_size; an existing database refused and left intact.

CI: snapshot-join / snapshot-corrupt now use export + import (format 2). New: snapshot-join-from-berkeleydb (pruned BerkeleyDB → pruned LMDB), snapshot-join-rocksdb-full (pruned RocksDB → full RocksDB), snapshot-join-format1 (old scripts, kept for the previous image).

🤖 Generated with Claude Code

EnzinoBB and others added 11 commits October 24, 2026 12:36
A format-2 snapshot is restored into an empty node folder: every file
must match the manifest (size, chunk and whole-file SHA-256); qs/<S> is
written from the checkpoint files, whose head and state digest must be
the manifest's; every block passes the BlockVerifier and the hash chain
ending at the manifest's head before it is written to a new database of
the configured backend (lmdb or rocksdb) with caches/hashdb and seqdb;
then the contract data and snapshot.json. Any failure removes what the
import created. The node then quick-starts from qs/<S> and rebuilds its
transaction index.

--import-snapshot <snapshot.json> restores from a local folder and exits
0, or 2 with the reason.

CI: the snapshot test uses --export-snapshot and --import-snapshot
(SNAPSHOT_FORMAT=1 keeps the scripts); new scenarios export from a
pruned BerkeleyDB node into a pruned LMDB one and from a pruned RocksDB
node into a full RocksDB one.

Step 3c of snapshots served by community nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Security review of the importer:
- every block's round confirmations must verify against the previous
  block's confidants (the node's own rule for a stored block), so a
  forged range needs the keys of every round's confidants;
- blocks.bin and contracts.bin are capped at their declared raw_size and
  a contract key or value at 64 MB before anything is buffered;
- the header states what the import proves and what anchors it (the
  agreement of sources, then the network, and the quick-start check).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…works can run the snapshot test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng the engine

A pruned RocksDB node whose blocks were still in the write-ahead log
was not recognised: detection opened the engine a second time in the
process to look for blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gest

run.sh's digest_offline runs in a subshell, where wait cannot wait for
the node (not its child): the node was still stopping while the offline
start and the export opened its database, which RocksDB's lock refuses.
Both scripts now wait for the start-up line, not the digest a quick
start logs for the checkpoint it loaded, before stopping the node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The checkpoint is written to qs/<S>.tmp, which the node's checkpoint
  manager removes at its start: qs/<S>.import was read as version S.
- The database and the hash databases are closed before a failed
  import removes their files (RocksDB writes on close); a hash database
  failure fails the import and its rebuild marker is removed too.
- The decompressed checkpoint files, the qs folder, snapshot.json and
  the node folder are synced to disk before the import reports success.
- --import-snapshot creates db/ first, so that it holds db/.lock while it
  writes.
- Test: a failed RocksDB import leaves no table or CURRENT file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB merged commit c36f902 into master Oct 24, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant