Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/robustness.yml
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,16 @@ jobs:
env: DB_BACKEND=lmdb PRUNE_NODE=6 SNAPSHOT_AT=2300 TARGET_SEQ=3100
- name: snapshot-corrupt
env: DB_BACKEND=lmdb PRUNE_NODE=6 SNAPSHOT_AT=2300 SNAPSHOT_CORRUPT=1 TARGET_SEQ=3100
# format 2 across block databases: exported by a pruned BerkeleyDB node into a pruned LMDB one,
# and by a pruned RocksDB node into a full RocksDB one; format 1 kept for the previous image
- name: snapshot-join-from-berkeleydb
env: NODE_STORAGE=berkeleydb,lmdb,rocksdb,berkeleydb:pruned,lmdb:pruned,rocksdb:pruned,berkeleydb,lmdb:pruned PRUNE_NODE=4 SNAPSHOT_AT=2300 TARGET_SEQ=3300
fixed: true
- name: snapshot-join-rocksdb-full
env: NODE_STORAGE=berkeleydb,lmdb,rocksdb,berkeleydb:pruned,lmdb:pruned,rocksdb:pruned,berkeleydb,rocksdb PRUNE_NODE=6 JOIN_FULL=1 SNAPSHOT_AT=2300 TARGET_SEQ=3300
fixed: true
- name: snapshot-join-format1
env: DB_BACKEND=lmdb PRUNE_NODE=6 SNAPSHOT_AT=2300 SNAPSHOT_FORMAT=1 TARGET_SEQ=3100
# node 5 from full to pruned, node 6 from pruned to full, after node 6 pruned once
- name: storage-switch
env: DB_BACKEND=lmdb PRUNE_NODE=6 STORAGE_SWITCH_AT=2300 TARGET_SEQ=3300
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/scale.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ on:
timeout_min:
description: 'Minutes each network has to reach its target sequence'
default: '60'
scenario:
description: 'More run.sh variables as NAME=value pairs, as a robustness scenario (e.g. PRUNE_NODE=6 SNAPSHOT_AT=2300 TARGET_SEQ=3300)'
default: ''

permissions:
contents: read
Expand Down Expand Up @@ -210,10 +213,17 @@ jobs:
FUND: '1'
LOAD_SECONDS: '300'
WORK: ${{ github.workspace }}/testnet-run
SCENARIO: ${{ inputs.scenario || '' }}
run: |
chmod +x build/node
IFS=, read -r CONTRACT_LOAD_SECONDS CONTRACT_WRITE_RATE CONTRACT_READ_RATE <<< "$CONTRACT_LOAD"
export CONTRACT_LOAD_SECONDS CONTRACT_WRITE_RATE CONTRACT_READ_RATE
# NAME=value pairs only, as the robustness scenarios
for kv in $SCENARIO; do
echo "$kv" | grep -qE '^[A-Z_]+=[a-z0-9,:]+$' || { echo "bad scenario entry $kv"; exit 1; }
export "$kv"
done
[ -z "$SCENARIO" ] || echo "scenario: $SCENARIO"
ci/testnet/run.sh build/node 2>&1 | tee testnet-run.log
exit "${PIPESTATUS[0]}"

Expand Down
3 changes: 2 additions & 1 deletion ci/snapshot/digest_offline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ awk '
{ print }' config.ini > config-offline.ini
# the node's console output of this run (the packaged config.ini logs info to the console): a count
# of digest lines in log/ can go down when the start rotates the logs and removes the oldest file
digests() { grep -a 'STATE DIGEST #' offline.out 2>/dev/null || true; }
# the start-up line only: a quick start also logs the digest of the checkpoint it loaded
digests() { grep -aE 'STATE DIGEST #[0-9]+ [0-9A-Fa-f]+ wallets [0-9]+ *$' offline.out 2>/dev/null || true; }
./node --db-path db --config-file config-offline.ini </dev/null > offline.out 2>&1 &
pid=$!
deadline=$(( $(date +%s) + timeout ))
Expand Down
63 changes: 45 additions & 18 deletions ci/testnet/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@
# precedence over DB_BACKEND, and every pruned node must prune
# SNAPSHOT_AT (default 0 = off, needs PRUNE_NODE): at that sequence the pruned node is stopped,
# started once without peers to log the digest of its head, packaged with
# packaging/docker/snapshot-make.sh and started again; one more node (NODES + 1) restores the
# snapshot with snapshot-restore.sh and joins; with SNAPSHOT_CORRUPT=1 the archive is truncated
# first, so the restore must fail and that node syncs from block 0
# --export-snapshot (format 2) and started again; one more node (NODES + 1) restores the
# snapshot with --import-snapshot into its own backend and joins; with SNAPSHOT_CORRUPT=1 the
# blocks file is truncated first, so the restore must fail and that node syncs from block 0.
# SNAPSHOT_FORMAT=1: packaging/docker/snapshot-make.sh and snapshot-restore.sh instead
# STORAGE_SWITCH_AT (default 0 = off, needs PRUNE_NODE=6 and DB_BACKEND=lmdb): at that sequence
# node 5 is restarted as a pruned node and node 6 as a full one
# CONVERT_AT (default 0 = off, needs DB_BACKEND=berkeleydb): at that sequence n4 is converted to
Expand Down Expand Up @@ -190,6 +191,7 @@ CRASH_ALL_AT=${CRASH_ALL_AT:-0}
JOIN_PRUNED_AT=${JOIN_PRUNED_AT:-0}
# the joining node is node NODES + 1; it is in nobody's hosts or trusted list
SNAPSHOT_AT=${SNAPSHOT_AT:-0}
SNAPSHOT_FORMAT=${SNAPSHOT_FORMAT:-2}
SNAPSHOT_CORRUPT=${SNAPSHOT_CORRUPT:-0}
STORAGE_SWITCH_AT=${STORAGE_SWITCH_AT:-0}
CONVERT_AT=${CONVERT_AT:-0}
Expand Down Expand Up @@ -472,7 +474,9 @@ seq_of() {
# STATE DIGEST is the digest of its head, and stop it again
digest_offline() {
local i=$1 dir="$WORK/n$1" before
kill -TERM "${PIDS[$i]}"; wait "${PIDS[$i]}" 2>/dev/null || true
# this runs in a subshell ($(...)): wait cannot wait for the node, which is not its child
kill -TERM "${PIDS[$i]}"
for _ in $(seq 1 180); do kill -0 "${PIDS[$i]}" 2>/dev/null || break; sleep 1; done
pkill -f "cs.testnet.node=n$i " 2>/dev/null || true
# an empty hosts file stops the node ("No hosts specified"): one host nobody listens on
echo "127.0.0.1:9 11111111111111111111111111111111" > "$dir/hosts-none.txt"
Expand All @@ -481,18 +485,22 @@ digest_offline() {
/^hosts_filename=/ { print "hosts_filename=hosts-none.txt"; next }
section == "[host_input]" && /^port=/ { print "port=" port; next }
{ print }' "$dir/config.ini" > "$dir/config-offline.ini"
before=$(grep -ac 'STATE DIGEST #' "$dir/node.log" || true)
# the start-up line only ("... wallets <n>" at the end), not the checkpoint lines
local startup='STATE DIGEST #[0-9]+ [0-9A-Fa-f]+ wallets [0-9]+ *$'
before=$(grep -acE "$startup" "$dir/node.log" || true)
(cd "$dir" && exec "$BIN" --db-path db --config-file config-offline.ini </dev/null >> stdout.log 2>&1) &
local pid=$! deadline=$(( $(date +%s) + 300 ))
while [ "$(grep -ac 'STATE DIGEST #' "$dir/node.log" || true)" -le "${before:-0}" ] && [ "$(date +%s)" -lt "$deadline" ]; do
while [ "$(grep -acE "$startup" "$dir/node.log" || true)" -le "${before:-0}" ] && [ "$(date +%s)" -lt "$deadline" ]; do
sleep 2
done
kill -TERM "$pid"; wait "$pid" 2>/dev/null || true
if [ "$(grep -ac 'STATE DIGEST #' "$dir/node.log" || true)" -le "${before:-0}" ]; then
for _ in $(seq 1 180); do kill -0 "$pid" 2>/dev/null || break; sleep 1; done
pkill -f "[c]ontract-executor.*n$i/" 2>/dev/null || true
if [ "$(grep -acE "$startup" "$dir/node.log" || true)" -le "${before:-0}" ]; then
echo "FAIL: n$i logged no start-up digest without peers" >&2
return 1
fi
grep -a 'STATE DIGEST #' "$dir/node.log" | tail -1 | sed 's/^.*STATE DIGEST/STATE DIGEST/'
grep -aE "$startup" "$dir/node.log" | tail -1 | sed 's/^.*STATE DIGEST/STATE DIGEST/'
}

# the lowest sequence among the nodes of the restart test
Expand Down Expand Up @@ -670,14 +678,19 @@ while :; do
snapshot_done=1
head_digest=$(digest_offline "$PRUNE_NODE") || exit 1
echo "snapshot test: n$PRUNE_NODE stopped, head $head_digest"
"$REPO/packaging/docker/snapshot-make.sh" "$WORK/n$PRUNE_NODE" "$WORK/snapshot" "$MDB_COPY" local \
| sed 's/^/snapshot test: /'
if [ "$SNAPSHOT_FORMAT" = 1 ]; then
"$REPO/packaging/docker/snapshot-make.sh" "$WORK/n$PRUNE_NODE" "$WORK/snapshot" "$MDB_COPY" local \
| sed 's/^/snapshot test: /'
fi
# format 2: the node builds the snapshot of its latest periodic checkpoint itself; the
# manifest must declare the digest the checkpoint logged when it was saved
# who still holds the database files (RocksDB's LOCK allows one process)
{ fuser -v "$WORK/n$PRUNE_NODE"/db/LOCK "$WORK/n$PRUNE_NODE"/db/lmdb/lock.mdb 2>&1 || true; } | sed 's/^/snapshot test: holder: /'
ps -eo pid,ppid,etimes,args | grep -a "n$PRUNE_NODE\b\|contract-executor" | grep -v grep | sed 's/^/snapshot test: process: /' || true
rc=0
(cd "$WORK/n$PRUNE_NODE" && exec "$BIN" --db-path db --config-file config.ini --export-snapshot "$WORK/snapshot2") \
> "$WORK/export.log" 2>&1 || rc=$?
grep -a 'export-snapshot:' "$WORK/export.log" | sed 's/^/snapshot test: /'
grep -a 'export-snapshot:\|rocksdb: cannot' "$WORK/export.log" | sed 's/^/snapshot test: /'
[ "$rc" = 0 ] || { tail -20 "$WORK/export.log"; echo "FAIL: --export-snapshot exited $rc"; exit 1; }
python3 - "$WORK/snapshot2" "$WORK/n$PRUNE_NODE/node.log" <<'PY' || exit 1
import json, os, re, sys
Expand All @@ -693,14 +706,28 @@ if saved.get(m["seq"]) != (m["state_digest"], m["wallets"]):
print(f"snapshot test: format-2 snapshot of #{m['seq']} ({m['first_block']}..{m['seq']}), digest as logged at the checkpoint, {len(m['files'])} files")
PY
start_node "$PRUNE_NODE"
if [ "$SNAPSHOT_CORRUPT" = 1 ]; then
f=$(ls "$WORK"/snapshot/credits-pruned-*.tar.zst)
head -c $(( $(stat -c %s "$f") / 2 )) "$f" > "$f.cut" && mv "$f.cut" "$f"
echo "snapshot test: archive truncated"
fi
rc=0
"$REPO/packaging/docker/snapshot-restore.sh" "$WORK/n$JOIN_NODE" "$WORK/snapshot/latest.json" \
> "$WORK/restore.log" 2>&1 || rc=$?
if [ "$SNAPSHOT_FORMAT" = 1 ]; then
if [ "$SNAPSHOT_CORRUPT" = 1 ]; then
f=$(ls "$WORK"/snapshot/credits-pruned-*.tar.zst)
head -c $(( $(stat -c %s "$f") / 2 )) "$f" > "$f.cut" && mv "$f.cut" "$f"
echo "snapshot test: archive truncated"
fi
"$REPO/packaging/docker/snapshot-restore.sh" "$WORK/n$JOIN_NODE" "$WORK/snapshot/latest.json" \
> "$WORK/restore.log" 2>&1 || rc=$?
else
manifest=$(ls "$WORK"/snapshot2/*/snapshot.json)
if [ "$SNAPSHOT_CORRUPT" = 1 ]; then
f="$(dirname "$manifest")/blocks.bin.lz4"
head -c $(( $(stat -c %s "$f") / 2 )) "$f" > "$f.cut" && mv "$f.cut" "$f"
echo "snapshot test: blocks.bin.lz4 truncated"
fi
# the joining node restores the snapshot itself into its configured backend
(cd "$WORK/n$JOIN_NODE" && exec "$BIN" --db-path db --config-file config.ini --import-snapshot "$manifest") \
> "$WORK/restore.log" 2>&1 || rc=$?
grep -a 'import-snapshot:' "$WORK/restore.log" > "$WORK/restore.short" || true
mv "$WORK/restore.short" "$WORK/restore.log"
fi
sed 's/^/snapshot test: /' "$WORK/restore.log"
echo "snapshot test: restore exit $rc; n$JOIN_NODE joins at sequence $min"
if [ "$SNAPSHOT_CORRUPT" = 1 ] && [ "$rc" != 2 ]; then echo "FAIL: a truncated snapshot was restored"; exit 1; fi
Expand Down
6 changes: 6 additions & 0 deletions client/src/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ int main(int argc, char* argv[]) {
("convert-only", "convert the block database to the configured db_backend, then exit")
("convert-batch", po::value<uint32_t>()->default_value(100000), "blocks per conversion batch (1 to 1000000)")
("export-snapshot", po::value<std::string>(), "build the snapshot of the latest periodic checkpoint into the given folder, then exit (the node must be stopped)")
("import-snapshot", po::value<std::string>(), "restore the snapshot of the given snapshot.json into this node's empty database, then exit")
("config-file", po::value<std::string>(), "path to configuration file (default: \"config.ini\")")
("public-key-file", po::value<std::string>(), "path to public key file (default: \"NodePublic.txt\")")
("private-key-file", po::value<std::string>(), "path to private key file (default: \"NodePrivate.txt\")")
Expand Down Expand Up @@ -199,6 +200,11 @@ int main(int argc, char* argv[]) {
}
}

// no keys needed: the import writes only the database, the caches and the checkpoint
if (vm.count("import-snapshot") > 0) {
return snapshot_export::runImport(config, vm["import-snapshot"].as<std::string>());
}

if (vm.count(cmdline::argSeed) == 0) {
if (!config.readKeys(vm)) {
return EXIT_FAILURE;
Expand Down
43 changes: 42 additions & 1 deletion client/src/snapshot_export.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
#include <csdb/database.hpp>
#include <csdb/storage_backend.hpp>
#include <csnode/snapshot_builder.hpp>
#include <csnode/snapshot_importer.hpp>
#include <csnode/snapshot_manifest.hpp>
#include <csnode/statedigest.hpp>

namespace snapshot_export {
Expand Down Expand Up @@ -72,7 +74,9 @@ int run(const Config& config, const std::string& outDir) {
say() << "a node is running on " << dbPath << ": stop it first" << std::endl;
return 1;
}
const auto found = csdb::backend::detect(dbPath, true);
// from the files alone: the start-up check has already decided the folder holds the configured
// backend, and opening an engine twice in one process can fail on its own lock
const auto found = csdb::backend::detect(dbPath, false);
if (std::find(found.begin(), found.end(), storage.dbBackend) == found.end()) {
say() << dbPath << " holds no " << storage.dbBackend << " database (the configured db_backend)" << std::endl;
return 1;
Expand Down Expand Up @@ -141,4 +145,41 @@ int run(const Config& config, const std::string& outDir) {
return 0;
}

int runImport(const Config& config, const std::string& manifestFile) {
auto sayImport = []() -> std::ostream& { return std::cout << "import-snapshot: "; };
const std::string dbPath = config.getPathToDB();
// db/ first, so that the import holds db/.lock while it writes: a node started in this folder
// meanwhile finds it in use
{
std::error_code ec;
fs::create_directories(dbPath, ec);
}
if (dbcheck::lockedByAnotherProcess(dbPath)) {
sayImport() << "a node is running on " << dbPath << ": stop it first" << std::endl;
return 2;
}
std::ifstream in(manifestFile, std::ios::binary);
if (!in) {
sayImport() << "cannot read " << manifestFile << std::endl;
return 2;
}
std::string json((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
std::string error;
const auto manifest = cs::SnapshotManifest::fromJson(json, &error);
if (!manifest) {
sayImport() << "not restored: the manifest is refused: " << error << std::endl;
return 2;
}
const auto backend = config.getStorageSettings().dbBackend;
sayImport() << "block " << manifest->seq << " (blocks " << manifest->firstBlock << ".." << manifest->seq << "), state digest "
<< manifest->stateDigest << ", signed by " << manifest->nodeKey << ", into a new " << backend << " database" << std::endl;
const auto r = cs::importSnapshot(fs::path(manifestFile).parent_path(), *manifest, fs::current_path(), dbPath, backend);
if (!r.ok) {
sayImport() << "not restored: " << r.error << (r.fileAtFault.empty() ? "" : " (" + r.fileAtFault + ")") << std::endl;
return 2;
}
sayImport() << "restored block " << manifest->seq << std::endl;
return 0;
}

} // namespace snapshot_export
5 changes: 5 additions & 0 deletions client/src/snapshot_export.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,9 @@ namespace snapshot_export {
// state_digest.bin) of this stopped node into <dir>/<seq>, signed with the node's key. 0 on success.
int run(const Config& config, const std::string& outDir);

// --import-snapshot <snapshot.json>: restores the format-2 snapshot whose manifest and files are in
// that folder into this node's empty database (configured db_backend, lmdb or rocksdb). 0 on
// success, 2 when nothing was restored (the reason is printed), as snapshot-restore.sh.
int runImport(const Config& config, const std::string& manifestFile);

} // namespace snapshot_export
7 changes: 6 additions & 1 deletion csdb/src/storage_backend.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
#include <cctype>
#include <cstdio>
#include <fstream>
#include <iostream>
#include <fcntl.h>
#include <unistd.h>

Expand Down Expand Up @@ -120,7 +121,11 @@ std::shared_ptr<Database> open(const std::string& name, const std::string& dir)
#if defined(CSDB_USE_ROCKSDB)
if (name == "rocksdb") {
auto db = std::make_shared<DatabaseRocksDB>();
return db->open(dir) ? db : nullptr;
if (db->open(dir)) {
return db;
}
std::cerr << "rocksdb: cannot open " << dir << ": " << db->last_error_message() << std::endl;
return nullptr;
}
#endif
#if !defined(CSDB_USE_ROCKSDB) || defined(CSDB_USE_BERKELEYDB)
Expand Down
2 changes: 2 additions & 0 deletions csnode/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ add_library(csnode
include/csnode/statedigest.hpp
include/csnode/snapshot_manifest.hpp
include/csnode/snapshot_builder.hpp
include/csnode/snapshot_importer.hpp
include/csnode/blockverifier.hpp
include/csnode/sendcachedata.hpp
include/csnode/eventreport.hpp
Expand Down Expand Up @@ -114,6 +115,7 @@ add_library(csnode
src/statedigest.cpp
src/snapshot_manifest.cpp
src/snapshot_builder.cpp
src/snapshot_importer.cpp
src/blockverifier.cpp
src/sendcachedata.cpp
src/eventreport.cpp
Expand Down
6 changes: 4 additions & 2 deletions csnode/include/csnode/snapshot_builder.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,11 @@ struct SnapshotBuildResult {
SnapshotBuildResult buildSnapshot(const SnapshotBuildInput& input, const std::filesystem::path& outRoot,
const cs::PrivateKey& key);

// the blocks.bin record stream: "u32 seq, u32 len, bytes" (little-endian)
// the blocks.bin record stream: "u32 seq, u32 len, bytes" (little-endian); false beyond maxRawBytes
// decompressed bytes (a manifest's raw_size), for a record above 64 MB, or a truncated stream
bool forEachSnapshotBlock(const std::filesystem::path& blocksFile,
const std::function<bool(cs::Sequence, cs::Bytes&&)>& visit);
const std::function<bool(cs::Sequence, cs::Bytes&&)>& visit,
uint64_t maxRawBytes = UINT64_MAX);

} // namespace cs

Expand Down
38 changes: 38 additions & 0 deletions csnode/include/csnode/snapshot_importer.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#ifndef SNAPSHOT_IMPORTER_HPP
#define SNAPSHOT_IMPORTER_HPP

#include <atomic>
#include <filesystem>
#include <string>

#include <csnode/snapshot_manifest.hpp>

namespace cs {

struct SnapshotImportResult {
bool ok = false;
std::string error;
std::string fileAtFault; // the snapshot file that failed, empty when the fault is not a file's
};

// Restores a format-2 snapshot (manifest m, its files in srcDir, all downloaded) into a node folder
// whose database is empty: the files must match the manifest (sizes, whole-file and chunk hashes);
// qs/<seq> is written from the checkpoint files, whose head.bin and state_digest.bin must carry the
// manifest's block hash and digests; every block is checked with the BlockVerifier and as a hash
// chain ending at the manifest's head, then written to a new database of the given backend (lmdb or
// rocksdb) with caches/hashdb and caches/seqdb; then the contract data and nodeDir/snapshot.json.
// The node quick-starts from qs/<seq> and rebuilds its transaction index. Any failure removes
// everything the import created.
// Trust: this proves that the snapshot is consistent with itself (signed manifest, a hash chain of
// blocks signed by their confidants and confirmed by the previous round's, ending at the declared
// head, a checkpoint whose head and digests are the declared ones). It does not prove that the
// head is the network's: the caller anchors it, by the agreement of independent sources on
// (seq, head_hash, state_digest) before calling, and the node by continuing from that block with
// the network; the node also recomputes the state from the loaded caches at its quick start.
SnapshotImportResult importSnapshot(const std::filesystem::path& srcDir, const SnapshotManifest& m,
const std::filesystem::path& nodeDir, const std::filesystem::path& dbPath,
const std::string& backend, const std::atomic<bool>* stop = nullptr);

} // namespace cs

#endif // SNAPSHOT_IMPORTER_HPP
Loading
Loading