Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ crates/
├─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities
└─ test-fixtures/ # Spec-fixture loading (prod dep of rpc's Hive test driver)
net/
├─ api/ # Actor protocol traits wiring BlockChain ↔ P2P
├─ api/ # Actor protocol traits wiring BlockChain ↔ P2P, plus the shared sync-gate thresholds (`sync.rs`)
├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot, BlocksByRange, gloas envelopes)
│ ├─ src/gossipsub/ # Topic encoding, message handling
│ ├─ src/req_resp/ # Request/response codec and handlers
Expand Down Expand Up @@ -379,7 +379,7 @@ actual_slot = finalized_slot + 1 + relative_index
Both serve from the checkpoint-anchored store, and `build_status` advertises it
(head, finalized checkpoint, and the anchor's slot as `earliest_available_slot`), so peers
have a reason to ask. Asking runs too: a peer's Status starts a range session from the
higher of `P2PServer::beacon_fetched_through` (the highest slot a range answer handed to
higher of `P2PServer::range_fetched_through` (the highest slot a range answer handed to
the chain actor, since the store's head trails a delivered batch by the whole actor
mailbox) and the store's head (which gossip imports move and the watermark never sees).
Fetched blocks reach the actor as `BlockSource::Sync`.
Expand Down
268 changes: 212 additions & 56 deletions crates/blockchain/src/lib.rs

Large diffs are not rendered by default.

601 changes: 537 additions & 64 deletions crates/blockchain/src/sync_status.rs

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions crates/net/api/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ use spawned_concurrency::error::ActorError;
use spawned_concurrency::message::Message;
use spawned_concurrency::protocol;

pub mod sync;

// --- Protocol: BlockChain -> P2P ---

#[protocol]
Expand Down
145 changes: 145 additions & 0 deletions crates/net/api/src/sync.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
//! The sync gate's thresholds, shared by the two actors that act on them.
//!
//! The chain actor's sync tracker judges "behind" with these numbers, and the
//! p2p layer paces its `Status` refresh and gates the range syncs a refresh
//! answer may start by the same ones. Kept in one place so the two cannot
//! drift: a refresh cadence that no longer fits the lead it is meant to detect
//! is a silent sensitivity loss, not a compile error.

use ethlambda_types::beacon::preset::SLOTS_PER_EPOCH;

/// How far behind a node may fall before it reports itself syncing, per chain.
///
/// The node is syncing only when **both** of these hold:
///
/// - its head trails the wall clock by more than [`Self::clock_lag`], which is
/// local evidence no peer can forge, and
/// - the head its peers report is more than [`Self::peer_lead`] slots ahead of
/// its own.
///
/// This is Lighthouse's and Prysm's shape. Lighthouse serves duties unless a
/// range sync is running *and* the head is over `sync_tolerance_epochs` (2)
/// behind the clock, and only a peer more than 32 slots ahead starts a range
/// sync. Prysm resyncs only when the head is two epochs behind the clock *and*
/// peers report a head two epochs ahead. Peers decide between "the network is
/// ahead of me" and "the network is stalled"; the clock keeps a peer's claim
/// from idling a node whose own head is current.
///
/// A peer's head is a claim, and a claim that never turns into progress is
/// bounded by [`Self::stall_window`]: syncing means getting somewhere, and a
/// lead nobody delivers reads the same as a stalled network.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct SyncThresholds {
/// Slots the head may trail the wall clock before the node can be syncing.
pub clock_lag: u64,
/// Slots the peers' head must lead ours before the node can be syncing.
///
/// Also how far ahead a peer answering a `Status` *refresh* must be
/// before its answer starts a range sync. A handshake answer starts one
/// at any lead, but a refresh lands every few slots on a node that is
/// keeping up with gossip, where a peer a slot ahead is ordinary and a
/// one-block range request would only race gossip.
pub peer_lead: u64,
/// Recovery band: once syncing, both lags must drop this many slots below
/// their thresholds before the node reports synced again, so the status
/// does not flap at the boundary.
pub hysteresis: u64,
/// Slots after the first judgment during which the node assumes the
/// network is at least at the wall clock, unless [`Self::startup_peers`]
/// peers have reported before then.
///
/// The first judgment runs before any peer has answered a `Status`, and
/// the first peer to answer may itself be behind, so without this a node
/// resuming a stale database would read "no peer is ahead" as a stalled
/// network and do its duties on that stale head. Prysm holds a starting
/// node in initial sync until it has peers; this bounds that hold, so a
/// node that never finds a peer, or restarts into a real stall, still
/// falls back to treating the silence as a stall.
pub startup_grace: u64,
/// Connected peers whose `Status` ends the startup hold early.
///
/// Once this many have reported a head (one past the wall clock does
/// not count, one this node already holds does), the node judges by
/// their reports instead of the hold's assumption. One report is not
/// enough, since the first peer to answer may be as stale as this node,
/// but the highest of several is: a single current peer among them shows
/// the lead. A node with fewer peers than this keeps the whole
/// [`Self::startup_grace`] window. The same count on both chains.
pub startup_peers: usize,
/// Slots the node may read as syncing on a peer-reported lead without its
/// head advancing, before it treats the network as stalled after all.
///
/// A peer that claims a head it does not serve (an empty answer, invalid
/// blocks, no answer) would otherwise keep a node that is behind the
/// clock syncing for as long as it repeats the claim. Sized above what can
/// legitimately hold a catching-up head still: the range batch custody
/// wait (30 s), an execution client's retry ladder (about 26 s), lean
/// proof verification of a whole batch.
pub stall_window: u64,
/// How often the p2p layer re-asks each connected peer for its `Status`.
///
/// A peer's report is up to this many slots stale when the tracker reads
/// it, which shrinks the lead it shows by as much. [`Self::is_consistent`]
/// keeps that staleness inside the clock threshold, so a node the clock
/// says is behind still reads as behind its peers.
pub status_refresh_slots: u64,
}

impl SyncThresholds {
/// Lean consensus: leanSpec PR #708's lag threshold and hysteresis band,
/// with the stall override replaced by the peer lead (see
/// `docs/spec_deviations.md`). Lean has no epochs, so neither client's
/// epoch-sized tolerance carries over.
///
/// The peer lead is one slot rather than the clock's four because a
/// report is up to [`Self::status_refresh_slots`] old: a node five slots
/// behind a producing network sees peers that were current two slots ago
/// at least two slots ahead, which a lead threshold of one detects. A lead
/// of one is still not enough on its own: peers a single slot apart are
/// ordinary propagation, and a run of missed proposals leaves every peer
/// within that.
pub const LEAN: Self = Self {
clock_lag: 4,
peer_lead: 1,
hysteresis: 2,
startup_grace: 8,
startup_peers: 5,
stall_window: 16,
status_refresh_slots: 2,
};

/// The Beacon Chain: Lighthouse's numbers. Two epochs of clock lag
/// (`DEFAULT_SYNC_TOLERANCE_EPOCHS`, Prysm's `shouldReSync` too) and a peer
/// lead of one epoch (`SLOT_IMPORT_TOLERANCE`, the lead that makes a
/// Lighthouse peer `Advanced`). Neither client has a hysteresis band, but
/// neither re-judges on every block arrival either; this node does, and a
/// follower importing about one block a slot right at the boundary would
/// otherwise flip twice a slot. The startup hold and the stall window are
/// an epoch: finding beacon peers through discovery can take a while, and
/// Prysm holds until it has peers, however long; five peers reporting end
/// the hold sooner. The `Status` refresh is
/// Prysm's half epoch (`maintainPeerStatuses`).
pub const BEACON: Self = Self {
clock_lag: 2 * SLOTS_PER_EPOCH,
peer_lead: SLOTS_PER_EPOCH,
hysteresis: 4,
startup_grace: SLOTS_PER_EPOCH,
startup_peers: 5,
stall_window: SLOTS_PER_EPOCH,
status_refresh_slots: SLOTS_PER_EPOCH / 2,
};

/// Whether a report as stale as the refresh allows still shows the lead
/// of a node the clock says is behind: a node just past
/// [`Self::clock_lag`] behind peers that are a slot behind the clock and
/// last reported [`Self::status_refresh_slots`] ago sees a lead of
/// `clock_lag - status_refresh_slots`, which has to clear
/// [`Self::peer_lead`].
pub const fn is_consistent(&self) -> bool {
self.peer_lead + self.status_refresh_slots < self.clock_lag
&& self.hysteresis <= self.clock_lag
}
}

const _: () = assert!(SyncThresholds::LEAN.is_consistent());
const _: () = assert!(SyncThresholds::BEACON.is_consistent());
17 changes: 5 additions & 12 deletions crates/net/p2p/src/beacon/builder_market.rs
Original file line number Diff line number Diff line change
Expand Up @@ -180,13 +180,6 @@ pub(crate) fn publish_proposer_preferences(
);
}

/// The wall-clock slot, from the store's config.
pub(crate) fn wall_slot(server: &P2PServer) -> Slot {
let config = server.store.config();
let genesis_ms = config.genesis_time_ms();
unix_now_ms().saturating_sub(genesis_ms) / config.slot_duration_ms.max(1)
}

#[cfg(test)]
mod tests {
use ethlambda_state_transition::beacon::gossip::{IgnoreReason, Outcome, RejectReason};
Expand Down Expand Up @@ -273,7 +266,7 @@ mod tests {
#[tokio::test]
async fn a_far_slot_bid_is_not_current_or_next() {
let server = unconnected_beacon_server(gloas_from_genesis(), 0).await;
let bid = bid_at(wall_slot(&server) + 1_000);
let bid = bid_at(crate::wall_clock_slot(&server.store) + 1_000);
assert!(matches!(
triage_execution_payload_bid(&server, &bid.to_ssz()),
Dispatch::Report(Outcome::Ignore(IgnoreReason::NotCurrentOrNextSlot))
Expand All @@ -283,7 +276,7 @@ mod tests {
#[tokio::test]
async fn a_bid_with_a_nonzero_payment_is_rejected() {
let server = unconnected_beacon_server(gloas_from_genesis(), 0).await;
let mut bid = bid_at(wall_slot(&server) + 1);
let mut bid = bid_at(crate::wall_clock_slot(&server.store) + 1);
bid.message.execution_payment = 1;
assert!(matches!(
triage_execution_payload_bid(&server, &bid.to_ssz()),
Expand All @@ -297,7 +290,7 @@ mod tests {
#[tokio::test]
async fn a_valid_shaped_bid_is_validated_unless_its_builder_key_was_seen() {
let server = unconnected_beacon_server(gloas_from_genesis(), 0).await;
let bid = bid_at(wall_slot(&server) + 1);
let bid = bid_at(crate::wall_clock_slot(&server.store) + 1);
let payload = bid.to_ssz();
match triage_execution_payload_bid(&server, &payload) {
Dispatch::Validate(Validated::ExecutionPayloadBid {
Expand All @@ -320,7 +313,7 @@ mod tests {
#[tokio::test]
async fn preferences_are_judged_on_the_clock() {
let server = unconnected_beacon_server(gloas_from_genesis(), 0).await;
let wall = wall_slot(&server);
let wall = crate::wall_clock_slot(&server.store);
let past = preferences_at(wall.saturating_sub(5));
assert!(matches!(
triage_proposer_preferences(&server, &past.to_ssz()),
Expand All @@ -336,7 +329,7 @@ mod tests {
#[tokio::test]
async fn preferences_for_a_cached_key_are_already_seen() {
let server = unconnected_beacon_server(gloas_from_genesis(), 0).await;
let wall = wall_slot(&server);
let wall = crate::wall_clock_slot(&server.store);
let preferences = preferences_at(wall + 2);
let payload = preferences.to_ssz();
assert!(matches!(
Expand Down
39 changes: 26 additions & 13 deletions crates/net/p2p/src/beacon/handler.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@ use crate::req_resp::Request;
/// stream is eight bytes short and the codec refuses it, which killed the
/// connection outright. Answering a request means answering in its own version.
pub fn build_status(store: &Store, wire: &BeaconWire, version: StatusVersion) -> BeaconStatus {
status_in_version(&build_status_v2(store, wire), version)
}

/// [`build_status`] in v2, the version that carries every field, for a caller
/// that sends one `Status` in several versions (the periodic refresh) and so
/// should read the head, a whole-block decode, once.
pub fn build_status_v2(store: &Store, wire: &BeaconWire) -> StatusV2 {
let (finalized_root, finalized_epoch, head_root, head_slot, earliest_available_slot) =
match store.beacon_head() {
Some((head_slot, head_root)) => {
Expand All @@ -72,23 +79,27 @@ pub fn build_status(store: &Store, wire: &BeaconWire, version: StatusVersion) ->
None => (Root::ZERO, 0, Root::ZERO, 0, 0),
};

let v1 = StatusV1 {
StatusV2 {
fork_digest: wire.fork_digest,
finalized_root,
finalized_epoch,
head_root,
head_slot,
};
earliest_available_slot,
}
}

/// `status` in `version`: v1 is v2 without `earliest_available_slot`.
pub fn status_in_version(status: &StatusV2, version: StatusVersion) -> BeaconStatus {
match version {
StatusVersion::V1 => BeaconStatus::V1(v1),
StatusVersion::V2 => BeaconStatus::V2(StatusV2 {
fork_digest: v1.fork_digest,
finalized_root: v1.finalized_root,
finalized_epoch: v1.finalized_epoch,
head_root: v1.head_root,
head_slot: v1.head_slot,
earliest_available_slot,
StatusVersion::V1 => BeaconStatus::V1(StatusV1 {
fork_digest: status.fork_digest,
finalized_root: status.finalized_root,
finalized_epoch: status.finalized_epoch,
head_root: status.head_root,
head_slot: status.head_slot,
}),
StatusVersion::V2 => BeaconStatus::V2(status.clone()),
}
}

Expand Down Expand Up @@ -225,11 +236,13 @@ pub async fn request_metadata(server: &P2PServer, peer_id: PeerId) {
.await;
}

/// Re-open a refused handshake on the other `Status` version.
/// Re-open a refused `status/1` request on `status/2`.
///
/// A peer that supports neither version was never going to talk to us, and one
/// retry cannot loop: the retry is sent in the version the first attempt was
/// not.
/// retry cannot loop: the caller retries only a refused `status/1`, and this
/// sends `status/2`. A refresh asks in the version the peer last answered in,
/// so a peer that has dropped v1 is asked in v2 from then on rather than
/// refused and retried every time.
pub async fn retry_status_on_other_version(server: &P2PServer, peer_id: PeerId) {
let Some(wire) = server.wire.beacon() else {
return;
Expand Down
7 changes: 7 additions & 0 deletions crates/net/p2p/src/beacon/messages.rs
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,13 @@ impl BeaconStatus {
}
}

pub fn head_root(&self) -> Root {
match self {
BeaconStatus::V1(status) => status.head_root,
BeaconStatus::V2(status) => status.head_root,
}
}

pub fn finalized_epoch(&self) -> Epoch {
match self {
BeaconStatus::V1(status) => status.finalized_epoch,
Expand Down
9 changes: 5 additions & 4 deletions crates/net/p2p/src/beacon/verdict.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ use spawned_concurrency::message::Message;
use spawned_concurrency::tasks::{Context, Handler};
use tracing::{debug, error, warn};

use crate::beacon::{builder_market, column_checks};
use crate::beacon::column_checks;
use crate::gossipsub::operation_kind;
use crate::{P2PServer, metrics};

Expand Down Expand Up @@ -213,9 +213,10 @@ impl Validated {
Self::ExecutionPayloadBid { bid, .. } => {
server.builder_market.record_bid((**bid).clone())
}
Self::ProposerPreferences(preferences) => server
.builder_market
.record_preferences((**preferences).clone(), builder_market::wall_slot(server)),
Self::ProposerPreferences(preferences) => server.builder_market.record_preferences(
(**preferences).clone(),
crate::wall_clock_slot(&server.store),
),
Self::PayloadAttestation(message) => server
.seen_payload_attestations
.record(message.data.slot, message.validator_index),
Expand Down
Loading
Loading