Repository navigation
fix(sync): judge the sync gate against peers' Status heads - #791
MegaRedHand wants to merge 6 commits into
Conversation
A node reported itself synced while far behind (#559): the tracker took the freshest block the network knows from the node's own imported chain, which trails the head during a catch-up, so its network-stall override fired and the node attested and proposed on a stale head. The beacon follower had the same bug and reported synced through whole catch-ups. Follow Lighthouse and Prysm instead: a node is syncing only when its head trails the wall clock and the head its peers report in Status is ahead of its own. P2P keeps each connected peer's latest Status head (re-requested every few slots, dropped on disconnect, a head past the clock ignored) and publishes the second-highest of three or more on the Store. Thresholds are per chain: leanSpec's 4/2 on lean, Lighthouse's two epochs of clock lag and one epoch of peer lead on beacon.
The beacon tick is a message in the same mailbox range sync forwards a catch-up into, and each import holds the actor for an engine round trip, so on a catch-up the tick waited behind every block of it. The gate was judged only there, so it kept answering `Synced` for the whole catch-up: on a kurtosis devnet a node reconnecting 100 slots behind under a slow EL read synced for 53 s and its validator client proposed on a stale head. Judge it in the head recompute every beacon arrival, tick and envelope release ends in, ahead of the forkchoiceUpdated call, and after each lean arrival, whose tick waits the same way.
…urst The refresh timer re-asked every connected peer for its Status in the same instant: 200 peers on a beacon node with the default target. The bytes are small (about 0.5 KB per round trip), but 200 streams opened together, and 200 answers handled together, are a needless burst. Each peer is now due one interval after its own last Status exchange in either direction, and the timer ticks every slot and asks at most ceil(peers / interval) of the due peers, longest-waiting first, the way Lighthouse and Prysm track a status time per peer. 200 beacon peers that connected together are re-asked at most 13 per slot and stay spread afterwards.
🤖 Codex Code Review
Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Grok Code ReviewI'll inspect the sync-status and peer-head changes in context, especially ordering, overflow, and how Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Claude Code ReviewReview: PR 791, peer-reported network head for the sync gateThe direction is right. Comparing against peers' Correctness and security
Performance
Rust and maintainability
Spec
Overall the PR is solid. Item 1 (no delivery feedback on claimed heads) and Item 2 (post-disconnect insert) are the ones I'd want addressed or explicitly accepted before merge. Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 DeepSeek Code ReviewThe PR substantially improves sync gating by comparing against peer-reported heads rather than imported blocks. The logic is well-structured and documented. A few issues remain: High priority
Medium priority
Low priority / suggestions
Overall the core design is sound and the tests cover the important cases. The lean path deserves immediate attention. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Gemini Code ReviewThe changes look solid. Moving the sync tracker from One minor doc nit:
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Kimi Code Review
Otherwise, the threshold arithmetic, startup handling, cross-actor atomic state, and refresh batching look sound and are well covered by tests. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
Review fixes for the peer-head sync gate. - Count the highest connected peer's Status head instead of the second-highest of three, so one honest peer is enough. A lying peer is bounded by a new no-progress backstop instead: a node that reads Syncing on peer reports while its head stays flat for `stall_window` slots (16 lean, an epoch beacon) treats the network as stalled until its head advances. The startup hold and the unfollowed-fork guard are exempt. - A head this node already imported (its head, or a block it holds a state for) is not counted as a lead and starts no range sync. - The startup hold runs its whole window; a report can add lead during it but never end it early. - Move the thresholds and the Status refresh cadence into `ethlambda_network_api::sync::SyncThresholds`, shared by the tracker and p2p, with a const assertion that a report as stale as the refresh allows still shows the lead of a node past the clock threshold. Lean: peer lead 4 -> 1, refresh 4 -> 2 slots. Beacon: a 4-slot hysteresis band. - Beacon judges the gate inside `update_head_from_fork_choice`, so the INVALID-verdict paths are covered, the head is decoded once, and the tick no longer judges it a second time. - Refreshes go out in each peer's last Status version, the Status is built once per tick, and a refused `status/2` is never retried. - Lean range sync uses the fetched-through watermark too (renamed `range_fetched_through`), and refresh answers on both chains go through `status_sync_target`. - The beacon digest-mismatch warning and counter fire once per move onto a foreign digest, not on every Status. - One shared `wall_clock_slot` replaces `builder_market::wall_slot`.
The startup hold assumed the network head was at the clock for its whole window (8 slots lean, an epoch beacon), so a node that started level with its peers kept its duties off for that long even after every peer had told it where the chain was. It now ends early once 5 connected peers have reported a Status head, and stays over once it has: the window remains the fallback for a node with fewer peers. A peer counts once its latest Status is not past the clock, including a head this node already holds, since a peer level with us is the slow-start case. A disconnect drops it from the count. After the hold the highest report is the network head, so one current peer among the five still shows the lead. p2p publishes the network head and the reporting-peer count together as a `PeerView` (`Store::set_peer_view` / `Store::peer_view`, replacing `set_network_head`). The head is written before the count and the count read before the head, so the tracker never sees the fifth peer counted without its head.
🗒️ Description / Motivation
The sync gate reported Synced while a node was far behind (#559), so it attested and proposed on a stale head. It took the freshest block the network knows from the node's own imported chain, which trails the head during a catch-up, so the network-stall override fired. The beacon follower had the same bug:
docs/beacon_wire.mdrecords it reporting synced through a 10-minute catch-up up to 98 slots behind.This follows Lighthouse and Prysm: a node is syncing only when its head trails the wall clock and the head its peers report in
Statusis ahead of its own. Peers decide "behind vs. stalled"; the clock keeps any peer claim from idling a node that is current. A peer's claim is bounded by progress: a lead that never turns into head advances is read as a stall after a fixed window.A kurtosis run found a second cause on the beacon follower: the gate was judged only on the actor's slot tick, which waits behind a whole catch-up in the actor's mailbox. It is now judged after every block arrival too (second commit).
d2135a5 addresses a review of the first three, and ad9a587 ends the startup hold early once 5 peers have reported.
What Changed
crates/net/api/src/sync.rs(new):SyncThresholds, shared by the chain actor's tracker and the p2p layer so the refresh cadence cannot drift from the lead it detects. A const assertion keepspeer_lead + status_refresh_slots < clock_lag, so a report as stale as the refresh allows still shows the lead of a node past the clock threshold.StatusrefreshDEFAULT_SYNC_TOLERANCE_EPOCHS, also Prysm'sshouldReSync) and 1 epoch of peer lead (SLOT_IMPORT_TOLERANCE). It adds a 4-slot band because this node re-judges on every block arrival, and a follower importing about one block a slot at the boundary would otherwise flip twice a slot. The refresh is Prysm's half epoch.crates/blockchain/src/sync_status.rs: the tracker.startup_graceslots after the first judgment, the network head is assumed to be at least the wall clock. The hold ends early oncestartup_peers(5) connected peers have reported aStatushead that is not past the clock, a head this node already holds included, and stays over after that. A report can add lead during the hold.stall_windowslots, it treats the network as stalled and stays Synced until its head advances past where it tripped. That covers a peer that claims a head it does not serve (empty answers, invalid blocks, timeouts). The window is sized above the 30 s range custody wait and an execution client's ~26 s retry ladder. The startup hold and the beacon unfollowed-fork guard are exempt.crates/net/p2p/src/peer_heads.rs(new): each connected peer's latestStatushead, dropped on disconnect. The highest report and the number of reporting peers are published together throughStore::set_peer_view, head first, and read count first, so the tracker never sees a peer counted without its head.FUTURE_SLOT_TOLERANCE).p2p:
Statuseverystatus_refresh_slots, counted from that peer's own last exchange in either direction. The timer ticks every slot and asks at most ceil(peers / interval) of the due peers, longest-waiting first. With the default 200 beacon peers, that is at most 13 per slot.Statusis built once per tick, and each peer is asked in the version it last spoke. A refusedstatus/2is never retried.peer_leadbeyond what was already fetched (status_sync_target), so it never races gossip. A handshake answer starts one at any lead. Lean range sync now keeps the fetched-through watermark too (range_fetched_through, renamed frombeacon_fetched_through).lean_beacon_status_digest_mismatch_totalfire once per move onto a foreign digest, not on everyStatus.wall_clock_slotreplacesbuilder_market::wall_slot.crates/blockchain/src/lib.rs:Store::peer_view, and the beacon fork guard is kept.update_head_from_fork_choice, which every block arrival, tick, envelope release and INVALID verdict ends in, ahead of theforkchoiceUpdatedcall. The head is decoded once there. Lean judges on the tick and at the end ofon_block.Synceduntil the catch-up was over.Docs: a new
spec_deviations.mdentry (lean departs from leanSpec ci: add a Commit-Boost check to the kurtosis battery #708), plus updates toarchitecture.md,rpc.mdandbeacon_wire.md.Correctness / Behavior Guarantees
Stalledpassesnot_while_syncing_filter).is_syncingis the decision that gates duties, since its validator client readsis_syncing./eth/v1/node/syncingderivesis_syncingfrom its sync state (!is_synced(), except with zero target peers), so it reads true while range-syncing even within the 2-epoch tolerance, and whileStalled.not_while_syncing_filter) refuse only while range-syncing beyond the tolerance, and letStalledthrough. That is what ethlambda's single flag does.main.Tests Added / Run
Unit tests for the tracker rule, peer aggregation,
Store::peer_viewsharing, and the beacon follower cases.a_block_arrival_judges_the_gate_without_waiting_for_the_tickdrives the real arrival path (on_block) with no tick. It fails on db61388 (status staysIdle) and passes now.d2135a5 adds tests for each review scenario. Among them:
a_head_recompute_judges_the_gate_on_its_own.These are written against the new APIs, so they were not run on 6f08c98.
ad9a587 adds tests for the early end: 5 reporting peers end the hold, it stays over once they have, one current peer among the reporters still shows the lead (beacon), peers level with us count as reporting and a peer reporting the future does not, and a
PeerViewpublished by p2p ends the hold through the actor.make fmt,make lintandmake testpass on ad9a587 (2,965 tests, 0 failed).Kurtosis: lighthouse×2 plus ethlambda beacon (cl-3) with an ethlambda VC (vc-3), mainnet preset.
Two tools:
iptablesdrops on the CL p2p ports 9000–9001 in cl-3's network namespace ("cut"), and anetemdelay on cl-3's outbound engine-API traffic (TCP 8551) to model a slow EL.Results below are from 5e0a9d1 and were not rerun on 6f08c98 or d2135a5. With d2135a5's 4-slot band, the gate reopens at a clock lag of 60 rather than 64. The slow catch-up is simulated with the
netemdelay. Lean was not run on a live network: kurtosis runs the beacon chain.Synced: synced from genesis, 2 peers, duty endpoints 200, and the VC attests, aggregates and does sync-committee duties every slot.
Cut off with the VC stopped, then reconnected under a slow EL: the VC was stopped so the node's own head stalls, as a lone cut-off node's does. The cut ran from slot 12 to slot 112, and reconnect happened with a 1 s engine delay, so each import takes about 2 s.
is_syncing=falsewith duty endpoints at 200 throughout, up to 100 slots behind.Sync status changed current_slot=112 head_slot=14 network_head=111 clock_lag=98 peer_lead=97 starting_up=false syncing=true. The network head came from peers'Status, not from the startup hold.clock_lag=64 peer_lead=59 … syncing=false, 48 s later.Failed to refresh duties … beacon node is syncingevery slot and published nothing (no block, attestation or sync-committee message) until the gate reopened.Restart after downtime: cl-3 was stopped at slot 121 and started at slot 201 (
Resuming from existing DB head_slot=120 current_slot=201 gap=81).starting_up=true).Short cut with the VC running: 12 slots, from slot 207 to 219.
Before the second commit (db61388), same cut-and-slow-EL scenario:
is_syncing=falsefor about 53 s while 101 to 64 slots behind with a peer back, and no "Sync status changed" line was logged.target epoch is too far from this node's head).Expected, within tolerance: once the gate reopens, the VC works again while the node finishes catching up. The node refused its attestations there (
target epoch is too far …), and sync-committee messages went out. This is the 2-epoch tolerance Lighthouse's duty filter applies too.Before merging
Statusrefresh from every 4 slots to every 2, and lean range sync now keeps the watermark. Other lean clients'Statusrate limits at that cadence have not been checked.Follow-ups (not in this PR)
Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test(test-consensusplustest-node, atrelease-fast) — all passing