Repository navigation
fix(sync): judge the sync gate against peers' Status heads - #791
Conversation
A node reported itself synced while far behind (#559): the tracker took the freshest block the network knows from the node's own imported chain, which trails the head during a catch-up, so its network-stall override fired and the node attested and proposed on a stale head. The beacon follower had the same bug and reported synced through whole catch-ups. Follow Lighthouse and Prysm instead: a node is syncing only when its head trails the wall clock and the head its peers report in Status is ahead of its own. P2P keeps each connected peer's latest Status head (re-requested every few slots, dropped on disconnect, a head past the clock ignored) and publishes the second-highest of three or more on the Store. Thresholds are per chain: leanSpec's 4/2 on lean, Lighthouse's two epochs of clock lag and one epoch of peer lead on beacon.
The beacon tick is a message in the same mailbox range sync forwards a catch-up into, and each import holds the actor for an engine round trip, so on a catch-up the tick waited behind every block of it. The gate was judged only there, so it kept answering `Synced` for the whole catch-up: on a kurtosis devnet a node reconnecting 100 slots behind under a slow EL read synced for 53 s and its validator client proposed on a stale head. Judge it in the head recompute every beacon arrival, tick and envelope release ends in, ahead of the forkchoiceUpdated call, and after each lean arrival, whose tick waits the same way.
…urst The refresh timer re-asked every connected peer for its Status in the same instant: 200 peers on a beacon node with the default target. The bytes are small (about 0.5 KB per round trip), but 200 streams opened together, and 200 answers handled together, are a needless burst. Each peer is now due one interval after its own last Status exchange in either direction, and the timer ticks every slot and asks at most ceil(peers / interval) of the due peers, longest-waiting first, the way Lighthouse and Prysm track a status time per peer. 200 beacon peers that connected together are re-asked at most 13 per slot and stay spread afterwards.
🤖 Codex Code Review
Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Grok Code ReviewI'll inspect the sync-status and peer-head changes in context, especially ordering, overflow, and how Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Claude Code ReviewReview: PR 791, peer-reported network head for the sync gateThe direction is right. Comparing against peers' Correctness and security
Performance
Rust and maintainability
Spec
Overall the PR is solid. Item 1 (no delivery feedback on claimed heads) and Item 2 (post-disconnect insert) are the ones I'd want addressed or explicitly accepted before merge. Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 DeepSeek Code ReviewThe PR substantially improves sync gating by comparing against peer-reported heads rather than imported blocks. The logic is well-structured and documented. A few issues remain: High priority
Medium priority
Low priority / suggestions
Overall the core design is sound and the tests cover the important cases. The lean path deserves immediate attention. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Gemini Code ReviewThe changes look solid. Moving the sync tracker from One minor doc nit:
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Kimi Code Review
Otherwise, the threshold arithmetic, startup handling, cross-actor atomic state, and refresh batching look sound and are well covered by tests. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
Review fixes for the peer-head sync gate. - Count the highest connected peer's Status head instead of the second-highest of three, so one honest peer is enough. A lying peer is bounded by a new no-progress backstop instead: a node that reads Syncing on peer reports while its head stays flat for `stall_window` slots (16 lean, an epoch beacon) treats the network as stalled until its head advances. The startup hold and the unfollowed-fork guard are exempt. - A head this node already imported (its head, or a block it holds a state for) is not counted as a lead and starts no range sync. - The startup hold runs its whole window; a report can add lead during it but never end it early. - Move the thresholds and the Status refresh cadence into `ethlambda_network_api::sync::SyncThresholds`, shared by the tracker and p2p, with a const assertion that a report as stale as the refresh allows still shows the lead of a node past the clock threshold. Lean: peer lead 4 -> 1, refresh 4 -> 2 slots. Beacon: a 4-slot hysteresis band. - Beacon judges the gate inside `update_head_from_fork_choice`, so the INVALID-verdict paths are covered, the head is decoded once, and the tick no longer judges it a second time. - Refreshes go out in each peer's last Status version, the Status is built once per tick, and a refused `status/2` is never retried. - Lean range sync uses the fetched-through watermark too (renamed `range_fetched_through`), and refresh answers on both chains go through `status_sync_target`. - The beacon digest-mismatch warning and counter fire once per move onto a foreign digest, not on every Status. - One shared `wall_clock_slot` replaces `builder_market::wall_slot`.
The startup hold assumed the network head was at the clock for its whole window (8 slots lean, an epoch beacon), so a node that started level with its peers kept its duties off for that long even after every peer had told it where the chain was. It now ends early once 5 connected peers have reported a Status head, and stays over once it has: the window remains the fallback for a node with fewer peers. A peer counts once its latest Status is not past the clock, including a head this node already holds, since a peer level with us is the slow-start case. A disconnect drops it from the count. After the hold the highest report is the network head, so one current peer among the five still shows the lead. p2p publishes the network head and the reporting-peer count together as a `PeerView` (`Store::set_peer_view` / `Store::peer_view`, replacing `set_network_head`). The head is written before the count and the count read before the head, so the tracker never sees the fifth peer counted without its head.
For its first slots (8 lean, an epoch beacon) the tracker assumed the network was at the wall clock, so a node behind the clock refused its duties whatever its peers said. That is a clock-only rule: when every node is behind the clock at once (a slow network start, a restart after an outage) all of them skip their proposals together and the chain stays stalled. Ending the hold once 5 peers had reported (ad9a587) only shortened it for well-connected nodes. Lighthouse has no such hold. It starts in `Stalled`, which its duty filter lets through, and refuses duties only once a peer shows a lead. The tracker now does the same: with no peer report the node reads Synced from the first judgment, and a node resuming a stale database does its duties on its stale head until a peer reports a lead (its handshake `Status` answer, right after connecting). This removes `startup_grace` and ad9a587's `startup_peers` and `PeerView`, so p2p publishes the network head alone again through `Store::set_network_head`. The beacon unfollowed-fork guard keeps treating the network as at the clock, and stays exempt from the stall window.
…lead A review found the stall window could be restarted. It was measured from when the node last fell behind on its peers' word, and a single judgment without the lead reset that. A peer could make the lead lapse for a moment (a `Status` naming a block we hold, a head past the clock, a reconnect) every few slots and keep the node idle indefinitely, which is the one thing the window promises cannot happen. The tracker now adds up the slots spent behind on the peers' word since the head last advanced, so a lapse pauses the count instead of starting it over. A head advance still resets it, so a steady catch-up stays syncing. On lean the band also erased the lead threshold: 1 - 2 saturates to 0, so a node already syncing stayed syncing on any peer one slot ahead, such as one holding an orphan during a run of missed proposals, until the stall window ran out. The band no longer takes the lead's threshold below one slot. Beacon's (32 - 4) is unaffected. Also fixes two doc comments the review caught: the band releases the node when either lag drops below its threshold, not both, and the `/lean/v0/node/syncing` handler still described the old stall override. And it adds the lean half of the block-arrival test: a lean arrival judges the gate without waiting for the tick.
A peer's `Status` request postpones this node's own refresh to that peer, but only an answer could start a range sync. Between two nodes whose refresh timers fire at different points in the slot, the earlier one asks every interval and the later one only ever receives requests: it learned the lead, so its gate closed, and never synced towards it (the Codex review's P1, still open after d2135a5). A request now goes through the same path as an answer, held to a refresh answer's lead. Lighthouse does the same: `on_status_request` replies and hands the status to `process_status`, which an answer goes through too. Only a head the network head counts can start a sync now. A head past the clock was dropped from the network head but still merged into the range sync, where `next_batch` prefers the highest head, so a peer claiming the future got a batch on every refresh. A `Status` from a peer that is no longer connected is ignored, since `forget` has already run for it and nothing would remove its head. The lean request and answer share `on_lean_status`; both chains share `sync_towards_peer` for the merge-or-create on `range_sync_state`. A batch already in flight stays the only one.
A peer's reported head stayed in the network head for as long as the peer stayed connected, even after it stopped answering (the Kimi review's point, which d2135a5 only bounded through the stall window). A report now counts for `SyncThresholds::report_lifetime`, four refresh intervals (8 slots lean, 64 beacon), and the once-a-slot refresh timer drops older ones and republishes the network head. That outlasts a refresh that times out and the one after it, so one missed answer does not cost a node behind its peers its lead; expiry can only take a lead away. The refresh mark was kept per peer, so when a slow handshake answer arrived after the first refresh had gone out (lean refreshes every 8 s, under libp2p's 10 s timeout), the two answers swapped: the refresh answer was taken for the handshake and started a one-block range sync that raced gossip. A peer's first `Status` answer on a connection is now the handshake's and every later one a refresh's, which needs no per-request bookkeeping and stays right whatever order the answers come in.
`range_fetched_through` only ever rose: it is raised for every block a range answer hands to the chain actor, imported or not. Blocks that never import (an invalid answer, a fork the head did not follow) left it above the head for good, so every later `Status` was measured from the watermark and the missing span was never range-requested again; the node had to walk back to it one parent at a time. Lean picked this up when it started keeping the watermark in this PR; beacon already had it. The once-a-slot refresh timer now watches the head. When the head has stayed below the watermark, without moving and with no range session running, for longer than the stall window, the watermark drops back to the head and the count starts over. A head that is still importing moves, so a slow catch-up never trips this (which is what the watermark is there for: re-fetching blocks still queued for import), and a head that cannot move at all is fetched for at most once a window.
…age a whole window A review of bb643b5 found two holes in how the stall window adds up. A lapse still threw away the stretch on either side of it: the count grew only by the gap between consecutive judgments that found the node behind, and a judgment without the lead cleared the start of that gap. A peer that dropped its claim just before each slot's first judgment (a `Status` naming our head) and made it again before the second kept the count at zero, and lean, judged several times a slot, read Syncing at the judgments that gate duties. Now a stretch behind counts in full and a slot reached after a lapse counts once, so a lapse at every slot boundary pauses nothing; a claim made only every other slot stretches the window to twice its length, no more. Time counted before an outage also carried over: a node that spent most of a window behind, lost every peer for hours and reconnected gave up on the lead a slot later, before its first range batch could land. A lapse longer than the window now ends the count, so the next claim gets a window of its own. That cannot be used to idle a node: the lapse is time the node spends doing its duties. The review also noted that on lean the band no longer applies to the lead (its threshold of one cannot go lower without making a one-slot lead count again). That is the intended trade-off; the docs now say so.
…er head 030c23b let a lapse longer than the stall window end the count, so a node reconnecting after an outage gets a whole window. Every such lapse did, though: a peer that claimed a lead for most of a window, dropped it for a whole one and claimed again kept a flat node idle for about half of all slots, indefinitely, and the backstop never tripped. The fresh window now comes once per head position, cleared when the head advances, so a lying peer costs at most one window at each head the node reaches, or two around an outage. The docs that state the bound say so, and also that a node which had already given up on a lead at its head gets no new window after an outage (unchanged: that holds until its head advances).
The fresh window after a long lapse (0381cb3) also matched when nothing had been counted at the current head yet, so the first claim after a quiet stretch (a stalled network with no claims for a while) used up the renewal for nothing, and an outage after that cut the next window short. It now applies only once something was counted at this head; the first claim starts from zero anyway. The bound is unchanged.
…e dropping the watermark A review of acfd0b0..5b4c378 found: - A peer could take every range batch. Its own `Status` requests are not rate limited, the highest head gets the next batch, and a failed request carried no cost, so a peer claiming a head it does not serve (one slot past the clock still counts) rejoined after each empty answer and got the next batch too, stalling a catch-up. A peer that fails a range request is now not synced towards again for a refresh interval, the rate at which only refresh answers could re-add it before requests could. - The watermark could drop on the first tick after a range session ended: the flat count ran through the session, so a head that had been flat before it (peers lost, an EL down) dropped the watermark before the batch just handed over could import, and the next `Status` fetched it again. The count now restarts for as long as a session runs. - A head that can never move re-fetched the whole gap every window. Each drop at the same head now doubles the wait before the next, up to 64 windows, and a head that moves resets it. - An answer delivered after its peer's disconnect was counted in `answered`, so the next connection's handshake answer was taken for a refresh's. The connection is now checked first. - Two tests were weaker than their names: the pacing test now uses five peers, so asking nobody cannot pass it, and the request test goes through `on_lean_status_request`, the half of the request handler that chooses the lead.
…he backoff on a new head libp2p reports a closed connection before the request it failed, so a peer that disconnected with a range batch in flight had `forget` run first and its failure noted after, leaving a cooldown entry that outlived it and held back its next connection's handshake for a refresh interval. A failure is now noted only while the peer is connected. The watermark backoff kept its doublings when the head moved during a range session and then sat where the session's last tick saw it, so the next drop at that new head waited up to the cap. Restarting the count at a different head now resets them, and the cap is 16 windows rather than 64 (about 17 minutes on lean, under two hours on beacon).
|
/review |
🤖 Codex Code Review
Tests could not run: rustup requires writes to the read-only Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Claude Code ReviewReview: peer-head-based sync gate (PR 791)The core design is a clear improvement over the "freshest imported block" heuristic. The gate now requires both clock lag and a peer-reported lead, uses Lighthouse/Prysm thresholds, and bounds lying peers with a stall window. Tests are thorough and each one is tied to a concrete finding. Below are the issues I'd want resolved or at least acknowledged. Correctness / security
Performance
Minor
Overall the logic is sound. Items 1 and 2 are the ones I'd fix before merge. Automated review by Claude · claude-opus-5-5 · custom prompt |
🤖 Gemini Code ReviewThis is a well-designed PR. Moving the sync gate to compare against peer-reported I found two bugs in the new P2P tracking logic and one minor performance issue:
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Grok Code ReviewThe peer-head sync gate is a real improvement over the imported-block stall rule, and the stall-window, hysteresis, and refresh-spreading logic is carefully tested. A few correctness holes will still leave a node attesting or not fetching when it should not.
The threshold consistency asserts, refresh pacing, range-failure cooldown, and disconnect-before-failure ordering look right. I would not merge until Items 1 through 4 are fixed; Item 3 is the one that can put attestations on a head fork choice has already rejected. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Kimi Code Review
Otherwise, the shared thresholds, atomic Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
When the head moved back (an `INVALID` verdict cutting a branch out, a reorg onto a shorter branch), the tracker kept a stall it had given up on and the time already spent behind, so the node went on doing its duties on peers that may have been right about the chain all along (the Grok review's item 3). A move back now starts the stall window over and releases the stall, the way an advance does. Only fork choice moves the head, so no peer can use this to idle a node.
pablodeymo
left a comment
There was a problem hiding this comment.
Thanks for the round-ten fixes; the pause caps and the skill updates look right to me. Three more points inline, from a second pass on 58ed6d4.
Two more devnet-skill files outside the diff still describe the old gate: .claude/skills/multi-server-devnet/references/operations.md:125 says heads lagging past the sync-gate threshold stop attesting and proposing, and scripts/start-devnet.sh:124 says stale nodes "sit in the sync gate". Under the new rule a node that far behind reads synced unless a peer shows a lead.
| /// the same head doubles the wait before the next one, so a head that | ||
| /// cannot move at all re-fetches the gap ever more rarely. | ||
| pub(crate) fn expire_range_watermark(&mut self, wall_slot: u64) { | ||
| let head_slot = self.head_slot(); |
There was a problem hiding this comment.
head_slot() reads the slot the chain actor publishes, and on_block publishes it once, after run_import_cascade has drained (blockchain/src/lib.rs:2135 on lean, recompute_beacon_head on beacon). So for the length of one cascade the head looks flat from here, however many blocks it is importing.
If a cascade outlasts the stall window (16 slots on lean, 32 on beacon) while a range batch sits above the head, this tick takes it for blocks that never imported: the watermark drops, the session ends, the sender of the lowest batch becomes a suspect, and blocks already queued are fetched again. A by-root backfill that connects and releases a long chain of pending children is the case I have in mind. It recovers once the cascade ends and the head is published, but the session and the batch in flight are gone by then.
I have not measured how long cascades run in practice. Could the cascade publish the head slot as it goes, or could this check read Store::head_slot on lean, where it is a header read?
There was a problem hiding this comment.
Done in 71d23c2, as "the cascade publishes as it goes", on both chains. Reading Store::head_slot would only cover lean: there every import moves the head (update_head in store::on_block), but a beacon import records no head at all, so nothing P2P reads can follow a beacon cascade.
- The change: between two blocks, after an import with more blocks queued,
run_import_cascadejudges the gate (judge_within_cascade) once the wall clock is in a slot the tracker has not judged yet.- Lean judges the head as the imports left it.
- Beacon runs fork choice first (
update_head_from_fork_choice, which judges). - That is one judgment a slot, the cadence of the tick the cascade starves.
judged_head_slotkeeps its meaning (the head at the latest judgment) and the P2P reader is unchanged.
- Kept at the cascade's end:
forkchoiceUpdatedand the head events. The judgment needs only the head. AnINVALIDverdict mid-cascade would invalidate blocks the queue was built on, and the end-of-cascade publish still announces the net move. - The beacon branch with a real head: the
beacon-spec-testschain-actor cases (--features beacon-spec-tests, whereget_headfinds a head) pass with it, 370 of 370. - Not in the import tree as its own row: the mid-cascade fork-choice time lands in the next child's
cascade_waitrow. - Tests:
a_cascade_publishes_the_head_slot_between_its_blocks(beacon) runsrun_import_cascadealone over two blocks that already have post-states. The head slot is published beforeon_block's own judgment, and the test fails with the call removed.a_cascade_judges_once_a_wall_clock_slot(lean): nothing new within the slot judged, then the moved head in the next one.- Neither test crosses a real slot boundary mid-cascade: the wall slot is passed in, or the fresh tracker has judged none.
- Not measured: how long cascades run in practice.
| .as_ref() | ||
| .and_then(|state| state.peer_set.get(peer).copied()); | ||
| if server.peer_heads.failure_streak(peer) >= SUSPECT_AFTER_FAILURES { | ||
| let through = server |
There was a problem hiding this comment.
The suspicion runs through the peer's current claim first, and the peer controls that.
A peer joins the session claiming head 5000, then sends a Status with head 1 and a root we do not hold. The claim is Counted, status_sync_target returns None, and sync_towards_peer returns before merge_peer, so peer_set still has it at 5000 and it keeps getting batches. When its third batch fails, through is 1, and the next clear_suspects removes the suspicion because the head is already past it. Only the cooldown is left to bound it.
Taking the highest of the claim, the session head and end_slot would close this.
There was a problem hiding this comment.
Fixed in 97a59e1. note_range_failure now suspects through the highest of the claim, the session's head for the peer, and the failed request's last slot. Any of the first two can be missing; the last never is.
a_peer_that_lowers_its_claim_is_suspected_through_its_session_head is your scenario:
- The peer joins on 5000 and fails twice. It is back in the session at 5000, then a
Statuswith head 1 and an unknown root leaves its claim at 1 and its session head at 5000. - After the third failure,
clear_suspectsat 64 and at 4999 leaves it a suspect, and 5000 clears it. - On the old code it fails at the 64 check.
The comment there and SUSPECT_AFTER_FAILURES's doc now say the same.
| @@ -915,7 +953,7 @@ What each failure looks like: | |||
| | `Peer said goodbye reason=129` right after a connection | nothing local: 129 is lighthouse's `TooManyPeers` | | |||
| | `fork_digest` is not what a live crawl reports | the blob-schedule branch, or the schedule itself | | |||
| | connections but no `Beacon handshake complete` | the `Status` encoding, or answering the wrong version | | |||
There was a problem hiding this comment.
This row still tells the reader to look for Beacon handshake complete. That line is gone; the one in handlers.rs is now Beacon Status answered.
There was a problem hiding this comment.
Fixed in 15578b0: the row now names Beacon Status answered. So does the "Peer lifecycle lines" sentence a few lines above it, which named the old line too.
…t skill docs/beacon_wire.md still pointed at 'Beacon handshake complete', which is now 'Beacon Status answered'. The multi-server devnet skill's operations guide and start-devnet.sh still said a lagging node stops its duties; under the peer-head rule a node far behind with no peer ahead reads synced.
The link that holds a short batch only to its last block is named by the upper batch's sender. A match only credits the peer below; a mismatch leaves it held, so a lying upper sender can still push a drop's blame onto an honest short batch. Written in HeadWatch::note_delivery's doc and in the stall window's cost in docs/spec_deviations.md.
…ed towards note_range_failure ran suspicion through the peer's current claim first, which the peer controls: one that joined the range session on a high head and then sent a Status with a head just above ours (a root this node does not hold) kept its place in the session at the high head, while its suspicion, through the lowered claim, cleared at the next tick. Suspicion now runs through the highest of the claim, the session's head for the peer and the failed request's last slot. Test: a_peer_that_lowers_its_claim_is_suspected_through_its_session_head.
The head slot P2P reads (Store::judged_head_slot) was published once a cascade had drained. A cascade outlasting the stall window, such as a by-root backfill releasing a long chain of pending children, therefore looked like a flat head to expire_range_watermark however many blocks it imported: the watermark was dropped, the range session ended and the lowest batch's sender was suspected. Between two blocks, once the wall clock has moved into a slot not judged yet, the cascade now judges the gate, as the tick it starves would have. On lean that reads the head each import has moved; on beacon, where an import records no head, it runs fork choice first. forkchoiceUpdated and the head events stay at the cascade's end. Tests: a_cascade_publishes_the_head_slot_between_its_blocks (beacon), a_cascade_judges_once_a_wall_clock_slot (lean).
|
The two devnet-skill files from your second pass are fixed in 15578b0:
I also swept |
|
/review |
🤖 Claude Code ReviewReview: PR 791, peer-based sync gate,
|
🤖 Codex Code ReviewNo actionable correctness or security issues found in the reviewed changes. The sync thresholds, peer cooldowns, and range-session isolation appear consistent with the documented behavior. Tests could not run: rustup attempted to write to a read-only toolchain directory. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Gemini Code ReviewThis PR is very well thought out and thoroughly tested. The shift from store-local max slot tracking to peer-reported Key highlights:
No issues found — LGTM. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Grok Code ReviewThe peer-head sync gate, stall accounting, and watermark expiry are carefully built and well tested. One lean tick path undoes that under load.
Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 DeepSeek Code ReviewThe broad design is sound and the new tests cover many subtle regressions. The concerns below are mostly edge cases rather than consensus-critical faults.
Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewAutomated review by Kimi (Moonshot AI) |
|
Round thirteen (the automated review of 71d23c2). Nothing changed: no correctness defect was found, and the real points that remain are follow-ups. The code in 71d23c2 is what was verified before the push; only one paragraph of
Follow-ups
Not changed
Codex and Gemini found nothing, and Kimi's job was cancelled. |
|
The review loop stops here: this PR is past the 10-round review cap (thirteen rounds, counting pablodeymo's two reviews). The last round raised no correctness defect. What stays open is tracked in follow-up issues:
|
# Conflicts: # crates/net/p2p/src/lib.rs
# Conflicts: # crates/net/p2p/src/req_resp/handlers.rs
# Conflicts: # crates/blockchain/src/lib.rs
🗒️ Description / Motivation
The sync gate reported Synced while a node was far behind (#559), so it attested and proposed on a stale head. It took the freshest block the network knows from the node's own imported chain, which trails the head during a catch-up, so the network-stall override fired. The beacon follower had the same bug:
docs/beacon_wire.mdrecords it reporting synced through a 10-minute catch-up up to 98 slots behind.This follows Lighthouse and Prysm: a node is syncing only when its head trails the wall clock and the head its peers report in
Statusis ahead of its own. Peers decide "behind vs. stalled"; the clock keeps any peer claim from idling a node that is current. A peer's claim is bounded by progress: a lead that never turns into head advances is read as a stall after a fixed window.A kurtosis run found a second cause on the beacon follower: the gate was judged only on the actor's slot tick, which waits behind a whole catch-up in the actor's mailbox. It is now judged after every block arrival too (second commit).
d2135a5 addresses most of the automated review of the first three. ad9a587 ended the startup hold early once 5 peers had reported, and 46fa600 removes the hold altogether, following Lighthouse: with no peer report the node reads Synced from the first judgment.
Further sub-agent review rounds (bb643b5 through 499c0d4) fix what that left open:
Statusrequest postponed our own refresh to it, and only an answer could start a range sync (the Codex review's P1, which d2135a5 had not fixed). A request now can, as in Lighthouse, where both go throughprocess_status.The automated review of 499c0d4 (fixed in f44b096, 733ff37, 6075e1b, 9a48189 and c882d0c; the per-finding answers are in the PR comments) adds:
Statuscomes from a digest this node does not sync from.INVALID, or a reorg onto a shorter branch) starts the stall window over.The second automated review (of c882d0c; fixed in ab64725, 474383d, 2c56311, f13a7d1 and 6cdbcd1, each checked by a sub-agent; answers in the PR comments) adds:
INVALIDcut) still counts as progress.Statusrequests are paced: at most one per peer per slot is acted on.publishhonours the report lifetime.status/2falls back tostatus/1on the next refresh.The third automated review (of 6cdbcd1; fixed in d07639d, afadb14, 0277b34 and 0425622; answers in the PR comments) adds:
spec_deviations.mdstates the trade-off: honest deliverers stop counting if the node's own import stalls for a window.The fourth automated review (of 0425622; fixed in a597371, 2334c9e and 03b1692; answers in the PR comments) adds:
on_tickjudges after the tick if the tick did not.The fifth automated review (of 03b1692; fixed in baabc8f, 085e358, e784215, 4eed99b and 963e0b3; answers in the PR comments) adds:
Status: built once per slot and per fork digest, and answered before the peer's head is recorded.Statusrequests:Statusrequest is in flight per peer.Statusversions is no longer refreshed.StallCount.INVALIDcut releases a stall once per high-water mark.The sixth automated review (of 963e0b3; fixed in 310eee0, 34af7b3 and 6b00710) found no correctness bugs apart from one:
Statusfollows the published head as well as the slot and the fork digest.The seventh automated review (of 6b00710; fixed in 2aca292, eda7080 and ae89e34):
Statusis keyed on the head root and the finalized checkpoint (both metadata reads) besides the wall slot and the fork digest, so a reorg within a slot shows at once. The connect handshake and thestatus/2retry send it too.Statusit sent. The trade-off is stated indocs/spec_deviations.md.The eighth automated review (of ae89e34; fixed in 68cf259, cd139be, 9f0794a and eb75547):
Statusrequest that never ends (one whose send went nowhere) is taken as lost after four refresh intervals, so its peer is asked again.Statusclaims a head past the clock, comes from a digest this node does not sync from, or shows a head it holds) takes no more batches. An emptied session ends throughend_range_session, purging its batch in flight.Statusis not kept while the head names no stored block, and reads its key withoutexpect.Store::head_slot_hintis renamedjudged_head_slot, and the beacon stall window's trade-off is stated indocs/spec_deviations.md.The ninth automated review (of eb75547; fixed in d35f6b0, a1aa871 and 0bd2337):
lean_sync_peers_held_back{reason=suspect|cooling}counts the claims held back from the network head.The tenth review (pablodeymo's, of 78db839; fixed in 5fd6ee1; answers in its threads):
lean_sync_peers_held_back{reason="future"}.host-check.shreads that from the SYNC column.make fmt,make lintandmake testpass on 5fd6ee1 (3,462 tests, 0 failed).The eleventh automated review (of 5fd6ee1; fixed in 58ed6d4):
PeerHeads' own slot stamps are unchanged. A step back lengthens them by the step, and the tracker's window bounds what that costs in duties.Statusis the handshake's. An error answer carries no head; the docs said "the first answer".end_range_session, with its requests.make fmt,make lintandmake testpass on 58ed6d4 (3,466 tests, 0 failed).The twelfth review (pablodeymo's second pass, of 58ed6d4; fixed in 15578b0, 03b5ba9, 97a59e1 and 71d23c2; answers in its threads):
forkchoiceUpdatedand head events stay at the cascade's end. Range blocks still importing no longer read as a flat head to the watermark.note_deliverystates that the upper sender chooses the parent link (also indocs/spec_deviations.md);beacon_wire.mdnamesBeacon Status answered; the devnet skill'soperations.mdandstart-devnet.shno longer describe the old gate.cargo fmt, workspace clippy and the touched crates' tests pass: p2p 439, blockchain 327 (370 withbeacon-spec-tests), storage 214, 0 failed.make testwas not run for this round.Follow-up, not in this PR: range sync does not step past a span a whole batch long with no block, as on
main. Advancing over it safely needs re-asking skipped spans and blaming whoever answered empty.What Changed
crates/net/api/src/sync.rs(new):SyncThresholds, shared by the chain actor's tracker and the p2p layer so the refresh cadence cannot drift from the lead it detects. A const assertion keepspeer_lead + status_refresh_slots < clock_lag, so a report as stale as the refresh allows still shows the lead of a node past the clock threshold.Statusrefreshreport_lifetime). That outlasts a refresh that times out and the one after it.DEFAULT_SYNC_TOLERANCE_EPOCHS, also Prysm'sshouldReSync) and 1 epoch of peer lead (SLOT_IMPORT_TOLERANCE). It adds a 4-slot band because this node re-judges on every block arrival, and a follower importing about one block a slot at the boundary would otherwise flip twice a slot. The refresh is Prysm's half epoch.crates/blockchain/src/sync_status.rs: the tracker.Stalledstate. A hold that assumed the network was at the clock (an earlier version of this PR) is a clock-only rule: when every node is behind the clock at once (a slow network start, a restart after an outage), all of them skip their proposals and the chain stays stalled.stall_windowslots in all while its head stays flat, it treats the network as stalled and stays Synced until its head advances past where it tripped. The slots are added up, so a lead that lapses for a moment pauses the count rather than restarting it. That covers a peer that claims a head it does not serve (empty answers, invalid blocks, timeouts). The window is sized above the 30 s range custody wait and an execution client's ~26 s retry ladder. The beacon unfollowed-fork guard is exempt.crates/net/p2p/src/peer_heads.rs(new): each connected peer's latestStatushead, dropped on disconnect or once older thanreport_lifetime. The highest report is published throughStore::set_network_head. AStatusfrom a peer no longer connected is ignored.FUTURE_SLOT_TOLERANCE).p2p:
Statuseverystatus_refresh_slots, counted from that peer's own last exchange in either direction. The timer ticks every slot and asks at most ceil(peers / interval) of the due peers, longest-waiting first. With the default 200 beacon peers, that is at most 13 per slot.Statusis built once per tick, and each peer is asked in the version it last spoke. A refusedstatus/2is never retried.Statusrequest, starts range sync only pastpeer_leadbeyond what was already fetched (status_sync_target), so it never races gossip. A handshake answer starts one at any lead.Statusis the handshake's; every later one is a refresh's.range_fetched_through, renamed frombeacon_fetched_through). On both chains, a watermark the head stays below, unmoved with no range session running, for longer than the stall window drops back to the head. Blocks handed over that never imported are then fetched again, at most once a window.lean_beacon_status_digest_mismatch_totalfire once per move onto a foreign digest, not on everyStatus.wall_clock_slotreplacesbuilder_market::wall_slot.crates/blockchain/src/lib.rs:Store::network_head, and the beacon fork guard is kept.update_head_from_fork_choice, which every block arrival, tick, envelope release and INVALID verdict ends in, ahead of theforkchoiceUpdatedcall. The head is decoded once there. Lean judges on the tick and at the end ofon_block.Synceduntil the catch-up was over.Docs: a new
spec_deviations.mdentry (lean departs from leanSpec ci: add a Commit-Boost check to the kurtosis battery #708), plus updates toarchitecture.md,rpc.mdandbeacon_wire.md.Correctness / Behavior Guarantees
Stalledpassesnot_while_syncing_filter).Statusanswer right after connecting. A Lighthouse node inStalleddoes the same. This is accepted: any rule that holds duties on the clock alone makes every node skip them at once when all are behind the clock.is_syncingis the decision that gates duties, since its validator client readsis_syncing./eth/v1/node/syncingderivesis_syncingfrom its sync state (!is_synced(), except with zero target peers), so it reads true while range-syncing even within the 2-epoch tolerance, and whileStalled.not_while_syncing_filter) refuse only while range-syncing beyond the tolerance, and letStalledthrough. That is what ethlambda's single flag does.main.Tests Added / Run
Unit tests for the tracker rule, peer aggregation,
Store::network_headsharing, and the beacon follower cases.a_block_arrival_judges_the_gate_without_waiting_for_the_tickdrives the real arrival path (on_block) with no tick. It fails on db61388 (status staysIdle) and passes now.d2135a5 adds tests for each review scenario. Among them:
a_head_recompute_judges_the_gate_on_its_own.The second review round adds a test for each finding:
Statusrequest past the lead starts range sync;Statusfrom a peer no longer connected is ignored;These are written against the new APIs, so they were not run on 6f08c98.
46fa600 replaces the hold's tests with
a_stale_node_keeps_its_duties_until_a_peer_shows_a_lead(Synced with no report or a stale one, Syncing on the first report that shows a lead) and keepsa_late_first_report_still_gets_a_whole_stall_window. ad9a587's early-end tests went with the hold.make fmt,make lintandmake testpass on 34af7b3 (2,994 tests, 0 failed); 6b00710 reorders two blocks in one function, and its crate's tests pass.Kurtosis: lighthouse×2 plus ethlambda beacon (cl-3) with an ethlambda VC (vc-3), mainnet preset.
Two tools:
iptablesdrops on the CL p2p ports 9000–9001 in cl-3's network namespace ("cut"), and anetemdelay on cl-3's outbound engine-API traffic (TCP 8551) to model a slow EL.Results below are from 5e0a9d1 and were not rerun on 6f08c98 or d2135a5. With d2135a5's 4-slot band, the gate reopens at a clock lag of 60 rather than 64. The slow catch-up is simulated with the
netemdelay. Lean was not run on a live network: kurtosis runs the beacon chain.Synced: synced from genesis, 2 peers, duty endpoints 200, and the VC attests, aggregates and does sync-committee duties every slot.
Cut off with the VC stopped, then reconnected under a slow EL: the VC was stopped so the node's own head stalls, as a lone cut-off node's does. The cut ran from slot 12 to slot 112, and reconnect happened with a 1 s engine delay, so each import takes about 2 s.
is_syncing=falsewith duty endpoints at 200 throughout, up to 100 slots behind.Sync status changed current_slot=112 head_slot=14 network_head=111 clock_lag=98 peer_lead=97 starting_up=false syncing=true. The network head came from peers'Status, not from the startup hold.clock_lag=64 peer_lead=59 … syncing=false, 48 s later.Failed to refresh duties … beacon node is syncingevery slot and published nothing (no block, attestation or sync-committee message) until the gate reopened.Restart after downtime: cl-3 was stopped at slot 121 and started at slot 201 (
Resuming from existing DB head_slot=120 current_slot=201 gap=81).starting_up=true). Without the hold (46fa600) it would read Synced until its first peer'sStatus; not rerun.Short cut with the VC running: 12 slots, from slot 207 to 219.
Before the second commit (db61388), same cut-and-slow-EL scenario:
is_syncing=falsefor about 53 s while 101 to 64 slots behind with a peer back, and no "Sync status changed" line was logged.target epoch is too far from this node's head).Expected, within tolerance: once the gate reopens, the VC works again while the node finishes catching up. The node refused its attestations there (
target epoch is too far …), and sync-committee messages went out. This is the 2-epoch tolerance Lighthouse's duty filter applies too.Before merging
Statusrefresh from every 4 slots to every 2, and lean range sync now keeps the watermark. Other lean clients'Statusrate limits at that cadence have not been checked.Follow-ups (not in this PR)
Syncblocks back to p2p would catch it at the first one, as Lighthouse's batch scoring does.Related Issues / PRs
✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test(test-consensusplustest-node, atrelease-fast) — all passing