Skip to content

[WIP] Gracefully handle nodes that now prune historical blocks - #4360

Draft
eranrund wants to merge 2 commits into
ProvableHQ:stagingfrom
eranrund:eran/authority-prune
Draft

[WIP] Gracefully handle nodes that now prune historical blocks#4360
eranrund wants to merge 2 commits into
ProvableHQ:stagingfrom
eranrund:eran/authority-prune

Conversation

@eranrund

Copy link
Copy Markdown

This is a companion PR to ProvableHQ/snarkVM#3329

It contains two major changes, presented as individual commits that hopefully make it easier to review/reason about.


Background

The companion snarkVM PR prunes authority data (authority_map + certificate_map) for blocks older than AUTHORITY_RETENTION_BLOCKS (100,000 blocks, roughly three days at 2.5s block times). Per review discussion there, peers do not advertise whether they keep full history. Instead the protocol contract becomes implicit and uniform: every peer serves [tip − AUTHORITY_RETENTION_BLOCKS, tip], and anything older is fetched from the trusted CDN (automatic on startup) or from a ledger snapshot (manual).

That design needs two things from snarkOS, which today does neither: a node must be able to decline to serve blocks it no longer has without punishing the requester, and a node must avoid asking peers for blocks they cannot possibly still hold - and know what to do when nobody can serve it.

Commit 1: Don't treat get_blocks as failure and penalize requesting peer

Today, when a node fails to load the blocks in a BlockRequest, the handler returns false/Err and the inbound dispatcher turns that into "peer sent an invalid block request", disconnecting the requester for a protocol violation. The failure is attributed to the wrong side: with pruning, an honest node would disconnect every peer that legitimately asks for a range it has pruned (and even without pruning, it's not the requesters fault that get_blocks failed...).

This commit makes an unavailable range a normal, benign outcome:

  • Serving side: In the client router, validator router, and BFT gateway, a get_blocks failure now logs a warning and replies with an empty BlockResponse rather than failing the message. An empty response is expressible in the existing wire format (DataBlocks serializes a u8 count), so there is no format change and no new message variant.
  • Requesting side: DataBlocks::ensure_response_is_well_formed now accepts an empty response, documented as the peer's explicit "I cannot serve this range" signal. Ordering and range checks still apply to non-empty responses.
  • InsertBlockResponseError::EmptyBlockResponse is now classified as benign. The existing machinery then does the right thing without further changes: the peer's outstanding requests are removed and marked for re-issue to other peers, so failover is immediate rather than waiting out the 60s request timeout.

Compatibility. No wire format change. Against an un-upgraded requester, an empty response is rejected and that peer disconnects the server and retries elsewhere - noisier, but assumed to be better than today, where the server disconnects an innocent requester. Un-upgraded servers never emit empty responses, so their behavior is unchanged.

Commit 2: Don't request blocks that peers have pruned, and shut down when no peer can serve them

With commit 1 alone, a node far behind the network would request pruned ranges, receive empty responses, re-issue to the same peers, and churn indefinitely. This commit "teaches" the requester the retention contract and gives it an exit.

  • Retention-aware peer selection: find_sync_peers_inner now skips peers whose assumed pruning floor lies above the next block we need. The floor is derived from the peer's advertised tip: tip − (AUTHORITY_RETENTION_BLOCKS − RETENTION_SLACK). The slack (1,000 blocks, chosen arbitrarily) exists because an advertised tip is only a snapshot - by the time our request arrives the peer may have advanced and pruned correspondingly, so we cannot assume it serves exactly tip − retention.
  • Stuck detection: BlockSync records when peers ahead of us exist but every one of them has pruned what we need, exposed as sync_stuck_below_peer_floors(). This state cannot resolve itself - peers prune further with every block they produce, so the gap only widens. It is deliberately distinguished from simply having no useful peers (not yet connected, isolated), which is transient and does not count.
  • Fatal shutdown: After the condition persists for five minutes, the node logs how to recover and shuts down gracefully via the new SignalHandler::stop_with_failure(), which causes the process to exit non-zero so a supervisor restarts it - at which point the existing startup CDN sync closes the gap automatically. Restoring a snapshot is the manual alternative; nodes run with --nocdn must use one.

The choice of shutdown over in-process CDN fetching was deliberate. Mid-run CDN sync would require reworking the one-shot CdnBlockSync lifecycle, re-routing its ledger writes through the advancement lock it currently bypasses, and adding a pause/resume surface to BlockSync - and it would be difficult to cover validators, where a mid-run CDN jump would strand BFT/DAG state. Restarting into the existing, well-tested startup path achieves the same outcome. This is a thing we can improve upon if we decide to invest resources in it, but in reality I think we are not going to experience nodes being down for almost 3 days and then suddenly coming back to life and trying to sync. If a node fell that far behind, it might as well restart. This keeps the code and potential edgecases smaller and easier to reason about. Its also worth emphasizing that CDN sync takes place every time a node restarts, so this is standard practice.

Misc notes

  • Devnet validation is still outstanding (.ci/test_devnet.sh). We'll need to make the 100k prune block window configurable (maybe via a build env var), and we'd need a way to trigger a node falling far behind and aborting.
  • The stuck detector is inferred from peer claims. A peer advertising a tip far above the real chain is unverifiable. In a progressing network this is inert: every real block gives an honest peer a servable tip and resets the timer, but during a network halt, when no honest peer is ahead, a lying peer could push idle nodes to exit. I am not convinced this is a problem since the network will (should?) never halt for any meaningful duration of time... if this is a concern, we might need hardening around this such as requiring a majority to choose the next tip, or choosing the minimum out of all peer-reprted tips that are ahead of us instead of choosing the maximum (which is what I think the code currently does).
  • REST still answers misleadingly for pruned blocks (GET /block/{h} 404s; GET /blocks fails the whole range) - a separate follow-up. I'm considering using HTTP code 410 Gone (Gone client error response status code indicates that the target resource is no longer available at the origin server) to differentiate between a block that never existed and a block that used to exist but are no longer available due to pruning.

Another thing worth mentioning is that this shifts the entire network (after an upgrade) to assuming no node can actually provide post-prune-window block, even though some might (due to having the history feature enabled). That was a deliberate decision that favors simplicity over having each peer return different ranges it can fulfill (explicitly suggested in the companion PR).

eranrund added 2 commits July 29, 2026 16:06
Instead, return an empty blocks array which the requesting peer will
interpret as "the peer cannot provide the requested blocks".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant