feat: Let followers import zone blocks from leader sequencers by adityapk00 · Pull Request #668 · tempoxyz/zones

adityapk00 · 2026-07-15T18:25:53Z

follower sequencers can import blocks from leader sequencers.
- leader broadcasts canonical block
- follower recieves it over commonware p2p
- Validate block + re-execute it + ingest into DB

Note:

Catching up on blocks if follower falls behind is TODO (will be in feat: Allow follower nodes to backfill blocks #679)
Followers will need to consume L1 events, this is TODO (will be in feat: Split l1 subscriber so we can start with(lead sequencer) or without(follower) deposit queue #677)

# Conflicts: # Cargo.toml # crates/node/src/cli.rs # crates/p2p/README.md # crates/p2p/src/network.rs # crates/p2p/src/runtime.rs

tempoxyz-bot

👁️ Cyclops Review

This PR adds leader-to-follower block replication, but the current implementation has verified safety and liveness issues around delivery guarantees, oversized block handling, Zone-specific validation, and deterministic replay of L1-derived state. I left the actionable findings inline.

Reviewer Callouts

⚡ Follower role vs sequencer config: crates/node/src/node.rs:629 still starts ZoneEngine whenever sequencer_config exists, while follower behavior is keyed off the P2P role. A node configured as both follower and sequencer should hard-fail rather than both importing leader blocks and producing local blocks.
⚡ Block-channel memory/backpressure budget: MAX_MESSAGE_SIZE = 10 MiB, BLOCK_BACKLOG = 128, and receive-side role filtering after delivery mean authenticated peers can impose non-trivial memory/bandwidth pressure. Review Commonware queueing/backpressure before multi-operator exposure.
⚡ Broader replay determinism: The policy and portal-storage findings are concrete examples of the same class: follower replay currently depends on node-local L1-derived providers. When fixing them, audit every EVM-visible L1 read, especially “latest” fallback paths, and ensure imported block bytes or follower derivation fully determine execution inputs.

tempoxyz-bot · 2026-07-15T20:15:32Z

+        })
+        .await;
+        let sent = match sent {
+            Ok(sent) => sent?,


🚨 [SECURITY] Oversized canonical blocks can permanently stop leader-to-follower replication

sender.send(...) errors are re-raised here. Since the P2P channel is capped at 10 MiB and the leader broadcasts full RLP blocks without a byte-size cap, a canonical block over the Commonware limit returns MessageTooLarge, terminates run_commands, stops the P2P runtime, and leaves followers permanently behind with no backfill.

Recommended Fix:
Handle MessageTooLarge/send errors non-fatally, enforce a block byte cap below the P2P limit or chunk large blocks, and make unexpected P2P/broadcast termination restart or fail the node.

At 300m gas, the theorotical max is 7.5Mb (claude estimate) so i bumped it to 20MB by default for now. However, this point is moot because if the block size exceeds the max, then the chain will stall anyway since there won't be enough quorum in the ZonePortal to advance the batch

For reference, this number is 8MB on tempo L1

tempoxyz-bot · 2026-07-15T20:15:33Z

+    }
+
+    // 3. All txns in the block execute properly
+    let payload = ZonePayloadTypes::block_to_payload(block, None);


🚨 [SECURITY] Follower import relies on NoopConsensus instead of Zone validation

This wraps authenticated leader bytes directly into an execution payload. The Zone node is wired with NoopConsensus, so Reth's normal structural/header/receipt/gas validation hooks and Zone-specific derivation checks are not enforced; a compromised configured leader can canonicalize executable but non-Zone-valid blocks on followers.

Recommended Fix:
Use a Zone-specific consensus/derivation validator for imported blocks and verify the expected advanceTempo grammar, withdrawal rules, absence of extra privileged system calls, and L1-derived inputs before forkchoice.

All this is ultimately the provers job. The NoopConsensus is intentional, since we don't want any additional concensus at the zone layer.

tempoxyz-bot · 2026-07-15T20:15:33Z

+                    .send(Recipients::Some(followers.clone()), block.clone(), true)
+                    .await
+                    .map_err(|err| eyre::eyre!("failed broadcasting zone block: {err}"))?;
+                if !sent.is_empty() || followers.is_empty() {


⚠️ [ISSUE] Best-effort broadcast plus no backfill can permanently desync followers

Returning once sent is non-empty treats delivery to any one follower as success. A temporarily disconnected configured follower gets no retry for this block, and if it later receives N+1, import_leader_block rejects the gap because backfill is not implemented. That follower remains stale indefinitely.

Recommended Fix:
Track per-follower delivery/ACK high-water marks and add backfill or staged-sync recovery. At minimum, warn/metric partial delivery and do not treat it as sufficient for HA/failover assumptions.

Backfill is fixed in another PR

…k_import # Conflicts: # crates/node/src/node.rs # crates/node/tests/it/utils.rs

mattsse · 2026-07-16T10:35:30Z

can we move these functions somewhere else and keep this file to ZoneNode

moved to its own file

mattsse · 2026-07-16T10:36:49Z

+    while let Some(persisted_tip) = persisted.next().await {
+        if persisted_tip.number < last_broadcast {


I believe there's a race condition here, inverse to what's mentioned above:

// Subscribe before reading the database height so persistence cannot race task startup. let mut persisted = provider.persisted_block_stream(); let mut last_broadcast = match provider.last_block_number() {

last_block_number can already include the newest message in the stream I believe

Oh good. Catch. Fixed + added test

mattsse · 2026-07-16T10:38:49Z

+// At 30M gas, calldata is bounded below 7.5 MiB; leave headroom for block overhead.
+pub(crate) const MAX_MESSAGE_SIZE: u32 = 20 * 1024 * 1024;


this should be increased, we dont expect malicious messages anyway

I'll fix this in another PR, but there's a thread on slack about limiting this in the block builder (like we do in Tempo L1) so we don't end up with an unbroadcastalbe block

tempo-voight-kampff · 2026-07-16T10:42:49Z

Hi @mattsse — your changes-requested review was detected by Voight-Kampff but no live Voight-Kampff agent connection received it, so Voight-Kampff sent a push fallback. The push was not approved before the approval window closed. Your review did not count toward this PR.

You can also try requesting changes directly via CLI with

vk reject pr https://github.com/tempoxyz/zones/pull/668

# Conflicts: # crates/node/tests/it/e2e.rs # crates/node/tests/it/utils.rs

0xKitsune · 2026-07-17T00:25:42Z

+        if let Err(err) = import_leader_block(&provider, &engine, &block).await {
+            tracing::error!(target: "zone::p2p", %err, "Rejected leader block");
+        }
+    }


This logs an import failure and continues, but the local head has not advanced. Every subsequent leader block will then be rejected as a gap, so the follower remains running and serving a permanently stale chain.

We should either terminate the task or try to backfill when a leader block cant be imported. Silently continuing makes the failurelook recoverable but IIUC it isnt.

Yes, this is fixed in #679

if the imported block is really invalid (eg. leader is compromised), the follower should stall
if this is a transient issue, #679 will retry (with a backfill that will fill the missed block(s)

adityapk00 added 6 commits July 15, 2026 08:21

feat: Wire commonware for p2p between sequencer nodes

792aaf9

shutdown()

4971ca1

Merge remote-tracking branch 'origin/main' into aditya/p2p_block_import

175eb75

# Conflicts: # Cargo.toml # crates/node/src/cli.rs # crates/p2p/README.md # crates/p2p/src/network.rs # crates/p2p/src/runtime.rs

feat: follower listen to leader blocks and execute+ingest them

0e60ad3

harden sending

458f1ca

typo

a8ce5f3

adityapk00 added the cyclops label Jul 15, 2026

tempoxyz-bot reviewed Jul 15, 2026

View reviewed changes

adityapk00 added 2 commits July 15, 2026 17:14

persist blocks before broadcast

39cf7ea

Merge branch 'main' of github.com:tempoxyz/zones into aditya/p2p_bloc…

47b14d6

…k_import # Conflicts: # crates/node/src/node.rs # crates/node/tests/it/utils.rs

adityapk00 marked this pull request as ready for review July 16, 2026 02:02

adityapk00 requested review from 0xKitsune and mattsse as code owners July 16, 2026 02:02

mattsse requested changes Jul 16, 2026

View reviewed changes

adityapk00 added 2 commits July 16, 2026 09:23

handle broadcast subscriber race conditions

f58caae

Merge branch 'main' into aditya/p2p_block_import

cc69af6

# Conflicts: # crates/node/tests/it/e2e.rs # crates/node/tests/it/utils.rs

mattsse approved these changes Jul 16, 2026

View reviewed changes

adityapk00 added this pull request to the merge queue Jul 16, 2026

Merged via the queue into main with commit 8765c14 Jul 16, 2026
16 checks passed

adityapk00 deleted the aditya/p2p_block_import branch July 16, 2026 15:21

0xKitsune reviewed Jul 17, 2026

View reviewed changes

		while let Some(persisted_tip) = persisted.next().await {
		if persisted_tip.number < last_broadcast {

		// At 30M gas, calldata is bounded below 7.5 MiB; leave headroom for block overhead.
		pub(crate) const MAX_MESSAGE_SIZE: u32 = 20 * 1024 * 1024;

Uh oh!

Conversation

adityapk00 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tempoxyz-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tempo-voight-kampff Bot commented Jul 16, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adityapk00 commented Jul 15, 2026 •

edited

Loading