[Decentralisation] Protocol to interact with Provider Nodes directly

[mudigal]

Currently we use REST APIs, this would be more centralized approach. We need to find a better decentralized way to write and read data.

Would Smoldot be good here or do we need to evaluate something else?

[mudigal]

Investigation Summary: Decentralized Read & Write Protocols

After analyzing the codebase and the current checkpoint/multi-provider architecture, here's a breakdown of the protocol options for replacing REST APIs with decentralized alternatives.

---

The Core Asymmetry: Reads vs Writes

Reads and writes have fundamentally different properties and need different protocols:

• Reads are content-addressed — any peer with the data can serve it. This is naturally decentralized.
• Writes are inherently directed — they must reach the specific provider who holds the agreement, builds the MMR, and signs checkpoints. You cannot broadcast writes.

---

Read Path: libp2p + Bitswap

Content-addressed reads map cleanly to existing P2P content distribution protocols:

Layer	Current (REST)	Proposed
Discovery	RPC query for provider URL	DHT lookup by content hash (CID)
Transport	GET /s3/:bucket_id/object to one server	Bitswap request — any peer with the data serves it
Verification	Trust the provider	Content-addressed — hash verification on receipt
Availability	Single point of failure	Multiple peers (primaries + replicas) can serve

Smoldot is relevant here for light-client chain queries (e.g., verifying which providers hold a bucket) without depending on a centralized RPC node.

---

Write Path: Leader-Based Writes via libp2p

This is the harder problem. The current codebase reveals the answer through the checkpoint mechanism.

Key constraint: The checkpoint extrinsic (pallet/src/lib.rs:2071) requires all primary providers to sign the same (mmr_root, start_seq, leaf_count). This means all primaries must converge on identical MMR state. If the client wrote to multiple primaries independently, each would assign different sequence numbers and produce divergent MMR roots — the checkpoint would fail.

Leader Election Is Deterministic and Off-Chain

The leader election does NOT require any on-chain call. The algorithm (pallet/src/lib.rs:3654) is pure deterministic math:

fn calculate_leader_index(bucket_id: BucketId, window: u64, num_providers: u32) -> u32 {
    let mut data = [0u8; 16];
    data[..8].copy_from_slice(&bucket_id.to_le_bytes());
    data[8..].copy_from_slice(&window.to_le_bytes());
    let hash = sp_io::hashing::blake2_256(&data);
    let seed = u32::from_le_bytes([hash[0], hash[1], hash[2], hash[3]]);
    seed % num_providers
}

It's blake2_256(bucket_id || window) % num_providers. Any client or provider can compute the same result locally given three inputs:

1. bucket_id — known at upload time
2. window — derived from current_block / checkpoint_interval (block number is public knowledge)
3. num_providers — count of primary providers for the bucket

The only piece that requires a chain query is num_providers (and the provider list to map the index to an account ID). But this changes rarely — only when providers are added or removed from a bucket. So the client:

1. Caches primary_providers list at connection time (or on bucket creation)
2. Subscribes to ProviderAddedToBucket / ProviderRemovedFromBucket events to invalidate the cache
3. Computes the leader locally: blake2_256(bucket_id || current_window) % cached_num_providers
4. Writes directly to the leader via libp2p — zero chain calls per write

The leader rotation happens automatically as the checkpoint window advances (block-time driven), and every participant independently computes the same answer. This is exactly what the code comment says: "This ensures all providers can independently calculate who the leader is."

Write Protocol Flow

Client (computes leader locally)
  |
  |--- libp2p stream ---> Leader (Provider A)
                              |
                              | builds MMR, assigns canonical seq numbers
                              | returns signed delivery receipt
                              |
                     +--------+--------+
                     | gossip | gossip |
                     v        v        v
                Provider B  Provider C  Provider D
                (sync, verify, co-sign)
                     |        |        |
                     +--------+--------+
                              |
                     Checkpoint on-chain
                  (mmr_root + N signatures)

Protocol steps:

1. Client computes the current write leader locally using the deterministic formula (no chain call).
2. Client writes to the leader via libp2p protocol stream (e.g., /web3-storage/write/1.0.0).
3. Leader assigns canonical sequence numbers, builds MMR, returns a signed delivery receipt (hash + seq + timestamp).
4. Leader gossips data + sequence assignments to other primaries via libp2p GossipSub.
5. Other primaries verify, store, and build identical MMR state.
6. All sign the same checkpoint — leader submits with collected signatures.

Leader failover: The provider_checkpoint extrinsic already has a grace period after which any primary can submit the checkpoint. Extend the same logic to writes — if the leader is unreachable, the client falls back to the next primary in the deterministic rotation. No chain call needed for failover either — the client just tries (seed + 1) % num_providers.

---

Signed Delivery Receipts (New Proof Primitive)

Currently, if a provider accepts data via HTTP then denies it, the client has no evidence. Signed delivery receipts close this gap. The leader signs (data_hash, sequence_number, timestamp) on receipt, and this receipt is submittable on-chain as proof of delivery during disputes.

---

Smoldot's Role

Smoldot is not a replacement for REST — it's a light client for chain state queries. Its role here is:

• On providers: Verify on-chain state (agreements, checkpoints, challenges) without depending on a centralized RPC endpoint. Currently providers use subxt with RPC.
• On clients (JS): Already available via polkadot-api v2.1.0 in console-ui and drive-ui. Not yet in the Rust SDK.
• Not for data transfer: Smoldot queries chain state; it doesn't transfer storage data between client and provider.

---

Full Protocol Stack

┌─────────────────────────────────────────────────────┐
│  READS                          WRITES              │
│  ─────                          ──────              │
│  libp2p + Bitswap               libp2p streams      │
│  (content-addressed,            (to leader, with    │
│   any peer serves)               signed receipts)   │
│                                                     │
│  LEADER ELECTION (no chain call needed)             │
│  ──────────────────────────────────────             │
│  blake2_256(bucket_id || window) % num_providers    │
│  Locally computable by all participants             │
│                                                     │
│  CHAIN QUERIES        INTER-PROVIDER SYNC           │
│  ────────────         ───────────────────           │
│  Smoldot light        libp2p GossipSub              │
│  client               (leader → other primaries)    │
│  (rare: provider                                    │
│   list changes only)                                │
│                                                     │
│  DISCOVERY                                          │
│  ─────────                                          │
│  Kademlia DHT + on-chain peer ID registry           │
└─────────────────────────────────────────────────────┘

Why Not Write-to-All?

• MMR is append-ordered — if chunks arrive in different order at different providers, roots diverge.
• Checkpoint requires agreement on one root — would fail InsufficientSignatures.
• Would need a consensus protocol among providers (Raft/PBFT), which is overkill when deterministic leader rotation already exists.
• Sharding writes across providers breaks the challenge mechanism — which provider do you challenge for chunk N?

Suggested Next Steps

1. Implement the deterministic leader election in the client SDK and provider node (replicate the calculate_leader_index formula — it's just blake2 + modulo)
2. Define the /web3-storage/write/1.0.0 libp2p protocol spec (message format, flow control, receipt signing)
3. Define the GossipSub topic structure for inter-provider data sync
4. Prototype Bitswap reads on the provider node alongside existing REST (additive, not replacing)
5. Evaluate whether Smoldot embedding in the Rust provider node is worth the effort vs. RPC + finality checks (which already work via wait_for_finalized_success())
6. Design the leader failover protocol for writes (try next provider in deterministic rotation)

[eskimor]

While we can support content addressed lookup via a DHT, for e.g. compatibility. This was not meant to be the primary way. The idea was that the discover flow is via the chain - you know a bucket you are interested, you find the providers on chain and fetch directly. Just adding indirection on top does not make it more decentralized. It is decentralized, because you can have multiple providers per bucket and anyone can add more.

We specifically wanted to avoid DHT discovery because it is super slow. Also having the bucket id as identifier allows for mutable decentralized references.

For the leader election, I am still not 100% convinced it is needed*) and do have concerns. E.g. what about buckets without checkpoints? Does it still work? How do we deterministically arrive at the same leader exactly, are we considering network asynchronicity? Block of the last finished slot is mitigating asynchronicity most of the time.

*) Why would we arrive at different sequence numbers or roots? The assumption is a single or at least coordinated writers. Ordering can be done deterministically, new roots can be calculated if something arrives later than intended, ... not sure why arriving at the same root should be any problem ... Ok, I guess you are thinking of un-coordinated writers and indeed with a leader we could support that. 🤔 Ok, that might be worth it ..... especially if you use some conflict free data representation ... 🤔

[eskimor]

I also don't see why REST API means we have to trust the provider - we can also verify hashes via a REST API. For availability the same is true - you have a list of providers, you can query the data from all of them via REST.

Not saying that a HTTP REST API is the best way to access the providers, but it is an established way that works ... so not the worst option and I definitely don't see it has less decentralized than some other API.