XIP-83: Mutable subscription streams with liveness

[tylerhawkes]

Draft XIP, opened for circulation/discussion.

Summary

Defines a single bidirectional subscription RPC (Subscribe) on the MLS API. A client opens one long-lived stream and mutates its topic set in place (add/remove deltas) instead of tearing down and reopening on every membership change; the server delivers messages plus a periodic liveness heartbeat (StatusUpdate{ WAITING }) so clients can detect silent stream death that transport keepalives miss — a terminating L7 proxy answers HTTP/2 pings at the edge while the backend subscription is gone. One connection can carry the union of many topics, the enabling primitive for multi-tenant agent gateways.

Compatibility

Additive and backward-compatible: existing SubscribeGroupMessages / SubscribeWelcomeMessages are untouched, and WASM/browser clients (no bidirectional gRPC) stay on them. A client calling Subscribe against a node that lacks it gets UNIMPLEMENTED and falls back.

Status

Draft, for discussion. The client-side liveness floor — a WatchdogStream combinator that reconnects a stale subscription from its persisted cursor — is already implemented in libxmtp against the existing server-streaming subscriptions (xmtp/libxmtp#3718). The Subscribe RPC + node heartbeat handler standardized here are the remaining protocol work.

Note

Add XIP-83 specification for mutable subscription streams with liveness

Adds xip-83-mutable-subscription-streams.md, a protocol specification for a bidirectional Subscribe RPC supporting in-place subscription mutation (add/remove topics) without reconnecting.

• Defines application-level liveness via ping/pong with nonces, and live-boundary markers (TopicsLive and CatchupComplete) to signal when catch-up history is exhausted
• Specifies protobuf message shapes for MlsApi.Subscribe and a decentralized QueryApi.Subscribe binding, with versioned request/response envelopes and kind-prefixed binary topics
• Includes server/client requirements, bounded catch-up via history_only and half-close, explicit version pinning on the stream, test cases, and security considerations

📊 Macroscope summarized 14546e6. 1 file reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[macroscopeapp]

Approvability

Verdict: Needs human review

3 blocking correctness issues found. The new XIP specification is in a directory owned by @jhaaaa (not the author), and there are unresolved review comments identifying potential protocol design gaps around authorization and half-close semantics that the designated owner should evaluate.

^{You can customize Macroscope's approvability policy. Learn more.}

[neekolas]

Love this direction. Would be a huge unlock for Herald and also better for all mobile apps.

Main concern is that we have an even-more-different code path for the browser and everything else, and that browser issues can go unnoticed. Not a blocker, just an unfortunate side effect. Also means that our client streaming implementation needs to handle both mutable and immutable streams, which makes things harder to maintain. Such is life.

[macroscopeapp]

🟠 High XIPs/xip-83-mutable-subscription-streams.md:179

The Mutate.Subscription message (line 185) contains only topic and id_cursor, but server requirement 5 states that a single Subscribe stream may carry topics for multiple identities and that authorization MUST be evaluated per subscription. With no per-subscription identity or credential field, the node lacks data to authenticate which identity a newly added topic belongs to. An implementation must either authorize everything against the connection-level principal (breaking the gateway use case) or accept unauthenticated cross-identity adds, weakening authorization. Consider adding an installation_key or similar identity field to Subscription so the node can enforce per-subscription authorization.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @XIPs/xip-83-mutable-subscription-streams.md around line 179:

The `Mutate.Subscription` message (line 185) contains only `topic` and `id_cursor`, but server requirement 5 states that a single `Subscribe` stream may carry topics for multiple identities and that authorization MUST be evaluated per subscription. With no per-subscription identity or credential field, the node lacks data to authenticate which identity a newly added topic belongs to. An implementation must either authorize everything against the connection-level principal (breaking the gateway use case) or accept unauthenticated cross-identity adds, weakening authorization. Consider adding an `installation_key` or similar identity field to `Subscription` so the node can enforce per-subscription authorization.

[macroscopeapp]

🟡 Medium XIPs/xip-83-mutable-subscription-streams.md:337

The half-close rule in server requirement 9 (line 337) is incomplete for streams carrying live subscriptions. After the client half-closes, the node must stop sending Pings, finish catch-up waves, and close with OK — but the requirement never says whether live subscriptions are unregistered. If they stay registered, the node must keep delivering live traffic forever, so the promised OK close never happens. If they are dropped immediately, the client loses its signal for when live delivery ends. Either way, behavior is undefined for the common case (mutating a live stream) versus the special case (history_only bounded sync).

-When the client **half-closes** its request stream, the node MUST stop sending `Ping`s (a half-closed peer cannot answer; the client suspends its watchdog for the bounded drain and relies on gRPC transport timeouts — see client requirement 4), MUST finish all in-flight catch-up waves, and MUST then close the stream with `OK`; if no waves are in flight, it closes immediately. Together these give the bounded catch-up ("sync") flow with no extra protocol: open → `Mutate{ history_only }` → half-close → read until the server hangs up.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @XIPs/xip-83-mutable-subscription-streams.md around line 337:

The half-close rule in server requirement 9 (line 337) is incomplete for streams carrying live subscriptions. After the client half-closes, the node must stop sending `Ping`s, finish catch-up waves, and close with `OK` — but the requirement never says whether live subscriptions are unregistered. If they stay registered, the node must keep delivering live traffic forever, so the promised `OK` close never happens. If they are dropped immediately, the client loses its signal for when live delivery ends. Either way, behavior is undefined for the common case (mutating a live stream) versus the special case (`history_only` bounded sync).

[macroscopeapp]

🟡 Medium XIPs/xip-83-mutable-subscription-streams.md:218

TopicsLive at line 219 carries only repeated bytes topics with no mutate_id field. When overlapping Mutate waves include the same topic (per requirement 7 and the re-add with lower cursor case at line 283), the client cannot determine which wave's catch-up completed for that topic. This breaks the per-topic attribution for multiplexing clients that need to signal readiness per consumer. Consider adding mutate_id to TopicsLive so receivers can attribute live-boundary signals to specific waves.

     // Emitted when topics finish catch-up, AFTER their last history frame — including any
     // live messages that queued up behind the catch-up, which were equally historical from
     // the client's perspective — so that every later frame for a listed topic is live tail.
     message TopicsLive {
       repeated bytes topics = 1; // kind-prefixed topics now tailing live
+      uint64 mutate_id = 2; // echoes the Mutate wave that added these topics, 0 if none
     }

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @XIPs/xip-83-mutable-subscription-streams.md around lines 218-220:

`TopicsLive` at line 219 carries only `repeated bytes topics` with no `mutate_id` field. When overlapping `Mutate` waves include the same topic (per requirement 7 and the re-add with lower cursor case at line 283), the client cannot determine which wave's catch-up completed for that topic. This breaks the per-topic attribution for multiplexing clients that need to signal readiness per consumer. Consider adding `mutate_id` to `TopicsLive` so receivers can attribute live-boundary signals to specific waves.