Repository navigation
ensemble: bound every send, order the opening frame, prune dead routes - #62
Merged
Merged
Conversation
Harden the mesh against the failure modes PR #61 left open. Sends: every write now has a deadline. Liveness pongs, Kademlia queries and `kad_find` were unbounded; the opening hello was awaited raw, so a peer that never read left the task hung forever and the gate shut behind it. Fanout for publish, gossip and probes was serial, so wall time was the sum of the peers rather than the slowest one. Opening: `install` spawned the hello and raced application traffic onto the same link — a routed tone could reach the wire before the frame that authenticates the peer. A `Gate` now parks every send until the opening frame lands. The gate registers with `Notify` before reading its flag (a check-then-park ordering drops wakeups), `Default` matches `new`, and a shut gate stays shut. Identity: no tone is published before its link completes the signed hello; drops are counted in `tones_before_handshake()`. `strict_auth` defaults to true, so an unsigned peer is refused unless a caller explicitly waives it. `from` is deliberately left unvalidated — it names the originator, not the forwarding link, and this mesh relays addressed tones. Routing: a removed peer is forgotten from every bucket instead of staying addressable, and the table is persisted to `state_dir()/routing.json` so a restart resumes it. Framing: TCP, TLS and iroh share one NDJSON codec with a 256KiB frame ceiling. An oversize frame is discarded and the stream resyncs on the next newline rather than growing without bound. Tests: the opening-order test is pinned to the bug — with the gate removed it fails with `prompt` ahead of `hello`. CI gains a `mesh` job running clippy and the ensemble/sim/humd suites.
Rust 1.99 added `double_must_use`, which fires on the `#[async_trait]` attribute in handshake.rs: the macro stamps `#[must_use]` onto a `Pin<Box<dyn Future>>` return, so the lint lands in macro expansion rather than in our source. Allow it alongside the existing `type_complexity` allowance. Both lints are newer than the 1.98.1 toolchain this branch was verified on, and CI tracks `stable`, so they could not be reproduced or fixed here — they are allowed, not fixed.
`eviction_only_touches_the_dead_peer` sleeps 40ms and asserts the peer is evicted under a 120ms TTL. Every arriving tone renews the lease (lib.rs:821, the probe's pong included), so the TTL branch cannot fire in 40ms — the test only ever passed because humd's drain loop noticed the dropped sender and flipped the lease to TransportClosed within that window. Under CI load it does not, and the sweep returned []. Three sibling tests shared the shape. Replace the fixed sleeps with a poll on the non-destructive `peer_liveness`; `evict_expired` stays a single destructive call once the peer is genuinely dead.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #61. That PR landed a fault model and an honest message identity; this one closes the failure modes it left open.
Sends all have deadlines
Liveness pongs, Kademlia queries and
kad_findwrote without a bound. The opening hello was worse — awaited raw inside a spawned task, so a peer that never read left that task hung forever with the gate shut behind it and every later send parked. Now bounded, and a stalled opening closes the link instead of leaking.Fanout for publish, gossip and probes was serial, so wall time was the sum of the peers. Now the slowest peer.
a_fanout_costs_the_slowest_peer_not_the_sumpins it.The opening frame lands first
installspawned the hello and raced application traffic onto the same link — a routed tone could reach the wire before the frame that authenticates the peer. AGateparks every send until the opening frame has landed.The test is pinned to the bug, not to the fix. Remove the gate and it fails:
Three defects found while writing it:
Defaultproduced a permanently-shut gate (Mutex<bool>defaults false).wait()checked the flag then parked, so anopened()in that window was lost forever. Fixed by registering withNotifybefore reading the flag viaNotified::enable().opened()resurrected an already-shut gate.Identity
No tone publishes before its link completes the signed hello; drops are counted in
tones_before_handshake().strict_authnow defaults to true, so an unsigned peer is refused unless a caller explicitly waives it.fromis deliberately left unvalidated. This branch first enforcedfrom == transport_peerfor tones addressed to us.simproved it wrong:fromnames the originator — often a nestler, not a humd — and the mesh relays addressed tones, so the check rejected legitimate fan-out. Retracted rather than papered over.Routing
A removed peer is forgotten from every bucket instead of staying addressable forever, and the table is persisted to
state_dir()/routing.jsonso a restart resumes it. Restore is versioned and a corrupt file yields an empty table rather than a panic.Framing
TCP, TLS and iroh now share one NDJSON codec with a 256KiB frame ceiling. An oversize frame is discarded and the stream resyncs on the next newline instead of growing without bound.
CI
New
meshjob: clippy gate plus the ensemble/sim/humd suites — 209 tests, which nothing ran before.The
cargo fmt --checkgate was deliberately not added: HEAD is 567 hunks off rustfmt-clean, so it would be red on arrival and stay that way. Making the repo fmt-clean is its own change. For the same reason this branch carries no formatting churn — 21 files, all substance.Verification