Skip to content

ensemble: bound every send, order the opening frame, prune dead routes - #62

Merged
adiled merged 3 commits into
mainfrom
ensemble/production-grade
Oct 4, 2026
Merged

adiled merged 3 commits into
mainfrom
ensemble/production-grade

Conversation

@adiled

@adiled adiled commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Follows #61. That PR landed a fault model and an honest message identity; this one closes the failure modes it left open.

Sends all have deadlines

Liveness pongs, Kademlia queries and kad_find wrote without a bound. The opening hello was worse — awaited raw inside a spawned task, so a peer that never read left that task hung forever with the gate shut behind it and every later send parked. Now bounded, and a stalled opening closes the link instead of leaking.

Fanout for publish, gossip and probes was serial, so wall time was the sum of the peers. Now the slowest peer. a_fanout_costs_the_slowest_peer_not_the_sum pins it.

The opening frame lands first

install spawned the hello and raced application traffic onto the same link — a routed tone could reach the wire before the frame that authenticates the peer. A Gate parks every send until the opening frame has landed.

The test is pinned to the bug, not to the fix. Remove the gate and it fails:

left: String("prompt")
right: "hello"

Three defects found while writing it:

  • Default produced a permanently-shut gate (Mutex<bool> defaults false).
  • wait() checked the flag then parked, so an opened() in that window was lost forever. Fixed by registering with Notify before reading the flag via Notified::enable().
  • opened() resurrected an already-shut gate.

Identity

No tone publishes before its link completes the signed hello; drops are counted in tones_before_handshake(). strict_auth now defaults to true, so an unsigned peer is refused unless a caller explicitly waives it.

from is deliberately left unvalidated. This branch first enforced from == transport_peer for tones addressed to us. sim proved it wrong: from names the originator — often a nestler, not a humd — and the mesh relays addressed tones, so the check rejected legitimate fan-out. Retracted rather than papered over.

Routing

A removed peer is forgotten from every bucket instead of staying addressable forever, and the table is persisted to state_dir()/routing.json so a restart resumes it. Restore is versioned and a corrupt file yields an empty table rather than a panic.

Framing

TCP, TLS and iroh now share one NDJSON codec with a 256KiB frame ceiling. An oversize frame is discarded and the stream resyncs on the next newline instead of growing without bound.

CI

New mesh job: clippy gate plus the ensemble/sim/humd suites — 209 tests, which nothing ran before.

The cargo fmt --check gate was deliberately not added: HEAD is 567 hunks off rustfmt-clean, so it would be red on arrival and stay that way. Making the repo fmt-clean is its own change. For the same reason this branch carries no formatting churn — 21 files, all substance.

Verification

cargo clippy -p ensemble -p hum-paths --all-targets -- -D warnings -A clippy::type_complexity
cargo test --locked -p ensemble -p sim -p humd -p hum -p humctl
# 32 suites, 209 passed, 0 failed

adiled added 3 commits October 4, 2026 15:32
Harden the mesh against the failure modes PR #61 left open.

Sends: every write now has a deadline. Liveness pongs, Kademlia
queries and `kad_find` were unbounded; the opening hello was awaited
raw, so a peer that never read left the task hung forever and the gate
shut behind it. Fanout for publish, gossip and probes was serial, so
wall time was the sum of the peers rather than the slowest one.

Opening: `install` spawned the hello and raced application traffic onto
the same link — a routed tone could reach the wire before the frame
that authenticates the peer. A `Gate` now parks every send until the
opening frame lands. The gate registers with `Notify` before reading its
flag (a check-then-park ordering drops wakeups), `Default` matches
`new`, and a shut gate stays shut.

Identity: no tone is published before its link completes the signed
hello; drops are counted in `tones_before_handshake()`. `strict_auth`
defaults to true, so an unsigned peer is refused unless a caller
explicitly waives it. `from` is deliberately left unvalidated — it
names the originator, not the forwarding link, and this mesh relays
addressed tones.

Routing: a removed peer is forgotten from every bucket instead of
staying addressable, and the table is persisted to
`state_dir()/routing.json` so a restart resumes it.

Framing: TCP, TLS and iroh share one NDJSON codec with a 256KiB frame
ceiling. An oversize frame is discarded and the stream resyncs on the
next newline rather than growing without bound.

Tests: the opening-order test is pinned to the bug — with the gate
removed it fails with `prompt` ahead of `hello`. CI gains a `mesh` job
running clippy and the ensemble/sim/humd suites.
Rust 1.99 added `double_must_use`, which fires on the `#[async_trait]`
attribute in handshake.rs: the macro stamps `#[must_use]` onto a
`Pin<Box<dyn Future>>` return, so the lint lands in macro expansion
rather than in our source. Allow it alongside the existing
`type_complexity` allowance.

Both lints are newer than the 1.98.1 toolchain this branch was verified
on, and CI tracks `stable`, so they could not be reproduced or fixed
here — they are allowed, not fixed.
`eviction_only_touches_the_dead_peer` sleeps 40ms and asserts the peer
is evicted under a 120ms TTL. Every arriving tone renews the lease
(lib.rs:821, the probe's pong included), so the TTL branch cannot fire
in 40ms — the test only ever passed because humd's drain loop noticed
the dropped sender and flipped the lease to TransportClosed within that
window. Under CI load it does not, and the sweep returned [].

Three sibling tests shared the shape. Replace the fixed sleeps with a
poll on the non-destructive `peer_liveness`; `evict_expired` stays a
single destructive call once the peer is genuinely dead.
@adiled
adiled merged commit f968e39 into main Oct 4, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant