Skip to content

fix: drain bridge replies before reporting disconnects - #9

Merged
steipete merged 1 commit into
openclaw:mainfrom
SebTardif:fix/readloop-tts-deadline
Oct 7, 2026
Merged

steipete merged 1 commit into
openclaw:mainfrom
SebTardif:fix/readloop-tts-deadline

Conversation

@SebTardif

@SebTardif SebTardif commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

A bridge can send pair-ok or hello-ok immediately before closing its socket. The reader previously published frames and EOF to separate channels, so a consumer could select EOF first and lose the successful reply. A real net.Pipe regression fails on main for both handshakes.

Use frame-stream closure to publish the terminal read error after all buffered frames. Pairing, hello, and the connected-node loop now drain frames before observing disconnect. This also removes the separate terminal error send raised by this PR; a departed consumer cannot strand the reader on that send. The previous saturated-error tests reused one client for two reader loops, which production never does. The replacement tests exercise the actual one-reader lifecycle, EOF, non-EOF read errors, and shutdown with a full frame buffer.

Thanks @SebTardif for the original reader-shutdown work. Contributor credit is retained in the commit.

Validation:

  • Baseline go test ./cmd/clawgo -run "TestBridgeHandshakeBeforeEOF|TestBridgeReaderStopsWithoutConsumer" -count=1 -v: both handshake cases fail with unexpected EOF; ordinary abandoned-consumer shutdown already passes.
  • Fixed focused tests pass, including 32 repetitions per handshake.
  • Independent Codex autoreview: no actionable P0–P2 findings.
  • Linux amd64 / Go 1.27.0 remote gate passed: go mod download, go test ./..., go test -race ./..., go build ./cmd/clawgo, and deadcode -test ./... with v0.49.0 produced no findings.

The first remote attempt stopped before testing because Go was absent from the bare image; the rerun installed the official checksum-verified Go 1.27.0 archive and passed all gates. No live gateway or speech hardware was required for this transport-level regression.

Production TCP proof (September 29, 2026)

Built the unmodified CLI at main 9d0188637d966f3c50b2977e2bce44fbf42835db and PR head ed3f5dc64ccbb779d96d2fb674c40ba2a6a26650 on Linux amd64 / Go 1.27.1. A Python loopback TCP peer received the real pair-request or hello, paused the CLI with SIGSTOP, sent a harmless event followed by the successful handshake reply and FIN, then resumed it with SIGCONT. Each connection had one production reader. The harmless event exposes the buffered-reply/EOF selection window after it is handled; a direct single-reply probe did not reproduce that timing window. Each binary was run with GOMAXPROCS=1 for 64 fresh connections per case.

For pairing, the probe asserted successful CLI exit and that the synthetic token was saved. For hello, the peer also sent a named synthetic event immediately after hello-ok; the probe asserted that the CLI processed that event before logging EOF and exited cleanly on SIGTERM. All state and frames were synthetic.

baseline: real TCP pair-ok immediately followed by FIN: 32/64 replies processed
baseline: real TCP hello-ok immediately followed by FIN: 15/64 replies processed
PR9: real TCP pair-ok immediately followed by FIN: 64/64 replies processed
PR9: real TCP hello-ok immediately followed by FIN: 64/64 replies processed

The PR head also passed go mod verify, go test ./..., go test -race ./..., go build ./cmd/clawgo, and deadcode -test ./... using the CI-pinned v0.49.0. Exact-head CI is green. No gateway deployment or speech hardware was used.

@clawsweeper

clawsweeper Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 2, 2026
@clawsweeper

clawsweeper Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed October 6, 2026, 3:32 PM ET / 19:32 UTC (Revision 18).

ClawSweeper review

What this changes

Clawgo drains buffered bridge replies and events before reporting disconnects, with regression tests and documentation for handshake ordering and reader shutdown.

Merge readiness

✅ Ready for maintainer review

This PR remains useful: current main still permits EOF to win over buffered handshake replies. The pinned patch resolves that ordering defect, has convincing production TCP proof, and has no actionable review findings.

Priority: P2
Reviewed head: ed3f5dc64ccbb779d96d2fb674c40ba2a6a26650

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused, correct patch with strong real TCP proof and useful regression coverage.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The pinned-head CLI's real TCP reader and handshake consumers were exercised against injected reply-plus-FIN timing on Linux; both handshakes succeeded in 64/64 runs, token persistence was asserted, and the connected loop processed a subsequent event before EOF. No stored-data contract changes.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The pinned-head CLI's real TCP reader and handshake consumers were exercised against injected reply-plus-FIN timing on Linux; both handshakes succeeded in 64/64 runs, token persistence was asserted, and the connected loop processed a subsequent event before EOF. No stored-data contract changes.
Evidence reviewed 7 items Pinned introduced change: The verified merge-base-to-head diff replaces the separate error channel with frame-stream closure in all three consumers. The reader writes its terminal error before closing the stream; consumers read that error only after observing closure.
Current main still needs the fix: Main sends EOF through errs separately from frames, and both handshake waits select between those channels. Buffered successful replies can therefore lose to EOF. GitHub confirms this PR remains unmerged at the pinned head; the releases endpoint returned no releases.
Real production TCP proof: The captured PR body, sourceRevision 2cc9cd521e43821a20b94e19b4e07db7d44483eb1040bef28f4540fe8b8444ba, records Linux amd64/Go 1.27.1 runs of the unmodified main and pinned-head CLI against a real loopback TCP peer. Successful replies followed by FIN were processed in 64/64 head runs for each handshake, versus 32/64 pairing and 15/64 hello baseline runs. Pairing asserted token persistence; hello asserted processing a subsequent named event before EOF and clean SIGTERM exit.
Findings None None.
Security None None.

How this fits together

Clawgo is a headless node client that receives handshake replies and events from a gateway bridge over TCP. Its bridge reader feeds pairing, connection setup, and the connected event loop.

flowchart TD
 A[Gateway TCP frames] --> B[Bridge reader]
 B --> C[Buffered frame stream]
 C --> D[Pairing and hello waits]
 C --> E[Connected event loop]
 B --> F[Stream closes after final frame]
 F --> G[Report disconnect and reconnect]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production versus test LOC production +19/-19; tests +133/-1 Production code has no net growth; added coverage targets ordering, terminal errors, and reader shutdown.

Technical review

Best possible solution:

Preserve one ordered bridge frame stream whose terminal error becomes observable only after buffered replies and events are consumed.

Do we have a high-confidence way to reproduce the issue?

Yes: the main source permits EOF to beat a buffered reply, and the posted real-CLI TCP transcript demonstrates that failure at the exact main SHA. This reviewer did not execute the reproduction.

Is this the best way to solve the issue?

Yes: publishing the terminal error through stream closure removes the competing notification path while preserving cancellation, deadlines, and read-error reporting.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against 9d0188637d96.

Labels

Label changes:

No label changes.

Label justifications:

  • P2: This repairs a timing-dependent bridge handshake failure with bounded scope and demonstrated TCP behavior.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The pinned-head CLI's real TCP reader and handshake consumers were exercised against injected reply-plus-FIN timing on Linux; both handshakes succeeded in 64/64 runs, token persistence was asserted, and the connected loop processed a subsequent event before EOF. No stored-data contract changes.
  • proof: sufficient: Contributor real behavior proof is sufficient. The pinned-head CLI's real TCP reader and handshake consumers were exercised against injected reply-plus-FIN timing on Linux; both handshakes succeeded in 64/64 runs, token persistence was asserted, and the connected loop processed a subsequent event before EOF. No stored-data contract changes.

Evidence

What I checked:

  • Pinned introduced change: The verified merge-base-to-head diff replaces the separate error channel with frame-stream closure in all three consumers. The reader writes its terminal error before closing the stream; consumers read that error only after observing closure. (cmd/clawgo/main.go:599, ed3f5dc64ccb)
  • Current main still needs the fix: Main sends EOF through errs separately from frames, and both handshake waits select between those channels. Buffered successful replies can therefore lose to EOF. GitHub confirms this PR remains unmerged at the pinned head; the releases endpoint returned no releases. (cmd/clawgo/main.go:687, 9d0188637d96)
  • Real production TCP proof: The captured PR body, sourceRevision 2cc9cd521e43821a20b94e19b4e07db7d44483eb1040bef28f4540fe8b8444ba, records Linux amd64/Go 1.27.1 runs of the unmodified main and pinned-head CLI against a real loopback TCP peer. Successful replies followed by FIN were processed in 64/64 head runs for each handshake, versus 32/64 pairing and 15/64 hello baseline runs. Pairing asserted token persistence; hello asserted processing a subsequent named event before EOF and clean SIGTERM exit. (ed3f5dc64ccb)
  • Focused regression coverage: The new tests cover 32 repetitions of each handshake before EOF, EOF propagation, non-EOF read errors, and shutdown with no consumer and a saturated frame buffer. Existing cancellation tests remain intact. Tests were inspected, not executed during this read-only review. (cmd/clawgo/readloop_test.go:40, ed3f5dc64ccb)
  • Related merged work is distinct: fix: honor context in waitForPair and waitForHello #6 merged cancellation-aware dialing and handshake waits, but retained separate frame/error channels. It does not supersede this ordering fix. The repository-wide PR listing identifies no replacement for this work. (cmd/clawgo/main.go, 840a348a3044)
  • Routing and history: Main's path history shows recent bridge setup work by Sebastien Tardif and adjacent connection/session work by Peter Steinberger; GitHub maps them to SebTardif and steipete. Local blame and deeper patch-history traversal failed when unavailable objects could not be fetched, so no source-line introduction attribution is claimed. GitHub commit patches independently verified the merged cancellation work. (cmd/clawgo/main.go, 9d0188637d96)

Likely related people:

  • SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (17 earlier review cycles; latest 8 shown)
  • reviewed 2026-09-23T07:25:27.119Z sha ed3f5dc :: needs real behavior proof before merge. :: none
  • reviewed 2026-09-30T05:06:23.001Z sha ed3f5dc :: needs maintainer review before merge. :: none
  • reviewed 2026-10-02T04:58:14.320Z sha ed3f5dc :: needs maintainer review before merge. :: none
  • reviewed 2026-10-02T11:24:58.759Z sha ed3f5dc :: needs maintainer review before merge. :: none
  • reviewed 2026-10-02T18:58:11.639Z sha ed3f5dc :: needs maintainer review before merge. :: none
  • reviewed 2026-10-04T10:50:27.822Z sha ed3f5dc :: needs maintainer review before merge. :: none
  • reviewed 2026-10-05T16:53:00.754Z sha ed3f5dc :: needs maintainer review before merge. :: none
  • reviewed 2026-10-05T23:24:17.810Z sha ed3f5dc :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. and removed P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. labels Sep 3, 2026
@SebTardif
SebTardif force-pushed the fix/readloop-tts-deadline branch from 55aa826 to b286a2b Compare September 18, 2026 16:47
@SebTardif

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Production saturation proof is now a live go run of readLoop / sendErr, not an injected unread value.

First server close fills the buffer-1 errs slot and leaves it unread (first_readLoop_returned=true, errs_slot_full=true). A second production readLoop then hits EOF again; Close() still returns (second_readLoop_returned=true).

Tip is b286a2b9cefd11515b716db9b3d22468f1d9e6b2 after rebasing onto current main.

@clawsweeper

clawsweeper Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Sep 18, 2026
Publish the reader error through frame-stream closure so EOF cannot win a
select against pending pair-ok, hello-ok, or event frames. Closing the stream
also avoids needing a separate blocking error send after consumer shutdown.

Replace the artificial saturated-client tests with one-reader pipe regressions
for reply ordering, EOF, read errors, and departed consumers.

Co-authored-by: Sebastien Tardif <sebtardif@ncf.ca>
@steipete steipete changed the title fix: do not block readLoop on a departed error consumer fix: drain bridge replies before reporting disconnects Sep 23, 2026
@steipete
steipete force-pushed the fix/readloop-tts-deadline branch from b286a2b to ed3f5dc Compare September 23, 2026 07:21
@steipete
steipete requested a review from vincentkoc September 23, 2026 07:21
@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. proof: sufficient Contributor real behavior proof is sufficient. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Sep 23, 2026
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Sep 30, 2026
@steipete
steipete merged commit bfb78a9 into openclaw:main Oct 7, 2026
12 of 13 checks passed
steipete pushed a commit that referenced this pull request Oct 7, 2026
Add -tts-system-timeout to bound one hung speech command so the queue can continue. The default remains 0, preserving unlimited healthy playback; shutdown cancellation and the espeak option separator remain intact.

The baseline regression failed after the child ran to completion, while the opt-in deadline passed. Linux module verification, tests, race tests, build, and deadcode passed again after integrating #9. Independent Codex review found no actionable P0–P2 issues, and exact-head CI passed at ae8b6d7.

Co-authored-by: Sebastien Tardif <sebtardif@ncf.ca>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants