Skip to content

fix: timeout synthetic probe invoke - #62

Merged
vincentkoc merged 3 commits into
openclaw:mainfrom
SebTardif:fix/f004-probe-invoke-timeout
Sep 9, 2026
Merged

vincentkoc merged 3 commits into
openclaw:mainfrom
SebTardif:fix/f004-probe-invoke-timeout

Conversation

@SebTardif

@SebTardif SebTardif commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes an issue where users running synthetic probes could hang on callbacks, continue dependent work after a timeout, or leave plugin code running after the helper CLI should have finished.

Why This Change Was Made

Keep programmatic probes in-process with finite callback deadlines, cooperative cancellation, and retained callback identity. Run CLI capture and callbacks together in the existing supervised capture runner, preserving its mock-SDK loader lifetime, bounded plugin output, and complete report delivery.

The CLI reads a fresh report separately from plugin stdout. The existing regular-file, nonblocking, byte-bounded reader is shared with import-loop profiling, so a FIFO or oversized artifact cannot bypass the child boundary. No timeout constants or resolver helpers are added to public exports.

User Impact

  • Callback budgets use valid API options, then PLUGIN_INSPECTOR_PROBE_TIMEOUT_MS, then 30 seconds. Zero, infinite, malformed, and overflowing values do not disable limits.
  • Timeout fails the current probe and blocks dependent work. Cancellation rejects the API call, propagates supported signals, and stops later inspector-owned work. Late promise rejection stays observed.
  • The CLI bounds the whole child, including import and registration, and handles cancellation, busy loops, retained intervals, and output floods. PROBE intentionally uses capture's 10 MiB report capacity; intercepted stdout and stderr retain their separate 1 MiB caps.
  • Complete reports with ordinary failed or blocked probe rows are still delivered successfully for the existing CI-policy consumer. Private CLI validation checks report shape, row identities and statuses, and all four summary counters before publication. Process timeout, cancellation, truncation, and oversized/nonregular/malformed artifacts fail without writing a new success artifact.
  • Public APIs preserve caller runtime objects and callback identity. In-process deadlines cannot forcibly stop synchronous JavaScript or arbitrary plugin side effects.

Evidence

  • Preserves contributor head bf4238e8e2e1a53e832559d744a358990a069cc3 and integrates main 340f18e26558c5f6530d322ac399b76b25e6b267, retaining the landed child supervisor, real capture policy, serial lifecycle order, and CJS loader changes.
  • Sanitized direct AWS proof on Node 22.19.0: exact contributor head plus fingerprinted candidate patch, IMDS IAM credential endpoint verified 404, isolated HOME, no hydration or provider credentials.
  • Red baseline: eight behavioral regressions failed as intended. A separate reader reproduction showed the FIFO blocking after child exit; the oversized-file control was already rejected.
  • Green candidate: 70 focused tests, then all 385 tests and package-content checks passed. Coverage includes actual mock/real CLI hangs, cancellation, retained intervals, roughly 9 MiB healthy reports with plugin logs, failed-row artifacts, output bounds, FIFO/oversized-file bypasses, API deadlines/identity, and retained CJS callbacks.
  • Original green source tree: c59d84a1bf3f08104a259d7d2c7c4026b763f7b3. Protocol red head 295746581c7381567e58aa9e6f07157cf62b7321 kept that production unchanged: secretless Node 22 Check 34376448341 produced exactly 52 expected malformed-report rejection failures and 389 passes, including the valid-report controls.
  • Final head: 6aa4aff1336005a75014b449b97e0aef6d129f60; tree: 145fd0a66a21491b44db772fed2d4f85889d6053. Exact-head secretless Node 22 Check 34376704253 passed all 441 tests with zero failures, cancellations, or skips, plus package-content validation (job 102551213467).
  • Fresh P2 identified malformed protocol acceptance; the approved correction is implemented, and scoped follow-up P2 is clean. The original finding is retained. This is protocol validation, not authenticated completion: same-process plugin code can still fabricate a valid report. No token, IPC authentication, HMAC, or sandbox mechanism was added.

The maintainer approved the finite 30-second default and override contract. The original test-only timeout exports were removed. No new public timeout resolver or authentication mechanism is introduced. Exact-head CI is green; final maintainer review and squash-merge clearance were granted for 6aa4aff1336005a75014b449b97e0aef6d129f60.

Linux owned-child proof only; Windows keeps the existing direct-child cleanup contract. Gateway response validation is deferred to its separate unit. No dependency, package version, or release change. Crabpot followthrough remains assigned to release integration.

Thanks @SebTardif for the original timeout repair.

@SebTardif
SebTardif requested a review from a team as a code owner August 29, 2026 18:27
@clawsweeper

clawsweeper Bot commented Aug 29, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 29, 2026
@clawsweeper

clawsweeper Bot commented Aug 29, 2026 •

Copy link
Copy Markdown

Codex review: blocked before merge. Reviewed September 4, 2026, 12:39 PM ET / 16:39 UTC.

ClawSweeper review

What this changes

This PR adds configurable deadlines to synthetic hook and registration invocations so never-settling handlers are reported as failed probes instead of blocking the runner.

Regression provenance

Possible regression — probable (reviewed change; failure trace). No predecessor PR is attributed.

Merge readiness

⛔ Blocked before merge - 6 items remain

Keep open: the timeout addresses a source-proven hang and has direct after-fix terminal evidence, but it introduces a new 30-second failure default and public timeout controls without an approved compatibility contract or the required Crabpot consumer smoke.

Priority: P2
Reviewed head: bf4238e8e2e1a53e832559d744a358990a069cc3
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The real-path proof is strong, but the public timeout and upgrade contract remains unresolved.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The PR body supplies a terminal trace from the changed public captureEntrypoint plus runCapturedSyntheticProbes path: a retained hanging hook under a 200ms override produced a failed result in 202ms. Source inspection confirms hooks and registrations both reach the changed runProbe owner; the trace is sufficient for the claimed hanging-invocation recovery, though it does not settle upgrade compatibility.
Patch quality 🦐 gold shrimp (3/6) 2 actionable review findings remain.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The PR body supplies a terminal trace from the changed public captureEntrypoint plus runCapturedSyntheticProbes path: a retained hanging hook under a 200ms override produced a failed result in 202ms. Source inspection confirms hooks and registrations both reach the changed runProbe owner; the trace is sufficient for the claimed hanging-invocation recovery, though it does not settle upgrade compatibility.
Evidence reviewed 6 items Current behavior is unbounded: The current main parent directly awaits each synthetic invocation, so a never-settling handler cannot produce a result.
Introduced default changes existing probe outcome: The branch resolves a 30-second default and races every probe invocation against it, changing a previously unbounded public runner into one that records slow handlers as failures.
Timeout helpers become package API: The package exports the synthetic-probes entrypoint, and the branch also adds defaultTimeoutMs to the documented root synthetic facade.
Findings 2 actionable findings [P1] Do not impose an unapproved 30-second probe default
[P2] Run the required Crabpot source-mode smoke
Security None None.

How this fits together

Plugin Inspector captures plugin hooks and registrations, then invokes synthetic inputs to produce an inspection report. This change sits in the in-process probe runner between captured handlers and the resulting pass/fail records.

flowchart LR
  A[Plugin entrypoint] --> B[Captured hooks and registrations]
  B --> C[Synthetic probe runner]
  C --> D[Invocation deadline]
  D --> E[Pass or fail result]
  E --> F[Inspection report]
Loading

Decision needed

Question Recommendation
Should synthetic probes adopt a stable 30-second default deadline with timeoutMs and PLUGIN_INSPECTOR_PROBE_TIMEOUT_MS as supported public controls? Adopt and document the timeout contract: Approve the bounded default, keep the controls public, and require slow-success, explicit-override, and Crabpot consumer validation before merge.

Why: This changes established execution semantics for existing users and exposes new package configuration; source evidence cannot establish the intended compatibility policy.

Before merge

  • Do not impose an unapproved 30-second probe default (P1) - Current main permits synthetic handlers to run without a deadline; this new default turns any legitimate slower handler into a failed result. The package also exposes the resolver through public entrypoints, but neither the README nor maintainer discussion establishes the timeout/override contract. Obtain that decision and document/test slow success and overrides, or keep the new controls and default out of the supported API.
  • Run the required Crabpot source-mode smoke (P2) - This patch changes synthetic-probe behavior and package entrypoints. Repository policy requires the Crabpot plugin-inspector pin/docs/smoke path to be updated as needed and its source-mode smoke run before completion; the supplied proof covers only the local API path.
  • Resolve merge risk (P1) - Existing consumers with legitimate synthetic handlers exceeding 30 seconds will receive failed probe results after upgrade unless the new default and override contract is explicitly accepted.
  • Resolve merge risk (P1) - The changed package behavior and entrypoint surface have not yet received the required Crabpot source-mode smoke follow-through.
  • Complete next step (P2) - Obtain maintainer approval for the timeout/default contract, then validate slow successful probes, explicit overrides, and the Crabpot source-mode smoke on the exact head.
  • Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

  • [P1] Do not impose an unapproved 30-second probe default — src/synthetic-probes.js:3-17
  • [P2] Run the required Crabpot source-mode smoke — src/index.js:148
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test delta production +46/-5, tests +23/-0 A small probe-runner change alters package-visible execution behavior while adding only hanging-hook coverage.

Merge-risk options

Maintainer options:

  1. Settle and prove the timeout policy (recommended)
    After maintainer approval, document the supported default and overrides, test a slow successful probe, and run Crabpot's source-mode plugin-inspector smoke on this head.
  2. Avoid the new default for now
    If a bounded probe deadline is not an accepted upgrade contract, remove the public timeout surface and retain existing behavior until a narrower policy is approved.

Technical review

Best possible solution:

Settle whether bounded synthetic execution and its override controls are a stable supported contract, then document and test slow-success/override behavior and validate the exact head through Crabpot.

Do we have a high-confidence way to reproduce the issue?

Yes: current main directly awaits probe invocation with no deadline, and the supplied post-fix terminal trace exercises the public captured-hook path with a hanging Promise.

Is this the best way to solve the issue?

No: racing the invocation is a targeted mechanism, but the forced 30-second default and newly public controls need an explicit compatibility decision plus slow-success and override validation.

Full review comments:

  • [P1] Do not impose an unapproved 30-second probe default — src/synthetic-probes.js:3-17
    Current main permits synthetic handlers to run without a deadline; this new default turns any legitimate slower handler into a failed result. The package also exposes the resolver through public entrypoints, but neither the README nor maintainer discussion establishes the timeout/override contract. Obtain that decision and document/test slow success and overrides, or keep the new controls and default out of the supported API.
    Confidence: 0.94
  • [P2] Run the required Crabpot source-mode smoke — src/index.js:148
    This patch changes synthetic-probe behavior and package entrypoints. Repository policy requires the Crabpot plugin-inspector pin/docs/smoke path to be updated as needed and its source-mode smoke run before completion; the supplied proof covers only the local API path.
    Confidence: 0.97

Overall correctness: patch is incorrect
Overall confidence: 0.91

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 92db8c57e1d5.

Labels

Label changes:

  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body supplies a terminal trace from the changed public captureEntrypoint plus runCapturedSyntheticProbes path: a retained hanging hook under a 200ms override produced a failed result in 202ms. Source inspection confirms hooks and registrations both reach the changed runProbe owner; the trace is sufficient for the claimed hanging-invocation recovery, though it does not settle upgrade compatibility.
  • remove status: 👀 ready for maintainer look: Current PR status label is status: ⏳ waiting on author.

Label justifications:

  • P2: The PR targets a bounded inspection-runner hang with limited operator blast radius.
  • merge-risk: 🚨 compatibility: The new default converts previously unbounded synthetic probes into failures after 30 seconds for existing consumers.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body supplies a terminal trace from the changed public captureEntrypoint plus runCapturedSyntheticProbes path: a retained hanging hook under a 200ms override produced a failed result in 202ms. Source inspection confirms hooks and registrations both reach the changed runProbe owner; the trace is sufficient for the claimed hanging-invocation recovery, though it does not settle upgrade compatibility.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies a terminal trace from the changed public captureEntrypoint plus runCapturedSyntheticProbes path: a retained hanging hook under a 200ms override produced a failed result in 202ms. Source inspection confirms hooks and registrations both reach the changed runProbe owner; the trace is sufficient for the claimed hanging-invocation recovery, though it does not settle upgrade compatibility.

Evidence

What I checked:

  • Current behavior is unbounded: The current main parent directly awaits each synthetic invocation, so a never-settling handler cannot produce a result. (src/synthetic-probes.js:970, 92db8c57e1d5)
  • Introduced default changes existing probe outcome: The branch resolves a 30-second default and races every probe invocation against it, changing a previously unbounded public runner into one that records slow handlers as failures. (src/synthetic-probes.js:3, bf4238e8e2e1)
  • Timeout helpers become package API: The package exports the synthetic-probes entrypoint, and the branch also adds defaultTimeoutMs to the documented root synthetic facade. (src/index.js:148, bf4238e8e2e1)
  • Consumer-smoke policy applies: Repository policy requires Crabpot pin/docs/smoke follow-through when plugin-inspector behavior or package entrypoints change; this PR changes both execution behavior and exported entrypoints. (AGENTS.md:18, bf4238e8e2e1)
  • Synthetic runner provenance: History identifies the original synthetic probe runner commit, supporting routing to the feature's earlier owner. (src/synthetic-probes.js, 61603b5ca3c6)
  • Real after-fix behavior proof: The PR body records a macOS Node run through captureEntrypoint and runCapturedSyntheticProbes where a hanging hook returns a failed result in 202ms under a 200ms override. (bf4238e8e2e1)

Likely related people:

  • Vincent Koc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Obtain maintainer approval for the 30-second default and supported override surface.
  • Add slow-success and explicit-override coverage, then attach redacted Crabpot source-mode smoke output for this head.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (11 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-31T06:05:18.860Z sha e3be165 :: needs changes before merge. :: [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-08-31T13:03:55.892Z sha e3be165 :: found issues before merge. :: [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-09-01T09:10:50.382Z sha e3be165 :: found issues before merge. :: [P1] Avoid committing an unapproved public timeout API | [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-09-01T21:20:07.305Z sha e3be165 :: found issues before merge. :: [P1] Keep the timeout helpers out of the public contract | [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-09-02T14:09:22.439Z sha e3be165 :: needs changes before merge. :: [P1] Avoid publishing an unapproved timeout API | [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-09-02T23:00:45.615Z sha e3be165 :: blocked before merge. :: [P1] Keep timeout controls out of the public contract | [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-09-03T13:51:39.237Z sha e3be165 :: blocked before merge. :: [P1] Keep timeout controls out of the public contract | [P2] Run the required Crabpot source-mode smoke
  • reviewed 2026-09-04T07:03:05.538Z sha e3be165 :: blocked before merge. :: none

@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Sep 4, 2026
@SebTardif SebTardif closed this Sep 4, 2026
@SebTardif
SebTardif force-pushed the fix/f004-probe-invoke-timeout branch from e3be165 to 92db8c5 Compare September 4, 2026 16:32
Signed-off-by: Sebastien Tardif <SebTardif@ncf.ca>
@SebTardif SebTardif reopened this Sep 4, 2026
@clawsweeper clawsweeper Bot added status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 4, 2026
Integrate main while preserving the contributor repair and retained callback identity. Add malformed-report regressions for the approved protocol-validation follow-up; these cases intentionally establish the red CI baseline.
Validate report shape, row identities and statuses, and exact safe-integer counters before artifact publication. Preserve empty and completed failed or blocked reports. This is protocol validation, not same-process authentication.
@vincentkoc
vincentkoc merged commit dc2bdb3 into openclaw:main Sep 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants