Skip to content

Investigate intermittent Windows cold-start failure in command supervisor #305

Open

Description

@vincentkoc

Problem

Unattributed intermittent Windows cold-start failure in the existing process owner, observed while validating #304. It is not established as preexisting or unrelated to that candidate.

The first Inspector checkout git init, before fetch or npm installation, failed during test/capture-contracts.test.mjs module import:

EOWNERSTART: command supervisor startup timed out; command cleanup was not confirmed

The caller is ensurePinnedInspectorCheckout() in scripts/plugin-inspector-source.mjs; command startup and cleanup are owned by scripts/owned-command.mjs.

Evidence

  • Original red, head 9b19ec198df5d9ff49e7910ec7c166545e753dc7: https://github.com/openclaw/crabpot/actions/runs/34452925787/job/102792576498. Windows Node 24.20.0 recorded 398 tests: 389 passed, one failed, eight skipped. The failure happened under the real cold static-suite invocation, node --test --test-concurrency=1 test/*.test.mjs. Other required platforms passed. Later best-effort report success does not override the failed suite.
  • One approved failure-only caller-diagnostic run, head 2515b0569e32e57a63af0550a116ac137f821916: https://github.com/openclaw/crabpot/actions/runs/34455244708/job/102799999787. All four required platforms and the aggregate passed. Windows Node 24.19.0 recorded 398 tests: 390 passed, zero failed, eight skipped; all 19 static steps completed.
  • The automatic diagnostic-head advisory also passed: https://github.com/openclaw/crabpot/actions/runs/34455244608/job/102800432709. Windows had the same counts and Node version.
  • Neither diagnostic-head Windows log emitted the failure-only JSON record. Successful cold contract import and native/bootstrap/cancellation/compiler/Job controls were observed, but those separate controls do not establish the original failed helper's fate.

The temporary diagnostic changed only the existing caller's error branch. It did not change the owner, PowerShell/native protocol, startup or cleanup budgets, cold preparation, test order, dependencies, or workflows. A passing instrumented run is not a reproduction, root-cause attribution, or fix. The Node version difference is evidence, not an explanation.

Earlier startup observations are retained in #292; no common cause is asserted. Searches for EOWNERSTART, cold-start, and Windows startup issues found no dedicated follow-up before creating this issue.

Diagnostic-Free Candidate Proof

The single authorized final run passed after removing only the temporary diagnostic. Head e7694efd9f595c3c514eab35c542a303b0081db0 has exactly the original 9b19ec1 tree, a0559d10c9d14ffff33ba0c8d4170c7c99908aed; no process-owner repair, timeout increase, prewarming, or test-order change was made.

Both logs show successful actual cold contract import, 398 tests (390 passed, zero failed, eight skipped), all 19 static steps, and the existing native/bootstrap/cancellation/compiler/Job controls. Both workflows completed on attempt 1.

The required result supplies direct clean-candidate Windows coverage on 24.20.0. It does not explain the original failure, establish a Node-version cause, prove a repair, or close this follow-up. Earlier 24.19.0 diagnostic passes and other PRs' 24.20.0 passes are not treated as controlled reproductions or attribution.

Follow-Up Boundary

Investigate the startup phase and cleanup uncertainty in the process owner using attributable evidence from the actual cold CI flow. Distinguish Worker startup, helper connection/request handling, READY, and terminal cleanup only where receipts support the distinction.

Do not infer extinction from separate passing controls or a successful subsequent run. Do not use retries-until-green, larger budgets, prewarming, or suppression as a fix. The agreed final-run stop condition was no landing on recurrence and a separately reviewed owner repair; no recurrence was observed in that single run.

No private raw artifacts are attached. Public run links above retain the original failure and instrumented results. The follow-up records unresolved risk; it does not claim a cause or repair.

Activity

  1. added
    P2Normal priority bug or improvement with limited blast radius.
    impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.
    issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.
    on Sep 10, 2026
  2. clawsweeper commented on Sep 10, 2026

    @clawsweeper
    Contributor

    Codex review: this still needs some work. Reviewed October 7, 2026, 1:29 PM ET / 17:29 UTC.

    Summary
    Keep open: current main improves cancellation and diagnostics but does not establish a repair for the intermittent cold-start failure. Later evidence records another startup failure, and this member-authored investigation remains necessary.

    Reproducibility: no. high-confidence deterministic current-main reproduction is established. Real Windows cold-flow failures are recorded, but their intermittent trigger remains unknown; no target code was executed during this read-only review.

    Root-cause cluster
    Relationship: canonical
    Canonical: #305
    Summary: This issue remains the dedicated startup investigation; diagnostics and narrow repairs cover related boundaries without establishing the original cause.

    Members:

    Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

    Next step
    Retain the member-led investigation and existing diagnostic work; a causal trigger and bounded repair are not yet established for autonomous implementation.

    Review details

    Best possible solution:

    Establish the stalled phase through attributable cold-flow receipts, then repair that owner boundary with a failing regression while preserving deadlines and explicit cleanup uncertainty.

    Do we have a high-confidence way to reproduce the issue?

    No high-confidence deterministic current-main reproduction is established. Real Windows cold-flow failures are recorded, but their intermittent trigger remains unknown; no target code was executed during this read-only review.

    Is this the best way to solve the issue?

    Yes: evidence-led investigation at the existing command owner is the narrowest supported path. Larger budgets, prewarming, retries, or treating successful controls as a repair would conceal the unresolved failure.

    AGENTS.md: found and applied where relevant.

    Remaining risk / open question:

    • The natural startup delay and the original failed helper's cleanup fate remain unattributed; controlled cancellation tests and later successful runs do not settle either.

    Codex review notes: model internal, reasoning medium; reviewed against 4ff73eb1397d.

    Label changes

    Label changes:

    • add clawsweeper:needs-info: Current issue advisory state selects this label.

    Label justifications:

    • P2: Intermittent Windows validation failures warrant investigation, but no urgent user-facing core outage is established.
    • impact:other: The concrete impact is failed or stalled Crabpot validation and uncertain command cleanup.
    Evidence reviewed

    What I checked:

    • Current checkout and repository policy: The origin identifies openclaw/crabpot; HEAD matches the supplied default-branch SHA and the initial status was clean. The complete root AGENTS.md was read and its cheap, credential-free validation boundary applied. (AGENTS.md:1, 4ff73eb1397d)
    • Owned startup and uncertainty reporting remain active: The supervisor retains its 10-second startup and 2-second cleanup limits. Missing terminal cleanup receipts produce EOWNERCLEANUP; startup expiry produces EOWNERSTART, and cleanup uncertainty is preserved in the error. Startup traces add observations rather than a recovery mechanism. (scripts/owned-command.mjs:12, 4ff73eb1397d)
    • The reported caller is owned by Crabpot: Cold Inspector preparation still executes git init through the shared command owner before fetch or npm installation. Unconfirmed cleanup retains checkout-lock ownership instead of publishing readiness. (scripts/plugin-inspector-source.mjs:96, 4ff73eb1397d)
    • Narrow parser repair and regression boundary: The Windows helper now installs its cancellation watcher before JSON parsing. Tests inject parser and compiler stalls, require helper closure before rescue, and retain a historical-order negative control. These controlled cases do not reproduce the natural cold-start delay. The supplied discussion explicitly preserves that distinction for fix: cancel Windows helpers stalled during request parsing #398 and the separate environment repair at fix: preserve the Windows command controller environment #374. (test/owned-command.test.mjs:437, 4ff73eb1397d)
    • Latest merged diagnostics do not claim a startup fix: GitHub verifies fix(ci): report command startup phases on failure #400 merged on September 28. Its body says controlled native failures qualify the diagnostics, while the previous sporadic failure did not recur and its cause was neither identified nor repaired. Current README likewise states that phase markers never prove Job extinction. (README.md:192, 8b984c058387)
    • Later related evidence preserves a startup recurrence: The live body of fix: retire lifecycle capture resources before Windows cleanup #405 records a first Inspector git-init failure with EOWNERSTART and unconfirmed cleanup: pipe connection at +9,534 ms and parser entry at +9,880 ms, without parser completion or READY. It explicitly says the subsequent unchanged-head pass did not repair the startup cause. Its lifecycle teardown work is a separate boundary, not a canonical replacement for this investigation.

    Likely related people:

    • vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
    • SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
    • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
  3. vincentkoc commented on Sep 10, 2026

    @vincentkoc
    MemberAuthor

    One diagnostic round completed for #306; no startup fix or root cause is established.

    Head 0f59a816fd5903e40508106848ebe56da3ec9640, based on b61d2d9c8750010a0a561bd3e2165eb170689ec5. Both attempt-1 workflows tested synthetic merge d40e2b43cbfc6f63db28c4e355cb3167460fedd7, tree 5831e3cbe93064217b58c2756c77f3d7b4037d25, identical to the diagnostic head.

    • Required Windows failed: https://github.com/openclaw/crabpot/actions/runs/34476116811/job/102867164401. The log flags behavior eval shell command timeout reports and terminates the command group, followed by the unchanged 600,000ms suite watchdog. The exact assertion and command child fate are unavailable. No cold-Inspector trace or EOWNERSTART was captured. This distinct observed surface is not a root-cause or preexistence claim. The later generated summary says pass, but does not override the failed suite.
    • Advisory Windows passed: https://github.com/openclaw/crabpot/actions/runs/34476116944/job/102867425172. 398 tests: 390 passed, 0 failed, 8 skipped; all 19 steps completed. The actual cold-contract-test trace records native creation, parent READY at +4,148ms, terminal receipt at +4,243ms, status 0 and no cleanup error. Separate native/bootstrap/cancellation/compiler controls passed.

    Required used Node 24.19.0, Windows image windows-2025-vs2026 20260824.214.3, host d9b8996498001e04f706255eb0ae818349fd594c. Advisory used Node 24.20.0, image 20260907.229.1, host 4aacaa0ee9dd387a23e2bad4747c1377d3a7457f. Both used Runner 2.337.0 and Git 2.55.0.windows.5. These differences are recorded, not treated as causes.

    Preserved producer receipts, under crabpot-startup-305/attempt-1/:

    • Required artifact, UUID eea08263-e59f-4d08-83ea-45c704952323: only later-report; not evidence of the missing cold attempt.
    • Advisory artifact, UUID 17075577-7ad0-4889-9007-bbddd08996d0: cold-contract-test.

    The healthy trace does not explain the original failure. Synchronous diagnostic I/O and added compilation have observer effects; diagnostic records are not cleanup authority. No repeat run, repair, merge or closure followed. The diagnostic draft is paused and this issue remains open for evidence-led disposition.

    The next scoped investigation is the earlier Windows behavior-eval timeout failure, capturing its full error and child/pipe closure evidence; it is separate from this diagnostic draft.

  4. vincentkoc commented on Sep 27, 2026

    @vincentkoc
    MemberAuthor

    Follow-up audit on September 27, 2026. This issue remains open; no cause for the original intermittent failure is established.

    The narrow controller-environment repair landed in #374 at 6e78c4df41dcecf8d198766451add5b68f39c582. It keeps the PowerShell controller's parent environment separate from the native target's caller-supplied environment. It does not change startup or cleanup budgets.

    The exact reviewed head, 1479da83d273651b4780f11b20d449c60659b894, passed required cross-platform CI: https://github.com/openclaw/crabpot/actions/runs/36204044910. The previously pending advisory Windows job has now finished and its log was inspected: https://github.com/openclaw/crabpot/actions/runs/36204044911/job/108296895398.

    • Advisory Windows used Node 24.21.0: 734 tests, 706 passed, zero failed, zero cancelled, 28 skipped.
    • The new controller/target environment regression ran and passed. The blocked-bootstrap, cancellation, and actual-compiler positive and negative controls also passed.
    • The advisory job's successful conclusion does not mean every report succeeded. Its best-effort import-loop report failed: three baseline and three plugin samples exited 1, with zero captured contracts or OpenClaw lifecycle samples. The inspected baseline/plugin error previews retain only the final two stack frames in run-openclaw-lifecycle-capture.mjs, not the originating exception. This is a separate unresolved report failure, not evidence of an EOWNERSTART recurrence or a timing regression.
    • The downloaded advisory artifact matched GitHub's SHA-256: 9e3995d24b296415d18296212f6b4e3a922b7f0b5f546daaed78c255a0db31fb. Artifact: https://github.com/openclaw/crabpot/actions/runs/36204044911/artifacts/10893826280.

    The next bounded owner-level proof is the parser cancellation boundary. At current main b95e26f0ac2fb893e920668066890a802ce8a0f6, scripts/owned-command-windows.ps1 parses the request before installing the owner-loss watcher. The tested blocked-bootstrap controls stall later, at compilation, after that watcher exists. A native fixture must test owner loss while the actual helper is stalled at parser entry, with helper closure before rescue and no native admission, before claiming a repair to that boundary. This source observation is not a reproduction or an attribution of the original failure.

    No new native fault-injection run, startup repair, timeout increase, prewarming, unchanged retry, or issue closure is claimed by this audit.

    September 28, 2026: Parser Cancellation Proof Landed

    The bounded parser-cancellation work above is complete in #398, squash-merged as ef55f955ce957406574e0462879d5da226ccac11. The existing watcher is now armed before request parsing. No timeout, request protocol, environment boundary, or runner changes were made.

    Exact reviewed head 273fe61d0b10c2dd92568564addad3cd3a4c3a51 passed required cross-platform CI: https://github.com/openclaw/crabpot/actions/runs/36365389409. The resulting main tree was verified identical to that tested tree.

    The completed native Windows log was inspected: https://github.com/openclaw/crabpot/actions/runs/36365389409/job/108750622184. It reports 764 tests: 736 passed, zero failed, zero cancelled, 28 skipped.

    • Both parser-stall positives passed: startup expiry and owner loss closed the actual helper before rescue or independent fixture expiry, with no native command admission.
    • The historical parser-before-watcher negative control left the helper alive after owner loss. The same closure assertion failed as intended; cooperative rescue then closed the helper before independent expiry, with its rescue receipt verified.
    • Existing compilation stalls, bootstrap cancellation, real-compiler positive/negative controls, and native Job cleanup also passed.

    This is controlled parser-entry stall evidence, not reproduction of a naturally stalled parser or attribution of the original intermittent cold-start failure. The issue and the separately recorded profiling investigation remain open; neither is claimed solved by this narrow repair.

  5. added
    clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.
    on Oct 7, 2026
  6. steipete commented on Oct 15, 2026

    @steipete
    Contributor

    Landed the separately reviewed owner repair in #502, commit 9bb9880. The observed main failure reached per-command C# compilation 8.154 seconds into the 10-second startup budget. The new path removes that compilation and cmdlet autoload, gates the request on PREPARED, and releases custody only after explicit non-admission or verified Job/helper/control-output cleanup.

    Exact-head required proof: https://github.com/openclaw/crabpot/actions/runs/38781124369 — Windows 776 passed / 28 skipped / 0 failed, including real suspended-csc positive and negative controls, cancellation/receipt checks, and interrupted-checkout retry. Identical owner/test sources also passed both Windows lanes in #503: https://github.com/openclaw/crabpot/actions/runs/38781299979 and https://github.com/openclaw/crabpot/actions/runs/38781299610.

    Keeping this issue open: #502's advisory Windows run still observed a preconnection EOWNERSTART (without cleanup uncertainty), a negative-control rescue timeout, and observer setup exceeding its bounded wait. Those residual failures are not explained by a passing run or claimed fixed. No startup/cleanup budgets were increased and no prewarming was added.

  7. steipete commented on Oct 15, 2026

    @steipete
    Contributor

    Final merged main 3f65dbd3d1c2b141f8031d1b471c99dbf791240f passed its required pinned gate across Linux, macOS, Windows, container, and all three changed-fixture lanes: https://github.com/openclaw/crabpot/actions/runs/38795785726. Required Windows recorded 777 passed, 28 platform skips, zero failures, including all four real-compiler controls and interrupted-checkout recovery.

    Keeping this issue open: the separate moving-HEAD Windows canary failed one instrumentation readiness wait, before that test's measured compiler target started: https://github.com/openclaw/crabpot/actions/runs/38795785494/job/116632387837. Its observer was admitted but did not produce observer-ready within the bounded setup wait. That observation does not establish a production-helper failure or explain the earlier preconnection/rescue failures. Production budgets and the outer test deadline remain unchanged.

  8. steipete commented on Oct 19, 2026

    @steipete
    Contributor

    PR #539 follow-up: the current Inspector heap exhaustion is inherited and cross-platform. This is distinct from this issue's original Windows command-supervisor startup failure; no common cause is established.

    Compared the latest PR canary (head a880a7f159b0573a27b46c66b0af76ef8ad368d1, run 39185192954) with main at 03d5bda30bdef5c2e2c7adb3ecfe1afedadea6f3 (run 39210866204). All six jobs fail in static step 4/23, node --test --test-concurrency=1 test/corpus/host.test.mjs, with V8 heap exhaustion near the enforced 2 GiB limit:

    Platform PR Unchanged main
    macOS xcode-27 macos-15
    Windows windows-latest windows-latest
    Linux ubuntu-latest ubuntu-latest

    The moving upstream inputs differ: PR used OpenClaw c5111f40, main used 80a48efb6f6b203a5898d0ec7d07c0c998602d9f. This establishes the same failure class on unmodified main, including its previous macOS runner; it is not a controlled attribution to an upstream commit.

    The active rules require only Default Track (pinned OpenClaw). That check and all its PR matrix/fixture prerequisites passed at the exact PR head; the three HEAD Canary (advisory, …) jobs are not required. No branch protection or ruleset was changed.

    A bounded diagnostic on a fresh AWS c7a.8xlarge, Node 24.19.0, source Inspector 8f458e7be92258ac1729451a0a4bcaea2ecdb68e, used the exact main-canary OpenClaw input with an 8 GiB diagnostic ceiling, without installing the host lifecycle dependencies. Sampling process.memoryUsage() every 200 ms recorded 4,899,506,912 bytes heap used (4.56 GiB) and 5,464,875,008 bytes RSS (5.09 GiB). The first compatibility assertion completed after 268 seconds; five of six assertions passed. The sixth was the workspace-plan test's file:…openclaw path assertion: this diagnostic checkout was named canary-host, so that result is not evidence of a CI regression. This was a memory measurement, not a green canary or full-suite qualification.

    The source inspection reaches src/sdk-usage/program.js:createUsageProgram, which constructs a TypeScript program over the plugin inputs and reachable explicit-host source. Existing host-syntax caching already uses WeakRef; an assumed cache leak is not established. The observed peak rules out a 4 GiB fix. A larger budget would need headroom plus full-suite qualification on each OS; this run does not establish that safe cross-platform budget. No heap limit, timeout, report policy, required check, or Inspector pin is changed in #539. The runner migration proceeds using the explicitly authorized inherited-advisory-failure exception; bounding Inspector's retained graph remains follow-up work.

    The integrated candidate passed independent Codex review through P2 and 51 focused workflow/policy/synthetic-probe tests on the same isolated lease. All required exact-head PR checks were green before landing. Thanks @vincentkoc; his contributor credit is retained.

    Merged #539 as de5d707b4a2c, at exact reviewed head a880a7f159b0573a27b46c66b0af76ef8ad368d1. The final check read showed 36 successes, 3 skipped jobs, and only the 3 inherited advisory failures. The resulting merge tree exactly matched the reviewed integrated candidate.

  9. steipete commented on Oct 19, 2026

    @steipete
    Contributor

    Round 13 narrowed the separate Inspector heap failure without changing the startup investigation or its ownership.

    A fresh Linux run of unchanged Crabpot 368a54f398e4ebb07f86b29d34c623d5456d050c, Node 24.19.0 and source Inspector 8f458e7be92258ac1729451a0a4bcaea2ecdb68e, against frozen OpenClaw 43e2d3ce2e9bafc5172b65a3d7c0b0a77fb78b6e, reproduced heap exhaustion with:

    CRABPOT_TEST_OPENCLAW_PATH=./openclaw node --max-old-space-size=2048 --test --test-concurrency=1 test/corpus/host.test.mjs

    Temporary fixture-boundary diagnostics showed collection between earlier fixtures and failure during Feishu inspection. A fresh process inspecting only Feishu through the public inspectCompatibilityFixtureSetConfig API also exhausted the same 2 GiB heap in about 38 seconds. This rules out splitting the corpus into one process per fixture as a sufficient fix for this input.

    A separate, temporary subphase diagnostic recorded 142,686,992 bytes of heap after SDK input discovery and 2,073,207,120 bytes immediately after createUsageProgram; the process exhausted the heap before reporting completion of provenance construction. Program construction therefore consumes nearly the entire budget, but this does not identify one exact allocating function as the root cause. Both instrumented source files were restored after their runs. No coverage filters, heap increases, retries-until-green, or production source changes were used.

    The current development dashboard also fails with explicit V8 heap exhaustion: https://github.com/openclaw/crabpot/actions/runs/39296241436/job/118209401892 . The earlier merged required Default Track has now completed successfully: https://github.com/openclaw/crabpot/actions/runs/39215227316 . These are separate verdicts.

    The remaining memory repair belongs to Inspector's host-aware analysis and must preserve bundled-file coverage and cross-file provenance. This evidence does not explain or fix the original Windows EOWNERSTART failure, runner communication losses, or the profiler descendant-cleanup question. Keeping this issue and the owner-led diagnostics hold open.

  10. steipete commented on Oct 19, 2026

    @steipete
    Contributor

    Fresh required-main failure after #541: Windows job 118223796523, commit dd797d0a2106834a692feaaf361006e5de48864b, Node 24.21.0.

    test/ci-policy.test.mjs failed during its top-level Inspector checkout preparation with EOWNERSTART: command supervisor startup timed out. The retained diagnostics now narrow the phase:

    owner-started: 1792433685750
    worker-entered: 1792433685801
    helper-spawned: 1792433685810
    windowsHelper.captured: true
    windowsHelper.truncated: false
    windowsHelper.terminal: { exitCode: null, signal: 'SIGTERM', requestIssued: false }
    

    No later startup phase was recorded, and no command request was issued. This is a pre-request helper startup failure, distinct from the development-host Inspector memory failure reported above. The diagnostics are useful, but do not yet establish why the helper failed to connect before the deadline. No rerun, timeout relaxation, or cleanup-policy change was used. Keeping this issue open for the existing Windows investigation.

  11. steipete commented on Oct 21, 2026

    @steipete
    Contributor

    Native Windows investigation update: the startup cause remains unresolved. No production repair or timeout change has been made.

    The two recent failure modes are distinct. Required-main run 39300406644 failed the first Inspector git init: Worker entered after 51 ms, helper spawned after 60 ms, then no bootstrap record or control-pipe connection was observed before the 10-second owner deadline. The helper closed under cancellation with requestIssued: false. This does not prove that the PowerShell script never entered, because bootstrap stderr writes can fail. PR #568's Windows run later recorded the same observation boundary in the first profiler contract-capture sample, at Inspector git rev-parse.

    By contrast, both attempts of main run 39353237234 passed their tests and exhausted the 225-minute static-suite budget. The last green Windows suite took about 227 minutes. That supports the separate headroom change in #569; it does not explain the startup stall.

    The native controls in draft #571 have now completed 2,560 owned-helper launches and 96 direct PowerShell launches without this failure, across Node 24.19.0/24.21.0 and runner images 20261007.264.1/20261018.275.1. These are non-reproductions, not evidence that a Node version or image fixes the issue.

    A more representative cold-import comparison also passed on both the last-green commit de5d707b4a2cf864522c7689017c170bfc577fc1 and current main 4c40f7f5e154b6fafb410321e8d1ace16eba49f7, using Node 24.21.0 and image 20261007.264.1. It preserved the subject checkout at the workspace root and ran the original contract check, fixture materialization and complete ordinary test stage through the existing static-step launcher. Current main recorded 814 tests (786 pass, 28 skip); the last-green commit recorded 807 (779 pass, 28 skip), with zero failures. Both native monitors completed normally.

    For helpers matched to the original owner path, observed READY latency was median/max 378/2,362 ms on current main and 540/759 ms on the last-green commit. Intentional copied-owner timeout fixtures were kept separate. Neither run triggered a natural pre-connection stack capture. The instrumentation adds file markers and pipe observations, so its observer effects remain a limitation.

    The next native diagnostic preserves the profiler's real report/inspection setup and original three-sample command sequence through contract-capture. It is still running. Inspection of the cold artifacts also found PID reuse in the diagnostic monitor; #571 now keys stack captures and progress markers by launch time as well as PID. The already-running profiler diagnostic predates that observer correction, so a missing stack there cannot rule out a stall after PID reuse.

    The pre-request startup investigation stays open. No retries, timeout increases, prewarming workaround, or suppression are being presented as a repair.

  12. steipete commented on Oct 21, 2026

    @steipete
    Contributor

    The retained-profiler diagnostic completed successfully. All twelve samples (three each of Node boot, fixture inspection, compatibility reporting, and contract capture) exited zero. It used frozen main 4c40f7f5e154b6fafb410321e8d1ace16eba49f7, Node 24.21.0, image 20261007.264.1.

    The first inspection sample recorded a -1,084 MiB harness heap delta, reproducing the large allocation/collection history that isolated helper controls omitted. Native counters reached 3,836 MiB of aggregate private process memory. Matched original-owner READY latency was median/max 578/2,561 ms. The monitor completed normally (17,407 snapshots; maximum sample gap 1,347 ms). No natural startup failure was reproduced. No stack capture occurred; as noted above, this run predates the PID-marker correction, so absence of a stack alone is not a negative finding.

    The remaining context difference is the preceding complete static pipeline. Run 39564565856, diagnostic head e84f50ba710e129bf2af4a01afaf738eada161a5 in #571, now observes the original 23-step pipeline against the same frozen main, including report writes, track inputs, and three-sample profiling. The monitor includes the PID-reuse correction. Only the outer diagnostic time limit provides extra evidence-collection time; command startup/execution limits and admission/cleanup protocol remain unchanged.

    This is still an investigation. No production startup repair, main merge, or release has been made. #569 remains separate and held pending the startup result. The earlier superseded uninstrumented draft Check run was cancelled after the complete instrumented Windows job started; its successful advisory run is not being presented as repair proof.

  13. steipete commented on Oct 21, 2026

    @steipete
    Contributor

    Correction to the earlier cold-run latency summary: the scratch analysis required helper-spawn and initial pipe-snapshot timestamps to match exactly, which dropped observations written one millisecond later. It omitted last-green’s first cold launch. Correlation now uses parent PID plus one unambiguous initial snapshot within 10 ms, consumed once per launch. All inspected commands with a helper-spawn phase correlate.

    Complete owner-start → READY median/max: last-green 541/2,638 ms, current main 378/2,362 ms, retained-profiler prefix 578/2,561 ms. Test pass/fail results are unchanged. The data does not establish a startup-latency regression.

    The first four of eight independent fresh Windows trials passed (original-owner maxima 2.16–2.80 s); the remaining trials and the replacement complete-pipeline run continue. The earlier full run 39564565856 was intentionally cancelled because PowerShell-based debugger verification could prewarm startup components. The replacement verifies pinned debugger hashes in Python. No natural 10-second startup stall or supported root-cause repair has been captured.

  14. steipete commented on Oct 21, 2026

    @steipete
    Contributor

    The eight independent fresh-runner trials in 39572677987 all passed, but trial 6 captured a much slower first helper. Owner → READY was 9,359 ms; helper-spawn → script-entry was 8,218 ms, and helper-spawn → pipe-connect was 8,495 ms. This is closer to the historical 10-second boundary than the earlier warm controls.

    The first six seconds are useful evidence: before debugger launch, the helper was progressing (328 ms accumulated CPU, 172 KB read-transfer count, 496 handles at +6,109 ms) while no script-entry marker or stderr bytes had appeared. Native and pipe observations continued roughly every 253 ms. That supports slow pre-entry initialization; it does not identify a specific loader, CLR, Defender, or Node defect.

    The full 9.36-second measurement has an observer limitation. The non-suspending debugger ran from +6,109 to +8,135 ms, and the script-entry marker appeared 83 ms after it exited. Its dot-command parser treated the semicolon-separated stack commands as part of the symbol path, so no full stacks were collected. The single automatically displayed instruction is insufficient to attribute the wait. The diagnostic branch is being corrected to use a command file, with a post-workload synthetic check that stack commands execute and detach while leaving the target alive.

    The unmodified-source cold control 39575390604 and full-context run 39565264952 remain in progress. No production fix or timeout mitigation is claimed, and #569 remains held.

  15. steipete commented on Oct 21, 2026

    @steipete
    Contributor

    The complete native Windows experiment 39582684761 passed all 23 static stages against frozen main 4c40f7f5e154b6fafb410321e8d1ace16eba49f7, including the original nine-command, three-sample profiler. Node 24.21.0, runner image 20261018.275.1. The retained artifact contains the native timeline and command observations. The target exited zero, native monitoring completed, and the post-workload debugger verification collected native stacks and detached with its target alive.

    There were 293 recorded original-owner commands; the error outcomes were the expected test controls. Owner-start → READY median/max was 561/2,502 ms. The monitor retained 52,054 samples with a maximum 366 ms gap and observed 4,216 MiB aggregate private memory. No natural slow-start stack capture was triggered.

    The unmodified-source control also passed: 786 tests, 28 skipped, both owner/helper hashes unchanged. The second eight-run cold cohort passed 8/8 with validated debugger command execution; readiness maxima ranged from 2,178 to 2,780 ms. Combined with the first cohort, that is sixteen independent fresh-runner attempts, separate from the thousands of warm helper controls.

    Correction about the earlier full run 39565264952: its live browser log stopped updating after workspace-plan, but the final raw log proves that stage completed normally at 09:02 UTC and the run advanced into profiling. I manually cancelled a healthy run based on that stale view. Its cancellation supplies no hang or startup-failure evidence; the complete replacement above resolves that missing experiment.

    Root cause remains unresolved. The historical receipts show missing observed bootstrap/pipe progress before the 10-second deadline and no issued target request. Trial 6's pre-debugger delay supports slow, progressing PowerShell initialization as a hypothesis, but it does not identify the blocking operation or establish a runner, Node, CLR, or Defender cause. Independent assessment of the completed evidence found no supported production fix or workaround. Retries, prewarming, parser/Job changes, and a larger startup deadline would be speculative.

    No production repair, merge, or release was made. #571 remains a diagnostic draft. #569 still addresses a separately evidenced suite-budget problem, but stays behind the requested root-fix prerequisite. Main's latest Check remains failed at 4c40f7f (225-minute suite budget); this diagnostic success is not merged-main CI proof. The incident stays open, and the merged-main Windows success criterion has not been met.

  16. steipete commented on Oct 21, 2026

    @steipete
    Contributor

    @vincentkoc Peter asked to hand the Windows root-cause investigation to you. The deep dive completed 16 independent fresh-runner trials and a green full 23-stage Windows pipeline, including the original three-sample profiler (full pipeline, first cohort, second cohort). One trial captured slow, progressing PowerShell initialization: 9,359 ms owner-to-READY, with 8,218 ms before script entry. Debugger observer effects limit attribution; no root cause or production fix is established.

    The recent required Windows red began at dd797d0a / #541; the last green was de5d707b. Draft #571 retains the diagnostics and evidence. #569 remains available for the separately evidenced suite-budget headroom problem. Peter authorized a one-off waiver of the pre-existing Windows timeout for #570 only; #569 and #571 remain unmerged, and this issue stays open for your investigation.

  17. vincentkoc commented on Nov 4, 2026

    @vincentkoc
    MemberAuthor

    Additional native Windows evidence from the current maintenance campaign, with a distinct observed phase:

    • Candidate a6eb86cd8bf618e3b93a483d7b0b0bf1e4d2c5a3, host 7c2ff8f775acca3e5c93c9c22b6247e37df29bae.
    • Advisory job: https://github.com/openclaw/crabpot/actions/runs/41864389013/job/126695586448.
    • Windows actual compiler lifecycle fails in its assembly-preparation control after the unchanged 35000 ms command deadline. The assertion is assembly preparation must return confirmed native closure.
    • Its retained diagnostic reports observer-compile-started and observer-compile-completed both absent, with ETIMEDOUT. Earlier startup phases include pipe-connected, preparation-started, prepared-received, and request-write-attempted.

    This does not establish the same cause as the original cold-start failure, identify the blocked native phase after the request, or establish confirmed closure. No timeout increase, owner modification, retry, or waiver was performed. The asynchronous-loader candidate does not change the process owner or this test.

    Separately, #725 records an EPERM from lifecycle state-directory cleanup in an older canary; it is a different observed error and is not treated as a supervisor diagnosis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal priority bug or improvement with limited blast radius.clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions