Repository navigation
Description
Problem
Unattributed intermittent Windows cold-start failure in the existing process owner, observed while validating #304. It is not established as preexisting or unrelated to that candidate.
The first Inspector checkout git init, before fetch or npm installation, failed during test/capture-contracts.test.mjs module import:
EOWNERSTART: command supervisor startup timed out; command cleanup was not confirmed
The caller is ensurePinnedInspectorCheckout() in scripts/plugin-inspector-source.mjs; command startup and cleanup are owned by scripts/owned-command.mjs.
Evidence
- Original red, head
9b19ec198df5d9ff49e7910ec7c166545e753dc7: https://github.com/openclaw/crabpot/actions/runs/34452925787/job/102792576498. Windows Node 24.20.0 recorded 398 tests: 389 passed, one failed, eight skipped. The failure happened under the real cold static-suite invocation,node --test --test-concurrency=1 test/*.test.mjs. Other required platforms passed. Later best-effort report success does not override the failed suite. - One approved failure-only caller-diagnostic run, head
2515b0569e32e57a63af0550a116ac137f821916: https://github.com/openclaw/crabpot/actions/runs/34455244708/job/102799999787. All four required platforms and the aggregate passed. Windows Node 24.19.0 recorded 398 tests: 390 passed, zero failed, eight skipped; all 19 static steps completed. - The automatic diagnostic-head advisory also passed: https://github.com/openclaw/crabpot/actions/runs/34455244608/job/102800432709. Windows had the same counts and Node version.
- Neither diagnostic-head Windows log emitted the failure-only JSON record. Successful cold contract import and native/bootstrap/cancellation/compiler/Job controls were observed, but those separate controls do not establish the original failed helper's fate.
The temporary diagnostic changed only the existing caller's error branch. It did not change the owner, PowerShell/native protocol, startup or cleanup budgets, cold preparation, test order, dependencies, or workflows. A passing instrumented run is not a reproduction, root-cause attribution, or fix. The Node version difference is evidence, not an explanation.
Earlier startup observations are retained in #292; no common cause is asserted. Searches for EOWNERSTART, cold-start, and Windows startup issues found no dedicated follow-up before creating this issue.
Diagnostic-Free Candidate Proof
The single authorized final run passed after removing only the temporary diagnostic. Head e7694efd9f595c3c514eab35c542a303b0081db0 has exactly the original 9b19ec1 tree, a0559d10c9d14ffff33ba0c8d4170c7c99908aed; no process-owner repair, timeout increase, prewarming, or test-order change was made.
- Required Windows, Node 24.20.0, fixed host
d9b8996498001e04f706255eb0ae818349fd594c: https://github.com/openclaw/crabpot/actions/runs/34457793050/job/102808148810. - Advisory Windows, Node 24.19.0, host
327f0d5afe09805d4e3e89ad0ff654dd8cbed6f1: https://github.com/openclaw/crabpot/actions/runs/34457793051/job/102808404114.
Both logs show successful actual cold contract import, 398 tests (390 passed, zero failed, eight skipped), all 19 static steps, and the existing native/bootstrap/cancellation/compiler/Job controls. Both workflows completed on attempt 1.
The required result supplies direct clean-candidate Windows coverage on 24.20.0. It does not explain the original failure, establish a Node-version cause, prove a repair, or close this follow-up. Earlier 24.19.0 diagnostic passes and other PRs' 24.20.0 passes are not treated as controlled reproductions or attribution.
Follow-Up Boundary
Investigate the startup phase and cleanup uncertainty in the process owner using attributable evidence from the actual cold CI flow. Distinguish Worker startup, helper connection/request handling, READY, and terminal cleanup only where receipts support the distinction.
Do not infer extinction from separate passing controls or a successful subsequent run. Do not use retries-until-green, larger budgets, prewarming, or suppression as a fix. The agreed final-run stop condition was no landing on recurrence and a separately reviewed owner repair; no recurrence was observed in that single run.
No private raw artifacts are attached. Public run links above retain the original failure and instrumented results. The follow-up records unresolved risk; it does not claim a cause or repair.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
clawsweeper commentedon Sep 10, 2026
Codex review: this still needs some work. Reviewed October 7, 2026, 1:29 PM ET / 17:29 UTC.
Summary
Keep open: current main improves cancellation and diagnostics but does not establish a repair for the intermittent cold-start failure. Later evidence records another startup failure, and this member-authored investigation remains necessary.
Reproducibility: no. high-confidence deterministic current-main reproduction is established. Real Windows cold-flow failures are recorded, but their intermittent trigger remains unknown; no target code was executed during this read-only review.
Root-cause cluster
Relationship:
canonicalCanonical: #305
Summary: This issue remains the dedicated startup investigation; diagnostics and narrow repairs cover related boundaries without establishing the original cause.
Members:
partial_overlap: chore: add temporary Windows startup diagnostics #306 - Open diagnostic draft observes cold startup without claiming a repair.partial_overlap: chore(ci): trace Windows cold command startup #363 - Open diagnostic draft records startup boundaries and a candidate later extracted into a narrow repair.partial_overlap: fix: preserve the Windows command controller environment #374 - Merged controller-environment repair explicitly leaves the intermittent failure unresolved.partial_overlap: fix: cancel Windows helpers stalled during request parsing #398 - Merged parser-cancellation repair proves a controlled boundary rather than the natural cold-start cause.partial_overlap: fix(ci): report command startup phases on failure #400 - Merged phase diagnostics improve evidence collection without repairing the intermittent failure.Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.
Next step
Retain the member-led investigation and existing diagnostic work; a causal trigger and bounded repair are not yet established for autonomous implementation.
Review details
Best possible solution:
Establish the stalled phase through attributable cold-flow receipts, then repair that owner boundary with a failing regression while preserving deadlines and explicit cleanup uncertainty.
Do we have a high-confidence way to reproduce the issue?
No high-confidence deterministic current-main reproduction is established. Real Windows cold-flow failures are recorded, but their intermittent trigger remains unknown; no target code was executed during this read-only review.
Is this the best way to solve the issue?
Yes: evidence-led investigation at the existing command owner is the narrowest supported path. Larger budgets, prewarming, retries, or treating successful controls as a repair would conceal the unresolved failure.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against 4ff73eb1397d.
Label changes
Label changes:
clawsweeper:needs-info: Current issue advisory state selects this label.Label justifications:
P2: Intermittent Windows validation failures warrant investigation, but no urgent user-facing core outage is established.impact:other: The concrete impact is failed or stalled Crabpot validation and uncertain command cleanup.Evidence reviewed
What I checked:
AGENTS.md:1, 4ff73eb1397d)scripts/owned-command.mjs:12, 4ff73eb1397d)scripts/plugin-inspector-source.mjs:96, 4ff73eb1397d)test/owned-command.test.mjs:437, 4ff73eb1397d)README.md:192, 8b984c058387)Likely related people:
How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper autofix,@clawsweeper automerge,@clawsweeper fix ci, or@clawsweeper address review.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.vincentkoc commentedon Sep 10, 2026
One diagnostic round completed for #306; no startup fix or root cause is established.
Head
0f59a816fd5903e40508106848ebe56da3ec9640, based onb61d2d9c8750010a0a561bd3e2165eb170689ec5. Both attempt-1 workflows tested synthetic merged40e2b43cbfc6f63db28c4e355cb3167460fedd7, tree5831e3cbe93064217b58c2756c77f3d7b4037d25, identical to the diagnostic head.behavior eval shell command timeout reports and terminates the command group, followed by the unchanged 600,000ms suite watchdog. The exact assertion and command child fate are unavailable. No cold-Inspector trace orEOWNERSTARTwas captured. This distinct observed surface is not a root-cause or preexistence claim. The later generated summary sayspass, but does not override the failed suite.cold-contract-testtrace records native creation, parent READY at +4,148ms, terminal receipt at +4,243ms, status 0 and no cleanup error. Separate native/bootstrap/cancellation/compiler controls passed.Required used Node 24.19.0, Windows image
windows-2025-vs202620260824.214.3, hostd9b8996498001e04f706255eb0ae818349fd594c. Advisory used Node 24.20.0, image20260907.229.1, host4aacaa0ee9dd387a23e2bad4747c1377d3a7457f. Both used Runner 2.337.0 and Git 2.55.0.windows.5. These differences are recorded, not treated as causes.Preserved producer receipts, under
crabpot-startup-305/attempt-1/:eea08263-e59f-4d08-83ea-45c704952323: onlylater-report; not evidence of the missing cold attempt.17075577-7ad0-4889-9007-bbddd08996d0:cold-contract-test.The healthy trace does not explain the original failure. Synchronous diagnostic I/O and added compilation have observer effects; diagnostic records are not cleanup authority. No repeat run, repair, merge or closure followed. The diagnostic draft is paused and this issue remains open for evidence-led disposition.
The next scoped investigation is the earlier Windows behavior-eval timeout failure, capturing its full error and child/pipe closure evidence; it is separate from this diagnostic draft.
vincentkoc commentedon Sep 27, 2026
Follow-up audit on September 27, 2026. This issue remains open; no cause for the original intermittent failure is established.
The narrow controller-environment repair landed in #374 at
6e78c4df41dcecf8d198766451add5b68f39c582. It keeps the PowerShell controller's parent environment separate from the native target's caller-supplied environment. It does not change startup or cleanup budgets.The exact reviewed head,
1479da83d273651b4780f11b20d449c60659b894, passed required cross-platform CI: https://github.com/openclaw/crabpot/actions/runs/36204044910. The previously pending advisory Windows job has now finished and its log was inspected: https://github.com/openclaw/crabpot/actions/runs/36204044911/job/108296895398.run-openclaw-lifecycle-capture.mjs, not the originating exception. This is a separate unresolved report failure, not evidence of anEOWNERSTARTrecurrence or a timing regression.9e3995d24b296415d18296212f6b4e3a922b7f0b5f546daaed78c255a0db31fb. Artifact: https://github.com/openclaw/crabpot/actions/runs/36204044911/artifacts/10893826280.The next bounded owner-level proof is the parser cancellation boundary. At current main
b95e26f0ac2fb893e920668066890a802ce8a0f6,scripts/owned-command-windows.ps1parses the request before installing the owner-loss watcher. The tested blocked-bootstrap controls stall later, at compilation, after that watcher exists. A native fixture must test owner loss while the actual helper is stalled at parser entry, with helper closure before rescue and no native admission, before claiming a repair to that boundary. This source observation is not a reproduction or an attribution of the original failure.No new native fault-injection run, startup repair, timeout increase, prewarming, unchanged retry, or issue closure is claimed by this audit.
September 28, 2026: Parser Cancellation Proof Landed
The bounded parser-cancellation work above is complete in #398, squash-merged as
ef55f955ce957406574e0462879d5da226ccac11. The existing watcher is now armed before request parsing. No timeout, request protocol, environment boundary, or runner changes were made.Exact reviewed head
273fe61d0b10c2dd92568564addad3cd3a4c3a51passed required cross-platform CI: https://github.com/openclaw/crabpot/actions/runs/36365389409. The resulting main tree was verified identical to that tested tree.The completed native Windows log was inspected: https://github.com/openclaw/crabpot/actions/runs/36365389409/job/108750622184. It reports 764 tests: 736 passed, zero failed, zero cancelled, 28 skipped.
This is controlled parser-entry stall evidence, not reproduction of a naturally stalled parser or attribution of the original intermittent cold-start failure. The issue and the separately recorded profiling investigation remain open; neither is claimed solved by this narrow repair.
fix: cancel Windows helpers stalled during request parsing (#398)
fix: remove Windows command startup compilation
steipete commentedon Oct 15, 2026
Landed the separately reviewed owner repair in #502, commit 9bb9880. The observed main failure reached per-command C# compilation 8.154 seconds into the 10-second startup budget. The new path removes that compilation and cmdlet autoload, gates the request on PREPARED, and releases custody only after explicit non-admission or verified Job/helper/control-output cleanup.
Exact-head required proof: https://github.com/openclaw/crabpot/actions/runs/38781124369 — Windows 776 passed / 28 skipped / 0 failed, including real suspended-csc positive and negative controls, cancellation/receipt checks, and interrupted-checkout retry. Identical owner/test sources also passed both Windows lanes in #503: https://github.com/openclaw/crabpot/actions/runs/38781299979 and https://github.com/openclaw/crabpot/actions/runs/38781299610.
Keeping this issue open: #502's advisory Windows run still observed a preconnection EOWNERSTART (without cleanup uncertainty), a negative-control rescue timeout, and observer setup exceeding its bounded wait. Those residual failures are not explained by a passing run or claimed fixed. No startup/cleanup budgets were increased and no prewarming was added.
steipete commentedon Oct 15, 2026
Final merged main
3f65dbd3d1c2b141f8031d1b471c99dbf791240fpassed its required pinned gate across Linux, macOS, Windows, container, and all three changed-fixture lanes: https://github.com/openclaw/crabpot/actions/runs/38795785726. Required Windows recorded 777 passed, 28 platform skips, zero failures, including all four real-compiler controls and interrupted-checkout recovery.Keeping this issue open: the separate moving-HEAD Windows canary failed one instrumentation readiness wait, before that test's measured compiler target started: https://github.com/openclaw/crabpot/actions/runs/38795785494/job/116632387837. Its observer was admitted but did not produce observer-ready within the bounded setup wait. That observation does not establish a production-helper failure or explain the earlier preconnection/rescue failures. Production budgets and the outer test deadline remain unchanged.
build(deps): qualify Inspector and Codex fixture updates (#506)
steipete commentedon Oct 19, 2026
PR #539 follow-up: the current Inspector heap exhaustion is inherited and cross-platform. This is distinct from this issue's original Windows command-supervisor startup failure; no common cause is established.
Compared the latest PR canary (head
a880a7f159b0573a27b46c66b0af76ef8ad368d1, run 39185192954) with main at03d5bda30bdef5c2e2c7adb3ecfe1afedadea6f3(run 39210866204). All six jobs fail in static step 4/23,node --test --test-concurrency=1 test/corpus/host.test.mjs, with V8 heap exhaustion near the enforced 2 GiB limit:The moving upstream inputs differ: PR used OpenClaw
c5111f40, main used80a48efb6f6b203a5898d0ec7d07c0c998602d9f. This establishes the same failure class on unmodified main, including its previous macOS runner; it is not a controlled attribution to an upstream commit.The active rules require only
Default Track (pinned OpenClaw). That check and all its PR matrix/fixture prerequisites passed at the exact PR head; the threeHEAD Canary (advisory, …)jobs are not required. No branch protection or ruleset was changed.A bounded diagnostic on a fresh AWS
c7a.8xlarge, Node 24.19.0, source Inspector8f458e7be92258ac1729451a0a4bcaea2ecdb68e, used the exact main-canary OpenClaw input with an 8 GiB diagnostic ceiling, without installing the host lifecycle dependencies. Samplingprocess.memoryUsage()every 200 ms recorded 4,899,506,912 bytes heap used (4.56 GiB) and 5,464,875,008 bytes RSS (5.09 GiB). The first compatibility assertion completed after 268 seconds; five of six assertions passed. The sixth was the workspace-plan test'sfile:…openclawpath assertion: this diagnostic checkout was namedcanary-host, so that result is not evidence of a CI regression. This was a memory measurement, not a green canary or full-suite qualification.The source inspection reaches
src/sdk-usage/program.js:createUsageProgram, which constructs a TypeScript program over the plugin inputs and reachable explicit-host source. Existing host-syntax caching already usesWeakRef; an assumed cache leak is not established. The observed peak rules out a 4 GiB fix. A larger budget would need headroom plus full-suite qualification on each OS; this run does not establish that safe cross-platform budget. No heap limit, timeout, report policy, required check, or Inspector pin is changed in #539. The runner migration proceeds using the explicitly authorized inherited-advisory-failure exception; bounding Inspector's retained graph remains follow-up work.The integrated candidate passed independent Codex review through P2 and 51 focused workflow/policy/synthetic-probe tests on the same isolated lease. All required exact-head PR checks were green before landing. Thanks @vincentkoc; his contributor credit is retained.
Merged #539 as
de5d707b4a2c, at exact reviewed heada880a7f159b0573a27b46c66b0af76ef8ad368d1. The final check read showed 36 successes, 3 skipped jobs, and only the 3 inherited advisory failures. The resulting merge tree exactly matched the reviewed integrated candidate.build(deps): refresh fixtures and use Xcode 27 CI
steipete commentedon Oct 19, 2026
Round 13 narrowed the separate Inspector heap failure without changing the startup investigation or its ownership.
A fresh Linux run of unchanged Crabpot
368a54f398e4ebb07f86b29d34c623d5456d050c, Node 24.19.0 and source Inspector8f458e7be92258ac1729451a0a4bcaea2ecdb68e, against frozen OpenClaw43e2d3ce2e9bafc5172b65a3d7c0b0a77fb78b6e, reproduced heap exhaustion with:Temporary fixture-boundary diagnostics showed collection between earlier fixtures and failure during Feishu inspection. A fresh process inspecting only Feishu through the public
inspectCompatibilityFixtureSetConfigAPI also exhausted the same 2 GiB heap in about 38 seconds. This rules out splitting the corpus into one process per fixture as a sufficient fix for this input.A separate, temporary subphase diagnostic recorded 142,686,992 bytes of heap after SDK input discovery and 2,073,207,120 bytes immediately after
createUsageProgram; the process exhausted the heap before reporting completion of provenance construction. Program construction therefore consumes nearly the entire budget, but this does not identify one exact allocating function as the root cause. Both instrumented source files were restored after their runs. No coverage filters, heap increases, retries-until-green, or production source changes were used.The current development dashboard also fails with explicit V8 heap exhaustion: https://github.com/openclaw/crabpot/actions/runs/39296241436/job/118209401892 . The earlier merged required Default Track has now completed successfully: https://github.com/openclaw/crabpot/actions/runs/39215227316 . These are separate verdicts.
The remaining memory repair belongs to Inspector's host-aware analysis and must preserve bundled-file coverage and cross-file provenance. This evidence does not explain or fix the original Windows
EOWNERSTARTfailure, runner communication losses, or the profiler descendant-cleanup question. Keeping this issue and the owner-led diagnostics hold open.steipete commentedon Oct 19, 2026
Fresh required-main failure after #541: Windows job 118223796523, commit
dd797d0a2106834a692feaaf361006e5de48864b, Node 24.21.0.test/ci-policy.test.mjsfailed during its top-level Inspector checkout preparation withEOWNERSTART: command supervisor startup timed out. The retained diagnostics now narrow the phase:No later startup phase was recorded, and no command request was issued. This is a pre-request helper startup failure, distinct from the development-host Inspector memory failure reported above. The diagnostics are useful, but do not yet establish why the helper failed to connect before the deadline. No rerun, timeout relaxation, or cleanup-policy change was used. Keeping this issue open for the existing Windows investigation.
build(deps): qualify official plugin fallbacks and bound CI (#566)
steipete commentedon Oct 21, 2026
Native Windows investigation update: the startup cause remains unresolved. No production repair or timeout change has been made.
The two recent failure modes are distinct. Required-main run 39300406644 failed the first Inspector
git init: Worker entered after 51 ms, helper spawned after 60 ms, then no bootstrap record or control-pipe connection was observed before the 10-second owner deadline. The helper closed under cancellation withrequestIssued: false. This does not prove that the PowerShell script never entered, because bootstrap stderr writes can fail. PR #568's Windows run later recorded the same observation boundary in the first profilercontract-capturesample, at Inspectorgit rev-parse.By contrast, both attempts of main run 39353237234 passed their tests and exhausted the 225-minute static-suite budget. The last green Windows suite took about 227 minutes. That supports the separate headroom change in #569; it does not explain the startup stall.
The native controls in draft #571 have now completed 2,560 owned-helper launches and 96 direct PowerShell launches without this failure, across Node 24.19.0/24.21.0 and runner images
20261007.264.1/20261018.275.1. These are non-reproductions, not evidence that a Node version or image fixes the issue.A more representative cold-import comparison also passed on both the last-green commit
de5d707b4a2cf864522c7689017c170bfc577fc1and current main4c40f7f5e154b6fafb410321e8d1ace16eba49f7, using Node 24.21.0 and image20261007.264.1. It preserved the subject checkout at the workspace root and ran the original contract check, fixture materialization and complete ordinary test stage through the existing static-step launcher. Current main recorded 814 tests (786 pass, 28 skip); the last-green commit recorded 807 (779 pass, 28 skip), with zero failures. Both native monitors completed normally.For helpers matched to the original owner path, observed READY latency was median/max 378/2,362 ms on current main and 540/759 ms on the last-green commit. Intentional copied-owner timeout fixtures were kept separate. Neither run triggered a natural pre-connection stack capture. The instrumentation adds file markers and pipe observations, so its observer effects remain a limitation.
The next native diagnostic preserves the profiler's real report/inspection setup and original three-sample command sequence through
contract-capture. It is still running. Inspection of the cold artifacts also found PID reuse in the diagnostic monitor; #571 now keys stack captures and progress markers by launch time as well as PID. The already-running profiler diagnostic predates that observer correction, so a missing stack there cannot rule out a stall after PID reuse.The pre-request startup investigation stays open. No retries, timeout increases, prewarming workaround, or suppression are being presented as a repair.
steipete commentedon Oct 21, 2026
The retained-profiler diagnostic completed successfully. All twelve samples (three each of Node boot, fixture inspection, compatibility reporting, and contract capture) exited zero. It used frozen main
4c40f7f5e154b6fafb410321e8d1ace16eba49f7, Node 24.21.0, image20261007.264.1.The first inspection sample recorded a -1,084 MiB harness heap delta, reproducing the large allocation/collection history that isolated helper controls omitted. Native counters reached 3,836 MiB of aggregate private process memory. Matched original-owner READY latency was median/max 578/2,561 ms. The monitor completed normally (17,407 snapshots; maximum sample gap 1,347 ms). No natural startup failure was reproduced. No stack capture occurred; as noted above, this run predates the PID-marker correction, so absence of a stack alone is not a negative finding.
The remaining context difference is the preceding complete static pipeline. Run 39564565856, diagnostic head
e84f50ba710e129bf2af4a01afaf738eada161a5in #571, now observes the original 23-step pipeline against the same frozen main, including report writes, track inputs, and three-sample profiling. The monitor includes the PID-reuse correction. Only the outer diagnostic time limit provides extra evidence-collection time; command startup/execution limits and admission/cleanup protocol remain unchanged.This is still an investigation. No production startup repair, main merge, or release has been made. #569 remains separate and held pending the startup result. The earlier superseded uninstrumented draft Check run was cancelled after the complete instrumented Windows job started; its successful advisory run is not being presented as repair proof.
steipete commentedon Oct 21, 2026
Correction to the earlier cold-run latency summary: the scratch analysis required helper-spawn and initial pipe-snapshot timestamps to match exactly, which dropped observations written one millisecond later. It omitted last-green’s first cold launch. Correlation now uses parent PID plus one unambiguous initial snapshot within 10 ms, consumed once per launch. All inspected commands with a helper-spawn phase correlate.
Complete owner-start → READY median/max: last-green 541/2,638 ms, current main 378/2,362 ms, retained-profiler prefix 578/2,561 ms. Test pass/fail results are unchanged. The data does not establish a startup-latency regression.
The first four of eight independent fresh Windows trials passed (original-owner maxima 2.16–2.80 s); the remaining trials and the replacement complete-pipeline run continue. The earlier full run 39564565856 was intentionally cancelled because PowerShell-based debugger verification could prewarm startup components. The replacement verifies pinned debugger hashes in Python. No natural 10-second startup stall or supported root-cause repair has been captured.
steipete commentedon Oct 21, 2026
The eight independent fresh-runner trials in 39572677987 all passed, but trial 6 captured a much slower first helper. Owner → READY was 9,359 ms; helper-spawn → script-entry was 8,218 ms, and helper-spawn → pipe-connect was 8,495 ms. This is closer to the historical 10-second boundary than the earlier warm controls.
The first six seconds are useful evidence: before debugger launch, the helper was progressing (328 ms accumulated CPU, 172 KB read-transfer count, 496 handles at +6,109 ms) while no script-entry marker or stderr bytes had appeared. Native and pipe observations continued roughly every 253 ms. That supports slow pre-entry initialization; it does not identify a specific loader, CLR, Defender, or Node defect.
The full 9.36-second measurement has an observer limitation. The non-suspending debugger ran from +6,109 to +8,135 ms, and the script-entry marker appeared 83 ms after it exited. Its dot-command parser treated the semicolon-separated stack commands as part of the symbol path, so no full stacks were collected. The single automatically displayed instruction is insufficient to attribute the wait. The diagnostic branch is being corrected to use a command file, with a post-workload synthetic check that stack commands execute and detach while leaving the target alive.
The unmodified-source cold control 39575390604 and full-context run 39565264952 remain in progress. No production fix or timeout mitigation is claimed, and #569 remains held.
steipete commentedon Oct 21, 2026
The complete native Windows experiment 39582684761 passed all 23 static stages against frozen main
4c40f7f5e154b6fafb410321e8d1ace16eba49f7, including the original nine-command, three-sample profiler. Node 24.21.0, runner image20261018.275.1. The retained artifact contains the native timeline and command observations. The target exited zero, native monitoring completed, and the post-workload debugger verification collected native stacks and detached with its target alive.There were 293 recorded original-owner commands; the error outcomes were the expected test controls. Owner-start → READY median/max was 561/2,502 ms. The monitor retained 52,054 samples with a maximum 366 ms gap and observed 4,216 MiB aggregate private memory. No natural slow-start stack capture was triggered.
The unmodified-source control also passed: 786 tests, 28 skipped, both owner/helper hashes unchanged. The second eight-run cold cohort passed 8/8 with validated debugger command execution; readiness maxima ranged from 2,178 to 2,780 ms. Combined with the first cohort, that is sixteen independent fresh-runner attempts, separate from the thousands of warm helper controls.
Correction about the earlier full run 39565264952: its live browser log stopped updating after
workspace-plan, but the final raw log proves that stage completed normally at 09:02 UTC and the run advanced into profiling. I manually cancelled a healthy run based on that stale view. Its cancellation supplies no hang or startup-failure evidence; the complete replacement above resolves that missing experiment.Root cause remains unresolved. The historical receipts show missing observed bootstrap/pipe progress before the 10-second deadline and no issued target request. Trial 6's pre-debugger delay supports slow, progressing PowerShell initialization as a hypothesis, but it does not identify the blocking operation or establish a runner, Node, CLR, or Defender cause. Independent assessment of the completed evidence found no supported production fix or workaround. Retries, prewarming, parser/Job changes, and a larger startup deadline would be speculative.
No production repair, merge, or release was made. #571 remains a diagnostic draft. #569 still addresses a separately evidenced suite-budget problem, but stays behind the requested root-fix prerequisite. Main's latest Check remains failed at
4c40f7f(225-minute suite budget); this diagnostic success is not merged-main CI proof. The incident stays open, and the merged-main Windows success criterion has not been met.steipete commentedon Oct 21, 2026
@vincentkoc Peter asked to hand the Windows root-cause investigation to you. The deep dive completed 16 independent fresh-runner trials and a green full 23-stage Windows pipeline, including the original three-sample profiler (full pipeline, first cohort, second cohort). One trial captured slow, progressing PowerShell initialization: 9,359 ms owner-to-READY, with 8,218 ms before script entry. Debugger observer effects limit attribution; no root cause or production fix is established.
The recent required Windows red began at
dd797d0a/ #541; the last green wasde5d707b. Draft #571 retains the diagnostics and evidence. #569 remains available for the separately evidenced suite-budget headroom problem. Peter authorized a one-off waiver of the pre-existing Windows timeout for #570 only; #569 and #571 remain unmerged, and this issue stays open for your investigation.fix: qualify packed retirement smoke with reviewed inspector (#570)
fix: restore development dashboards and refresh fixture pins (#597)
vincentkoc commentedon Nov 4, 2026
Additional native Windows evidence from the current maintenance campaign, with a distinct observed phase:
a6eb86cd8bf618e3b93a483d7b0b0bf1e4d2c5a3, host7c2ff8f775acca3e5c93c9c22b6247e37df29bae.Windows actual compiler lifecyclefails in its assembly-preparation control after the unchanged 35000 ms command deadline. The assertion isassembly preparation must return confirmed native closure.observer-compile-startedandobserver-compile-completedboth absent, withETIMEDOUT. Earlier startup phases includepipe-connected,preparation-started,prepared-received, andrequest-write-attempted.This does not establish the same cause as the original cold-start failure, identify the blocked native phase after the request, or establish confirmed closure. No timeout increase, owner modification, retry, or waiver was performed. The asynchronous-loader candidate does not change the process owner or this test.
Separately, #725 records an
EPERMfrom lifecycle state-directory cleanup in an older canary; it is a different observed error and is not treated as a supervisor diagnosis.