Skip to content

test(windows): investigate missing supervisor startup progress - #571

Draft
steipete wants to merge 12 commits into
mainfrom
fix/windows-supervisor-startup
Draft

steipete wants to merge 12 commits into
mainfrom
fix/windows-supervisor-startup

Conversation

@steipete

@steipete steipete commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Investigate the intermittent Windows command-supervisor startup failure in #305. This is a diagnostic draft, not a production repair. No startup deadline, retry policy, admission boundary, or production helper source change is proposed. #569 remains held pending the startup investigation. Related work and evidence: @vincentkoc, #541, #568.

Historical failure and commit range

Main 39300406644 failed during the first Inspector git init. Worker entry arrived at +51 ms and helper spawn at +60 ms; neither bootstrap stderr nor a control-pipe connection was observed before the 10-second deadline. The helper closed under SIGTERM with requestIssued: false. The first profiler contract-capture sample in #568's run recorded the same observation boundary during git rev-parse. Missing bootstrap does not prove the script never entered, since bootstrap itself uses stderr.

The last green main was 39215227316, de5d707b4a2cf864522c7689017c170bfc577fc1. The intervening commits are dashboard output, #541's diagnostics, and #566's fixture/CI changes. The green and startup-failing runs used Node 24.21.0 and image 20261007.264.1; no causal commit or version regression is established. Issue #305 predates this range.

Both attempts of main run 39353237234 passed the ordinary tests and separately exhausted the 225-minute static-suite allowance. The last green suite took about 227 minutes. That supports #569's CI headroom independently of the startup failure.

Native Windows evidence

  • 39552446911 and 39553195567: 2,560 owned-helper launches and 96 raw PowerShell controls passed across Node 24.19/24.21 and both observed runner images. These are predominantly warm-runner launches, not thousands of independent cold OS starts.
  • 39554908105: the last-green and current-main cold-import paths passed on separate fresh runners. Ordinary tests: 779/786 pass respectively, 28 skipped in each. Complete owner-start → READY median/max was 541/2,638 ms for last-green and 378/2,362 ms for current main.
  • 39555680600: all 12 retained-profiler samples passed, including the original contract-capture sequence. The first inspection sample recreated a −1,084 MiB heap delta; observed aggregate private memory reached 3,836 MiB. Owner-start → READY median/max was 578/2,561 ms. This run predates the PID-reuse observer correction.
  • 39572677987: all eight independent fresh-runner trials passed. Trial 6 is a useful slow-start observation: owner → READY 9,359 ms; helper spawn → script-entry 8,218 ms; helper spawn → pipe-connect 8,495 ms. Before the debugger started at +6,109 ms, the helper had made progress (328 ms CPU, 172 KB read-transfer count, 496 handles) without script-entry or stderr output. The full latency is observer-affected: CDB overlapped +6,109–8,135 ms, and script entry followed 83 ms later. No external root cause is established.

The initial latency summarizer required exact timestamp equality and omitted snapshots written one millisecond after helper-spawn, including last-green's first cold launch. The corrected same-parent, unique initial-snapshot correlation accounts for every inspected command with a helper-spawn phase. The values above are corrected; test pass/fail results never changed.

Diagnostic boundaries and results

The instrumentation preserves original static-suite commands and target deadlines. It records file entry/pipe markers, pipe byte counts, and native process/thread/handle/memory counters. Stack capture uses noninvasive, non-suspending -pvr, is bounded, and excludes copied timeout fixtures. It requests no arguments, locals, environment dumps, or memory dumps. Early monitor loss makes evidence incomplete. Runtime source instrumentation is restored after the run.

Trial 6 exposed a debugger command-delivery bug: .sympath consumed the semicolon-separated remainder as a path, so the requested full stacks did not execute. The latest branch uses -y and a command file with separate lines, plus a post-workload synthetic check for executed stack commands and detach with the target still alive. A single automatic instruction display is not a substitute for the missing stacks.

Additional experiments:

  • 39565264952 is incomplete due to manual cancellation. The live browser log stopped updating at workspace-plan; the final raw log proved that stage completed normally and the run had advanced into profiling. Its first 20 static stages completed, and 235 recorded original-owner commands had no unexpected startup failure (maximum READY 2,713 ms). The cancellation is not failure evidence. Its debugger predates the command-file correction.
  • 39582684761 completed successfully: all 23 static stages, including nine-command/three-sample profiling, on Node 24.21.0 and image 20261018.275.1. The target and native monitor exited zero; debugger verification passed. Across 293 original-owner commands, error outcomes were expected test controls; owner-start to READY median/max was 561/2,502 ms. Native sampling completed with 52,054 observations, maximum gap 366 ms, and 4,216 MiB maximum aggregate private memory. No natural slow-start capture was triggered. Retained artifact.
  • 39575390604 passed: cold-import control with owner/helper source untouched and external native counters only. All 786 tests passed, 28 skipped. Both source-integrity checks matched; the monitor completed. No debugger attached.
  • 39575658239 passed 8/8: fresh cold runners using corrected debugger command delivery, maximum four concurrent. All post-workload debugger verifications executed native stack commands and detached. No natural slow-start capture was triggered; owner-to-READY maxima ranged from 2,178 to 2,780 ms. Further cold-run loops are stopped.

Earlier full-context run 39564565856 was intentionally cancelled because PowerShell-based signature verification could prewarm startup components. The replacement uses pinned hashes of the previously signature-verified Microsoft debugger files. That cancellation and the earlier corrected LF/CRLF/debugger setup attempts are not natural startup failures.

Validation: actionlint, JavaScript/Python syntax checks and LF/CRLF probe preparation pass. Unchanged local ownership tests: 16 pass, 26 Windows-only skips. Diagnostic changes received independent P0–P2 autoreview; identified observer defects were fixed and re-reviewed. Investigation commits intentionally skip broad CI. Any production repair still requires regression coverage, exact-head CI, and Windows success on merged main.

Outcome

Root cause remains unresolved after the complete pipeline, sixteen fresh cold-runner attempts, an unmodified-source control, and warm helper controls. Independent assessment found no supported production fix or workaround. No production code, merge, or release was made. Main remains at 4c40f7f with its latest Check failed for the separate 225-minute budget. #569 stays behind the requested root-fix prerequisite; merged-main Windows success has not been achieved. Keep this as an investigation draft and retain #305 as the incident.

@clawsweeper

clawsweeper Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@steipete steipete changed the title test(windows): investigate supervisor pre-entry stalls test(windows): investigate missing supervisor startup progress Oct 21, 2026
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 21, 2026
@clawsweeper

clawsweeper Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

What this changes

This PR adds Windows workflows and diagnostic scripts to observe command-supervisor startup, native process activity, and bounded stack captures.

Example: A Windows helper exceeds the existing 10-second startup deadline before connecting.

  • Before: Available receipts show helper spawn but cannot distinguish missing script progress from missing stderr delivery.
  • After: The diagnostic run additionally records file markers, pipe byte counts, native counters, and an eligible stack capture after six seconds.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) Useful hosted diagnostic evidence supports the investigation, but one monitor-failure defect remains and the draft establishes no startup repair.
Proof confidence 🐚 platinum hermit (4/6) This maintainer-authored investigation is outside the ordinary contributor-proof gate; supplied hosted Windows runs exercise native controls and cold-suite instrumentation, while the latest PID-reuse correction and natural-stall stack path remain unobserved.
Patch quality 🦐 gold shrimp (3/6) 1 actionable review finding remain.

Product

Kind: Test only · Worth it: Yes · Fix scope: Partial
User problem: Windows validation intermittently stops before a command is admitted, and existing receipts cannot establish the stalled startup boundary.
Reason: The maintainer-authored draft authorizes a focused investigation while explicitly preserving the distinction between diagnostic controls and a production repair.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep open: this maintainer-led diagnostic draft supports an unresolved Windows startup investigation and explicitly remains held from landing.

Priority: P2
Reviewed head: e0c9501c1ce63bf6e3db11ad87b7020ae75a3219

Before merge

  • Fail the startup probe when native monitoring is incomplete (P2) - If the Python monitor exits after readiness—for example, when NtQuerySystemInformation raises an error—the launch controls can all succeed and this script still exits zero. The monitor's code and signal are only logged, and failed reflects command outcomes alone. This lets the workflow publish a successful diagnostic run without native observations for the remaining launches. Propagate unsuccessful monitor termination and detect premature completion, as the cold-import runner already does.

Findings

  • [P2] Fail the startup probe when native monitoring is incomplete — scripts/diagnose-windows-startup.mjs:114-116

Tests

  • Missing end-to-end proof: The supplied runs do not exercise a natural pre-connection stall or the latest PID-reuse correction; their successful controls establish diagnostic execution rather than a regression repair.
Agent review details

How this fits together

Crabpot's static-suite and profiler commands use a shared command supervisor; these optional diagnostics instrument that existing owner and retain observations as GitHub Actions artifacts.

flowchart TD
  A[Windows diagnostic workflow] --> B[Startup controls or cold suite]
  B --> C[Existing command supervisor]
  C --> D[PowerShell helper]
  D --> E[Native command]
  C --> F[Pipe observations and file markers]
  G[Native monitor and debugger] --> D
  F --> H[Diagnostic artifacts]
  G --> H
Loading

Technical review

Best possible solution:

Make incomplete synthetic monitoring fail explicitly, then retain the bounded investigation until attributable evidence supports a repair in the existing supervisor.

Do we have a high-confidence way to reproduce the issue?

No deterministic reproduction of the natural startup stall is established: real Windows failures are recorded, but the hosted controls and cold-import comparison did not reproduce them; this review executed no target code.

Is this the best way to solve the issue?

Bounded observation at the existing command owner is appropriate for the approved investigation, but it supplies diagnostics rather than a startup repair.

Full review comments:

  • [P2] Fail the startup probe when native monitoring is incomplete — scripts/diagnose-windows-startup.mjs:114-116
    If the Python monitor exits after readiness—for example, when NtQuerySystemInformation raises an error—the launch controls can all succeed and this script still exits zero. The monitor's code and signal are only logged, and failed reflects command outcomes alone. This lets the workflow publish a successful diagnostic run without native observations for the remaining launches. Propagate unsuccessful monitor termination and detect premature completion, as the cold-import runner already does.
    Confidence: 0.97

Overall correctness: patch is incorrect
Overall confidence: 0.94

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 4c40f7f5e154.

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

  • Applicable repository policy: Read the complete root AGENTS.md and searched the changed-path ancestors; no nested AGENTS.md, maintainer-notes directory, or pr-simplify skill was present. The diagnostics preserve fixture-driven inputs and leave default checks unchanged. (AGENTS.md:1, e0c9501c1ce6)
  • Verified introduced change: The pinned introduction diff adds only the diagnostic workflow and five scripts; it does not modify the production supervisor, its deadlines, or cleanup protocol. (.github/workflows/windows-startup-debug.yml:1, e0c9501c1ce6)
  • Owner-recorded investigation hold: The full PR body explicitly says no root-cause repair is established and holds landing; it credits vincentkoc's related work and distinguishes passing controls from reproduction. (e0c9501c1ce6)
  • Canonical unresolved failure: Investigate intermittent Windows cold-start failure in command supervisor #305 (comment) records required-main and profiler startup failures with no command request issued, while explicitly separating them from suite-budget exhaustion and Inspector heap failures. (scripts/owned-command.mjs:12, 4c40f7f5e154)
  • Hosted diagnostic evidence and limits: GitHub confirms both cold-import comparison jobs succeeded at diagnostic head 3edbf5d. The body and linked discussion report completed monitors and no natural startup stall; the profiler diagnostic was still running at the read. Raw log retrieval was blocked at GitHub's artifact-storage redirect, so its contents were not independently inspected. (3edbf5d13fe1)
  • Synthetic monitor failure is ignored: The startup probe logs monitor termination but sets failure only from command results; the cold-import probe separately checks monitor completeness and propagates failure. (scripts/diagnose-windows-startup.mjs:114, e0c9501c1ce6)

Root-cause cluster

Relationship: partial_overlap
Canonical: #305
Summary: The canonical issue owns the unresolved startup failure; this PR adds diagnostic evidence, while suite-headroom work addresses a distinct failure.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Labels

Label changes:

  • add P2: The work investigates intermittent Windows validation failures without evidence of an urgent end-user runtime outage.
  • add merge-risk: 🚨 automation: The added diagnostic workflow can report success despite losing its native observation process.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

  • P2: The work investigates intermittent Windows validation failures without evidence of an urgent end-user runtime outage.
  • merge-risk: 🚨 automation: The added diagnostic workflow can report success despite losing its native observation process.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 21, 2026, 2:53 AM ET / 06:53 UTC.

Comment on lines +82 to +85
- name: Prepare pinned Microsoft console debugger without PowerShell warmup
shell: bash
run: python .crabpot/windows-diagnostic-tools/scripts/prepare-windows-debugger.py "$RUNNER_TEMP/crabpot-debugger"
- name: Reproduce the actual cold CI import with native observations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants