Skip to content

Restore green CI on main #21246

Closed
Closed

Description

@radical

[automated] Generated: 2026-10-30 20:30 UTC

❌ Latest explicit test/build failures

Latest CI: run · 59167390 · attempt 1 · failure.

Area / evidence Observed failure
Hosting.Maui (windows-latest) building-state wait: 30-second timeout
Hosting-1 (windows-latest) explicit-start readiness cancelled

Extension jobs, latest run: 54 passed, 0 failed, 0 pending/running. Job success alone does not establish every test passed.

What repeatedly keeps CI red?

Priority: Wed Oct 28, Thu Oct 29, Fri Oct 30 (UTC; weekends do not consume a day). Seven days: 53 failed/timed out, 15 cancelled, 0 pending/running, 0 green.

3d / 7d means affected runs in the last three weekdays / seven calendar days. Problem names link to evidence; — means no verified matching issue/PR. Current observations first, then last-seen recency.

Reason / last seen 3d / 7d Tracking
Explicit-start lifecycle / Windows · Oct 30 8 / 16 PR #21397 · #21328 · #21026
Windows runner loss · Oct 30 24 / 43 #20567 (closed)
Azure sandbox cancellation · Oct 30 3 / 3 #20931
Extension diagnostics lock holder · Oct 30 2 / 2 —
Extension resource debugger control · Oct 30 2 / 2 —

5 visible + 15 more test groups + 4 build/execution groups + 4 older groups = 28 recurring groups. Counts overlap across reasons.

🔍 More test failures — 15 groups
Reason / last seen 3d / 7d Tracking
Azure Functions shutdown · Oct 30 2 / 3 #21245
Terminal-host socket cleanup · Oct 30 2 / 2 #21409
Deno startup / telemetry fixtures · Oct 30 5 / 8 #21184 · #21185
DevTunnel disposed service provider · Oct 30 2 / 2 #20859
MongoDB replica-set startup · Oct 30 4 / 11 #19980
Rebuild-command assertions · Oct 30 3 / 3 #20750 · #20662
Terminal-host owner exit · Oct 30 1 / 2 #20902
Dashboard exemplar dialog · Oct 30 1 / 3 #21126
Terminal-host RPC cancellation · Oct 29 2 / 3 #20903
Java container telemetry assertion · Oct 29 1 / 4 —
Extension Deno debug startup · Oct 29 2 / 3 —
Integration-host registration · Oct 28 2 / 3 —
CLI ambient-discovery timeout · Oct 28 1 / 4 #21335
Extension Node Functions breakpoint · Oct 28 1 / 6 —
CLI graceful-stop escalation · Oct 28 1 / 2 —

Investigation notes

  • Explicit-start lifecycle / Windows: Failures span startup/readiness and shutdown. The current run fails waiting for the resource predicate, not in StopAsync. Next: Investigate the current startup/readiness failure separately. PR Add explicit-start shutdown regression coverage (startup-token cancellation and process exit) #21397 addresses shutdown-token ownership, not every failure of this method.
  • Azure sandbox cancellation: Assert.False: expected false, actual true. Next: Investigate cancellation ordering.
  • Extension diagnostics lock holder: Storage diagnostics' exclusively held lock test loses its lock-holder process. Next: Investigate lock-holder lifecycle.
  • Extension resource debugger control: Packaged debugger attach/breakpoint/detach test fails applying an E2E control revision. Next: Investigate debugger/control lifecycle; do not merge with runner-loss occurrences in this shard.
  • Azure Functions shutdown: Multi-project test is cancelled during application shutdown. Next: Investigate shutdown separately from the explicit-start test's fix.
  • Terminal-host socket cleanup: Socket-cleanup test throws cancellation on Windows. Next: Investigate shutdown and socket cleanup.
  • Deno startup / telemetry fixtures: Functional-fixture timeouts and telemetry initialization/address failures are grouped by area; they are not proven to share one root cause. Next: Investigate the fixture/telemetry variants separately. The linked issues track the two functional-fixture tests.
  • DevTunnel disposed service provider: Shared observer throws ObjectDisposedException for IServiceProvider on Windows and Ubuntu. Next: Investigate shared-observer lifetime.
  • MongoDB replica-set startup: Resource rs0 fails to start or become healthy across healthy-node, topology/data-reuse and Mongo Express test variants. These share the observed startup failure, not a proven underlying cause. Next: Investigate replica-set startup.
  • Rebuild-command assertions: Command-success and build-count assertions fail in the same test. Next: Keep Assert.True and build-count mismatch signatures distinct.
  • Terminal-host owner exit: The managed terminal-host owner-exit test fails. Next: Compare the current exception/exit details with the tracked owner-exit timeout.
  • Dashboard exemplar dialog: Assert.Single finds an empty exemplar-dialog collection. Next: Investigate dialog readiness and exemplar availability.
  • Terminal-host RPC cancellation: Cancellation-delivery test reaches its 30-second timeout on Ubuntu. Next: Investigate cancellation delivery.
  • Java container telemetry assertion: Telemetry URL assertion fails. The latest occurrence expects http://:4317 but receives http://. Next: Investigate the URL construction/configuration signature; do not assume a past hostname-assertion fix covers this.
  • Extension Deno debug startup: Debug AppHost startup times out; teardown also encounters EBUSY. Next: Investigate startup and cleanup; do not assume an infrastructure flake.
  • Integration-host registration: Registration/discovery test times out on Windows. Next: Investigate registration/discovery ordering.
  • CLI ambient-discovery timeout: The macOS command-resolution test times out after 30 seconds. Next: Investigate command/test lifetime; do not conflate this assertion with a later test-host exit.
  • Extension Node Functions breakpoint: Same E2E test times out waiting for the startup breakpoint in Program.cs:4. Next: Investigate the startup breakpoint/control path; do not infer an HTTP-request breakpoint failure.
  • CLI graceful-stop escalation: Both console-isolation variants hit the same 30-second timeout while testing escalation when a process ignores graceful stop. Next: Investigate the process wait/termination path; compare the two theory variants.
⚠️ Build/execution problems — 4 more recurring groups; other failures
Reason / last seen 3d / 7d Tracking
Homebrew local-tap trust · Oct 30 2 / 2 PR #21429 (merged) · #21410
Windows post-test process startup · Oct 29 2 / 4 #18720
Artifact-service errors · Oct 29 3 / 3 —
macOS CLI test-host exit 137 · Oct 28 7 / 18 #20543
  • Windows runner loss — investigate: Explicit runner-communication-loss annotations across different Windows .NET suites and extension E2E shards. Next: Correlate runner/process/resource diagnostics. The closed historical issue is not proof of today's cause.
  • Homebrew local-tap trust — fix merged; verify newer image: The cask audit rejects the untrusted local tap. Next: Recorded successful trust/audit evidence used image 20261021; verify the affected newer image before closing out.
  • Windows post-test process startup — investigate: Post-test processes fail with exit -1073741502 after a passing test summary. Next: Investigate process initialization rather than test assertions.
  • Artifact-service errors — investigate: Artifact listing/creation timeouts and a busy-server upload failure. These are service/transport symptoms, not one proven cause. Next: Separate service failures from assertions and runner loss.
  • macOS CLI test-host exit 137 — investigate: Test application exits 137; the outer test command exits 7. Some jobs also contain separate failed tests. Next: Investigate host/session termination. Exit 137 alone does not establish OOM.

Other recorded failures

Separate signatures can occur in the same suite; these rows are not additional recurring-pattern counts.

Area / evidence Affected runs Observed
Hosting.Maui (windows-latest) 1 building-state wait: 30-second timeout
Python SDK Validation 1 Python image: uv not found; exit 127
Hosting.CodeGeneration.Go (windows-latest) 2 GeneratedRuntime PassesGoRuntimeHardeningTests
Dashboard (windows-latest) 2 BrowserView ProducerDisconnectClosesBrowserWhileWaitingForAcknowledgement
Build packages 1 historical includeDocs compilation break; recovery evidence below
Hosting.RemoteHost (ubuntu-latest) 1 historical includeDocs compilation break; recovery evidence below
Hosting.RemoteHost (windows-latest) 1 historical includeDocs compilation break; recovery evidence below
Hosting.CodeGeneration.Go (ubuntu-latest) 1 historical includeDocs compilation break; recovery evidence below
Hosting.DevTunnels (windows-latest) 2 DcpStartupPublishesDevTunnelUrls
Infrastructure (ubuntu-latest) 3 LastSuccessfulMainRunFallsBackToOlderWindow
Cli (windows-latest) 3 ComputeRunArgumentsPopulatesRunPropertiesForSdkProject
Hosting-4 (windows-latest) 2 GetResourcesAsync MapsNonStringPropertiesAsJsonForV3Callers
VS Code extension E2E (Windows, apphost-tree) 1 recorded job failure; cause requires investigation
Hosting.Azure.Provisioning (windows-latest) 1 recorded job failure; cause requires investigation
Java SDK Validation 1 recorded job failure; cause requires investigation
Templates-BuildAndRunTemplateTests (ubuntu-latest) 1 CreateAndModifyAspireAppHostTemplate
Hosting.Native.Core (ubuntu-latest) 1 SilentAndPartialFramesExpireWithoutBlockingOtherCliClients
VS Code extension E2E (Linux, apphost-lifecycle-tools) 1 recorded job failure; cause requires investigation
Hosting-5 (windows-latest) 1 job failed; log unavailable, cause unknown
TypeScript SDK Validation (Node 24.x) 1 Nuxt/backend readiness timeout
VS Code extension E2E (Linux, diagnostic-logs) 1 recorded job failure; cause requires investigation
VS Code extension E2E (Windows, browser-debugger) 1 recorded job failure; cause requires investigation
Validate VS Code extension bootstrap (Linux) 1 recorded job failure; cause requires investigation
Dashboard.Components (windows-latest) 1 EmptyState HelpDisappearsAfterAnySignalAndStaysHiddenAfterClear; 1 more failed test(s)
Hosting.DevTunnels (ubuntu-latest) 1 UnrecognizedOutputWarnsOnceAndReconcilesReadiness; 1 more failed test(s)
Azure.Search.Documents (ubuntu-latest) 1 recorded job failure; cause requires investigation
VS Code extension E2E (Linux, launch-profiles) 1 dependency download: HTTP 504
Run VS Code extension unit tests (Windows) 1 dependency download: HTTP 500
VS Code extension E2E (Windows, launch-profiles) 1 recorded job failure; cause requires investigation
Cli (ubuntu-latest) 1 PrefetchingCancellationDueToShutdownLogsCleanMessage

Daily runner-loss counts

Run start date UTC Affected runs Lost jobs
2026-10-23 3 15
2026-10-24 1 4
2026-10-25 1 2
2026-10-26 6 14
2026-10-27 8 14
2026-10-28 9 24
2026-10-29 8 16
2026-10-30 7 15
⏳ Older observations — 4 groups, not confirmed fixed; coverage
Reason / last seen 3d / 7d Tracking
CLI missing-SDK command timeouts · Oct 27 0 / 4 #20940 · #20939 · #20911
Shared child-process shutdown · Oct 27 0 / 4 —
CLI pinned-template checkout · Oct 27 0 / 3 #21335
Guest-launcher graceful shutdown · Oct 27 0 / 2 #21335

No matching failure was observed in the last three weekdays. A quiet stretch is not proof of a fix or of sufficient later executions. Zero weekly counts can mean the last observation is outside this window.

Verified recoveries

Coverage and counting

  • Run-start window: 2026-10-23T20:24:02Z through 2026-10-30T20:24:02Z. Run inventories missing: 0; failed-job evidence available: 244/245.
  • Weekdays use run-start UTC date; last seen uses failed-job completion time. Monday uses Thursday, Friday and Monday. Weekend failures remain in weekly/current evidence.
  • Counts include failures observed before workflow cancellation and in still-running workflows. Cancellation alone is not a failure, and missing results are not passes.
  • Affected runs are deduplicated per reason. Groups overlap; do not add their affected-run totals. Execution eligibility has not been established, so no failure-rate denominator is claimed.
  • Current observations use the newest run and latest completed non-cancelled run, at their latest attempts. Older failures are not automatically carried forward as current.
  • Merged fixes, closed issues, later green jobs and quiet periods do not establish permanent resolution. This is a CI-blocker report, not an exhaustive flaky-test tracker.
  • Logs/annotations and retained local observations supply counts; issue/PR bodies supply only verified tracking context.

Activity

  1. added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Oct 21, 2026
  2. radical commented on Oct 23, 2026

    @radical
    Author
  3. radical commented on Oct 23, 2026

    @radical
    Author
  4. radical commented on Oct 23, 2026

    @radical
    Author
  5. radical commented on Oct 23, 2026

    @radical
    Author
  6. radical commented on Oct 23, 2026

    @radical
    Author
  7. radical commented on Oct 23, 2026

    @radical
    Author
  8. radical commented on Oct 23, 2026

    @radical
    Author
  9. radical commented on Oct 23, 2026

    @radical
    Author
  10. radical commented on Oct 23, 2026

    @radical
    Author
  11. radical commented on Oct 23, 2026

    @radical
    Author
  12. radical commented on Oct 23, 2026

    @radical
    Author
  13. radical commented on Oct 23, 2026

    @radical
    Author
  14. radical commented on Oct 23, 2026

    @radical
    Author
  15. radical commented on Oct 23, 2026

    @radical
    Author
  16. radical commented on Oct 23, 2026

    @radical
    Author
  17. radical commented on Oct 23, 2026

    @radical
    Author
  18. radical commented on Oct 23, 2026

    @radical
    Author
  19. radical commented on Oct 23, 2026

    @radical
    Author
  20. radical commented on Oct 23, 2026

    @radical
    Author
  21. radical commented on Oct 23, 2026

    @radical
    Author
  22. radical commented on Oct 23, 2026

    @radical
    Author
  23. added
    area-engineering-systemsinfrastructure helix infra engineering repo stuff
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Oct 30, 2026
  24. radical commented on Oct 30, 2026

    @radical
    MemberAuthor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-engineering-systemsinfrastructure helix infra engineering repo stuff

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions