Skip to content

Enable WSLC container tests on existing Windows CI runners - #21564

Open
Alex Crome (afscrome) wants to merge 18 commits into
microsoft:mainfrom
afscrome:afscrome-wslc-container-tests-on-windows-ci
Open

Alex Crome (afscrome) wants to merge 18 commits into
microsoft:mainfrom
afscrome:afscrome-wslc-container-tests-on-windows-ci

Conversation

@afscrome

@afscrome Alex Crome (afscrome) commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Description

Run container tests with WSLc in the existing Windows CI jobs, extending container coverage that previously ran only on Linux. The Windows runners prepare WSLc and trusted development certificates, and Aspire selects the runtime automatically. No additional CI jobs are introduced.

This PR also enables Aspire's container image publishing pipeline to use WSLc, including Dockerfile builds, SDK publishing to the local image store, and archive export. Runtime-log assertions accept WSLc output, and Redis readiness and Kusto log-category assertions are corrected so those scenarios run reliably.

Docker/Podman coverage and existing AzDO/Helix image-build restrictions are preserved. Compose tests remain limited to Docker/Podman. MongoDB's Windows bind-mount persistence case is disabled pending microsoft/WSL#42129; named-volume coverage remains enabled. WSLc Dockerfile builds support the native Linux architecture only.

Targeted publishing, archive execution, runtime-log, database REPL, Redis, Kusto, and capability-matrix tests passed locally on Windows. The updated PR CI run is pending verification.

Known Issues:

  • WSL upgrade sporadically fails due to WSL --install fails with "Forbidden (403)." WSL#40285 . This would stop being a problem once wsl 3.0.4 or greater is marked as a proper release with github actions picks it up in the default image. It can probably also be worked around by getting the WSL installer another way or with authentication.
  • REPL tests are particularly flakey with WSL
  • MongoDB currently fails on WSLc MongoDB 9.0 fails with Windows bind mounts in WSLC but works in Docker Desktop WSL#42129
  • tests/Aspire.Hosting.TestUtilities/Utils/ContainerRuntimeLogPatterns.cs is going to be problematic with localisation. (I already had some issues with recognised vs recognized - I'm sure non English languages will be worse).

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21564

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21564"

@github-actions github-actions Bot added the area-engineering-systems infrastructure helix infra engineering repo stuff label Oct 31, 2026
Alex Crome (afscrome) and others added 6 commits October 31, 2026 09:33
Preserve Linux Testcontainers initialization and other CI platform gates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Report existing runtime-specific test failures without changing their assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The preinstalled Windows engine reports healthy but cannot provide Linux bridge networks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Gate Compose deployment independently from generic container execution.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Alex Crome (afscrome) and others added 3 commits November 1, 2026 18:56
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Alex Crome (afscrome) and others added 5 commits November 1, 2026 20:02
Avoid connecting while WSLC is still pulling the Redis image.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com
Derive the shared database failure log category from the builder application name instead of a stale test assembly name.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add the WSLc publishing runtime, preserve image-build capability gating, and skip MongoDB Windows bind mounts pending microsoft/WSL#42129.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues remain in WSLc publishing, archive handling, runtime capability checks, and Windows test setup and coverage.

8 open issues
What changed in this PR

This PR extends Aspire’s container testing to existing Windows GitHub Actions runners using WSLc and adds WSLc support to the container image publishing pipeline.

Changes:

  • Prepare Windows runners with WSLc and trusted development certificates, and enable additional test projects.
  • Add WSLc image building, inspection, archive export, and SDK local-store selection.
  • Update capability gates, runtime-log assertions, and database tests for Windows container coverage.
File Description
tests/​Aspire.TestUtilities/​TestFeature.cs Document WSLc capabilities and add a Compose feature flag.
tests/​Aspire.TestUtilities/​RequiresFeatureAttribute.cs Extend Windows capability checks and separate Compose support.
tests/​Aspire.Playground.Tests/​Aspire.Playground.Tests.csproj Enable Windows GitHub Actions execution.
tests/​Aspire.Hosting.TestUtilities/​Utils/​ContainerRuntimeLogPatterns.cs Recognize WSLc build and failure messages.
tests/​Aspire.Hosting.Tests/​Publishing/​WslcContainerRuntimeTests.cs Test WSLc publishing commands and capability restrictions.
tests/​Aspire.Hosting.Tests/​Publishing/​ContainerRuntimeResolverTests.cs Cover explicit WSLc runtime selection.
tests/​Aspire.Hosting.Redis.Tests/​RedisFunctionalTests.cs Wait for Redis health before accessing it.
tests/​Aspire.Hosting.MongoDB.Tests/​MongoDbFunctionalTests.cs Skip Windows bind-mount persistence pending an upstream fix.
tests/​Aspire.Hosting.Dotnet.Tests/​DotnetProjectResourceTests.cs Adapt image inspection and archive execution for WSLc.
tests/​Aspire.Hosting.Docker.Tests/​DockerComposeTests.cs Gate deployment tests on Compose support.
tests/​Aspire.Hosting.Containers.Tests/​WithDockerfileTests.cs Update a comment to include WSLc log differences.
tests/​Aspire.Hosting.Containers.Tests/​ResourceFailureLoggingTests.cs Use runtime-aware invalid-option matching.
tests/​Aspire.Hosting.Containers.Tests/​RequiresFeatureTests.cs Verify CI capability expectations.
tests/​Aspire.Hosting.Containers.Tests/​ContainerRuntimeLogPatternsTests.cs Cover WSLc log patterns and unrelated output.
tests/​Aspire.Hosting.Azure.Kusto.Tests/​KustoFunctionalTests.cs Match resource logs using the actual application name.
tests/​Aspire.EndToEnd.Tests/​Aspire.EndToEnd.Tests.csproj Enable Windows GitHub Actions execution.
src/​Aspire.Hosting/​Publishing/​WslcContainerRuntime.cs Implement WSLc image operations and unsupported-operation handling.
src/​Aspire.Hosting/​Publishing/​ResourceContainerImageManager.cs Select the WSLc SDK local image store.
src/​Aspire.Hosting/​Publishing/​IContainerRuntime.cs Include WSLc in interface documentation.
src/​Aspire.Hosting/​Publishing/​ContainerRuntimeResolver.cs Allow automatic WSLc selection for publishing.
src/​Aspire.Hosting/​Publishing/​ContainerRuntimeBase.cs Expose image-config parsing to runtime subclasses.
src/​Aspire.Hosting/​DistributedApplicationBuilder.cs Register the WSLc publishing runtime.
docs/​ci/​azdo-public-pipeline.md Document Windows setup and runtime capability boundaries.
.github/​workflows/​run-tests.yml Prepare WSLc and certificate trust on Windows runners.

Review effort: Balanced

Comment on lines +175 to +176
wsl --update --web-download --pre-release
if ($LASTEXITCODE -ne 0) { throw "WSL update failed: $LASTEXITCODE" }
Comment on lines +63 to +66
if (options.TargetPlatform is not null && options.TargetPlatform != nativePlatform)
{
throw new DistributedApplicationException(
$"WSLc cannot build images for the requested platform '{options.TargetPlatform}'. Only the native Linux platform is supported.");
throw new ArgumentOutOfRangeException(nameof(options), options.ImageFormat, "Invalid container image format");
}

var arguments = $"image build --file \"{dockerfilePath}\" --tag \"{imageName}\" --progress plain";
}

var arguments = $"image build --file \"{dockerfilePath}\" --tag \"{imageName}\" --progress plain";
arguments += BuildArgumentsString(buildArguments);
arguments += BuildArgumentsString(buildArguments);
arguments += BuildSecretsString(buildSecrets);
arguments += BuildStageString(stage);
arguments += $" \"{Path.GetFullPath(contextPath).TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar)}\"";
[Theory]
[InlineData(true)]
[InlineData(false)]
[InlineData(false, Skip = "https://github.com/microsoft/WSL/issues/42129", SkipType = typeof(PlatformDetection), SkipWhen = nameof(PlatformDetection.IsWindows))]
Comment on lines +107 to +108
return OperatingSystem.IsLinux()
|| (OperatingSystem.IsWindows() && PlatformDetection.IsRunningOnGithubActions);
Comment on lines +131 to +133
return PlatformDetection.IsRunningOnCI
? OperatingSystem.IsLinux()
: IsOnPath(KnownContainerRuntimes.Docker) || IsOnPath(KnownContainerRuntimes.Podman);
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 8, 2026
## Description

`PostgresReplCommandTests.ReplExecutesAuthenticatedQuery` failed on main
in [run
42152057806](https://github.com/microsoft/aspire/actions/runs/42152057806)
([Hosting.PostgreSQL
job](https://github.com/microsoft/aspire/actions/runs/42152057806/job/127678069176)).
The shared REPL test base
(`tests/Shared/ContainerReplCommandTestBase.cs`) gave the resource 1
minute to become healthy. `StartAsync` returns before DCP pulls the
image, so that minute also had to cover a cold pull. The postgres pull
alone took about 55s in that run, which left too little time for the
container to start.

## Failure stats (14d, attempt 1)

`CI` workflow on microsoft/aspire, runs created 2026-10-25 06:28 →
2026-11-08 06:28 UTC, **attempt 1 only**. Denominator: attempt-1 jobs
that ran (success or failure). Hits: the failed job's log matches the
signature. Jobs that lost their runner, which have no log, are excluded.
Main runs have no reruns, so their flake check is the next main run of
the job.

| | |
|---|---|
| Signature | `PostgresReplCommandTests.ReplExecutesAuthenticatedQuery`
failed with `Resource 'postgres' failed to become healthy before the
operation was cancelled` |
| Job | Hosting.PostgreSQL (ubuntu + windows) |
| main pushes: hits / runs | 1 / 316 (0.32%) |
| main pushes, 30d: hits / runs | 1 / 546 (0.18%). The 16 days before
the 14d window add 0 hits. |
| PR runs: hits / runs | 0 / 921 (0.00%); 1 PR-caused excluded |
| Combined rate | 1 / 1237 = **0.08%** |
| Flake check | main: next main run of the job passed 1/1 |
| First / last hit | 2026-11-06 / 2026-11-06 |
| Still hitting main? | Last main hit: 2026-11-06. 44 main runs since
the last hit (2026-11-06), with 0 hits |
| Fix evidence | Local: FakeTimeProvider budget test 5/5 fail → 5/5
pass. Real Docker REPL suites pass (Postgres 24/24). This PR's CI:
Hosting.PostgreSQL ubuntu + windows passed 1/1. |

Failing attempt-1 jobs, most recent first (1 of 1 log-verified hits):

| Date (UTC) | Branch | Job | Signature excerpt |
|---|---|---|---|
| [2026-11-06
02:42](https://github.com/microsoft/aspire/actions/runs/42152057806/job/127678069176)
| **main** | Hosting.PostgreSQL (ubuntu-latest) | `Resource 'postgres'
failed to become healthy before the operation was cancelled` |

**Verdict: WEAK.** Only **1 non-PR-caused hit** (main, 2026-11-06). The
1 PR hit came from the in-flight WSLC-on-Windows PR and is excluded. The
other 7 failures of this test had different errors (`Terminal … has
already`, `CliPath` options validation, Polly timeout, `Stopped
waiting`). Across all shared-base REPL tests
(Postgres/Redis/Valkey/MongoDB/SqlServer), the same healthy-timeout hit
2/1580 main jobs, and 11 more PR hits were all on the WSLC PR. A timeout
increase is low risk, but the data does not show a recurring flake.

<!-- /failure-stats -->

The fix moves the readiness wait into a
`WaitForReplResourceHealthyAsync` helper that uses
`TestConstants.ExtraLongTimeoutTimeSpan` (180s locally, 360s on CI).
That constant already exists and is documented as sized for image pulls.
This raises a timeout to match a measured cold pull. It does not add
retries or sleeps, and the assertions are unchanged. The helper takes a
`TimeProvider` so the budget can be tested. The Redis, MongoDB and
SqlServer test projects now link `AsyncTestHelpers.cs`, because they
share the base class.

Other recent failures of this test came from in-flight PRs (for example
the WSLC Windows work in #21564) and are not addressed here.

Refs #21600

## Confidence

- New regression test `ReplReadinessWaitCoversColdImagePull`
(FakeTimeProvider, no Docker) moves the clock forward 2 minutes, then
reports the resource as failed to start. It expects the wait to fail for
that reason rather than be cancelled.
- Before the fix (1-minute budget): 5/5 runs failed with
`OperationCanceledException`.
  - After the fix: 5/5 passed.
- With real Docker: `PostgresReplCommandTests` 24/24,
`RedisReplCommandTests` 20/20 and `ValkeyReplCommandTests` 20/20 passed.
- All five affected test projects build.
- **Not run locally:** the MongoDB and SqlServer REPL container tests.
They failed on my machine with `No space left on device` during image
extraction. CI covers them.

**How we stop this recurring:** container readiness waits in shared test
bases should use `TestConstants.ExtraLongTimeoutTimeSpan`, which
accounts for image pulls, instead of hand-picked minute values.

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 10, 2026
## Description

`PostgresReplCommandTests.ReplExecutesAuthenticatedQuery` failed on main
in [run
42152057806](https://github.com/microsoft/aspire/actions/runs/42152057806)
([Hosting.PostgreSQL
job](https://github.com/microsoft/aspire/actions/runs/42152057806/job/127678069176)).
The shared REPL test base
(`tests/Shared/ContainerReplCommandTestBase.cs`) gave the resource 1
minute to become healthy. `StartAsync` returns before DCP pulls the
image, so that minute also had to cover a cold pull. The postgres pull
alone took about 55s in that run, which left too little time for the
container to start.

## Failure stats (14d, attempt 1)

`CI` workflow on microsoft/aspire, runs created 2026-10-25 06:28 →
2026-11-08 06:28 UTC, **attempt 1 only**. Denominator: attempt-1 jobs
that ran (success or failure). Hits: the failed job's log matches the
signature. Jobs that lost their runner, which have no log, are excluded.
Main runs have no reruns, so their flake check is the next main run of
the job.

| | |
|---|---|
| Signature | `PostgresReplCommandTests.ReplExecutesAuthenticatedQuery`
failed with `Resource 'postgres' failed to become healthy before the
operation was cancelled` |
| Job | Hosting.PostgreSQL (ubuntu + windows) |
| main pushes: hits / runs | 1 / 316 (0.32%) |
| main pushes, 30d: hits / runs | 1 / 546 (0.18%). The 16 days before
the 14d window add 0 hits. |
| PR runs: hits / runs | 0 / 921 (0.00%); 1 PR-caused excluded |
| Combined rate | 1 / 1237 = **0.08%** |
| Flake check | main: next main run of the job passed 1/1 |
| First / last hit | 2026-11-06 / 2026-11-06 |
| Still hitting main? | Last main hit: 2026-11-06. 44 main runs since
the last hit (2026-11-06), with 0 hits |
| Fix evidence | Local: FakeTimeProvider budget test 5/5 fail → 5/5
pass. Real Docker REPL suites pass (Postgres 24/24). This PR's CI:
Hosting.PostgreSQL ubuntu + windows passed 1/1. |

Failing attempt-1 jobs, most recent first (1 of 1 log-verified hits):

| Date (UTC) | Branch | Job | Signature excerpt |
|---|---|---|---|
| [2026-11-06
02:42](https://github.com/microsoft/aspire/actions/runs/42152057806/job/127678069176)
| **main** | Hosting.PostgreSQL (ubuntu-latest) | `Resource 'postgres'
failed to become healthy before the operation was cancelled` |

**Verdict: WEAK.** Only **1 non-PR-caused hit** (main, 2026-11-06). The
1 PR hit came from the in-flight WSLC-on-Windows PR and is excluded. The
other 7 failures of this test had different errors (`Terminal … has
already`, `CliPath` options validation, Polly timeout, `Stopped
waiting`). Across all shared-base REPL tests
(Postgres/Redis/Valkey/MongoDB/SqlServer), the same healthy-timeout hit
2/1580 main jobs, and 11 more PR hits were all on the WSLC PR. A timeout
increase is low risk, but the data does not show a recurring flake.

<!-- /failure-stats -->

The fix moves the readiness wait into a
`WaitForReplResourceHealthyAsync` helper that uses
`TestConstants.ExtraLongTimeoutTimeSpan` (180s locally, 360s on CI).
That constant already exists and is documented as sized for image pulls.
This raises a timeout to match a measured cold pull. It does not add
retries or sleeps, and the assertions are unchanged. The helper takes a
`TimeProvider` so the budget can be tested. The Redis, MongoDB and
SqlServer test projects now link `AsyncTestHelpers.cs`, because they
share the base class.

Other recent failures of this test came from in-flight PRs (for example
the WSLC Windows work in #21564) and are not addressed here.

Refs #21600

## Confidence

- New regression test `ReplReadinessWaitCoversColdImagePull`
(FakeTimeProvider, no Docker) moves the clock forward 2 minutes, then
reports the resource as failed to start. It expects the wait to fail for
that reason rather than be cancelled.
- Before the fix (1-minute budget): 5/5 runs failed with
`OperationCanceledException`.
  - After the fix: 5/5 passed.
- With real Docker: `PostgresReplCommandTests` 24/24,
`RedisReplCommandTests` 20/20 and `ValkeyReplCommandTests` 20/20 passed.
- All five affected test projects build.
- **Not run locally:** the MongoDB and SqlServer REPL container tests.
They failed on my machine with `No space left on device` during image
extraction. CI covers them.

**How we stop this recurring:** container readiness waits in shared test
bases should use `TestConstants.ExtraLongTimeoutTimeSpan`, which
accounts for image pulls, instead of hand-picked minute values.

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 9fcb379)
Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 10, 2026
## Description

`PostgresReplCommandTests.ReplExecutesAuthenticatedQuery` failed on main
in [run
42152057806](https://github.com/microsoft/aspire/actions/runs/42152057806)
([Hosting.PostgreSQL
job](https://github.com/microsoft/aspire/actions/runs/42152057806/job/127678069176)).
The shared REPL test base
(`tests/Shared/ContainerReplCommandTestBase.cs`) gave the resource 1
minute to become healthy. `StartAsync` returns before DCP pulls the
image, so that minute also had to cover a cold pull. The postgres pull
alone took about 55s in that run, which left too little time for the
container to start.

## Failure stats (14d, attempt 1)

`CI` workflow on microsoft/aspire, runs created 2026-10-25 06:28 →
2026-11-08 06:28 UTC, **attempt 1 only**. Denominator: attempt-1 jobs
that ran (success or failure). Hits: the failed job's log matches the
signature. Jobs that lost their runner, which have no log, are excluded.
Main runs have no reruns, so their flake check is the next main run of
the job.

| | |
|---|---|
| Signature | `PostgresReplCommandTests.ReplExecutesAuthenticatedQuery`
failed with `Resource 'postgres' failed to become healthy before the
operation was cancelled` |
| Job | Hosting.PostgreSQL (ubuntu + windows) |
| main pushes: hits / runs | 1 / 316 (0.32%) |
| main pushes, 30d: hits / runs | 1 / 546 (0.18%). The 16 days before
the 14d window add 0 hits. |
| PR runs: hits / runs | 0 / 921 (0.00%); 1 PR-caused excluded |
| Combined rate | 1 / 1237 = **0.08%** |
| Flake check | main: next main run of the job passed 1/1 |
| First / last hit | 2026-11-06 / 2026-11-06 |
| Still hitting main? | Last main hit: 2026-11-06. 44 main runs since
the last hit (2026-11-06), with 0 hits |
| Fix evidence | Local: FakeTimeProvider budget test 5/5 fail → 5/5
pass. Real Docker REPL suites pass (Postgres 24/24). This PR's CI:
Hosting.PostgreSQL ubuntu + windows passed 1/1. |

Failing attempt-1 jobs, most recent first (1 of 1 log-verified hits):

| Date (UTC) | Branch | Job | Signature excerpt |
|---|---|---|---|
| [2026-11-06
02:42](https://github.com/microsoft/aspire/actions/runs/42152057806/job/127678069176)
| **main** | Hosting.PostgreSQL (ubuntu-latest) | `Resource 'postgres'
failed to become healthy before the operation was cancelled` |

**Verdict: WEAK.** Only **1 non-PR-caused hit** (main, 2026-11-06). The
1 PR hit came from the in-flight WSLC-on-Windows PR and is excluded. The
other 7 failures of this test had different errors (`Terminal … has
already`, `CliPath` options validation, Polly timeout, `Stopped
waiting`). Across all shared-base REPL tests
(Postgres/Redis/Valkey/MongoDB/SqlServer), the same healthy-timeout hit
2/1580 main jobs, and 11 more PR hits were all on the WSLC PR. A timeout
increase is low risk, but the data does not show a recurring flake.

<!-- /failure-stats -->

The fix moves the readiness wait into a
`WaitForReplResourceHealthyAsync` helper that uses
`TestConstants.ExtraLongTimeoutTimeSpan` (180s locally, 360s on CI).
That constant already exists and is documented as sized for image pulls.
This raises a timeout to match a measured cold pull. It does not add
retries or sleeps, and the assertions are unchanged. The helper takes a
`TimeProvider` so the budget can be tested. The Redis, MongoDB and
SqlServer test projects now link `AsyncTestHelpers.cs`, because they
share the base class.

Other recent failures of this test came from in-flight PRs (for example
the WSLC Windows work in #21564) and are not addressed here.

Refs #21600

## Confidence

- New regression test `ReplReadinessWaitCoversColdImagePull`
(FakeTimeProvider, no Docker) moves the clock forward 2 minutes, then
reports the resource as failed to start. It expects the wait to fail for
that reason rather than be cancelled.
- Before the fix (1-minute budget): 5/5 runs failed with
`OperationCanceledException`.
  - After the fix: 5/5 passed.
- With real Docker: `PostgresReplCommandTests` 24/24,
`RedisReplCommandTests` 20/20 and `ValkeyReplCommandTests` 20/20 passed.
- All five affected test projects build.
- **Not run locally:** the MongoDB and SqlServer REPL container tests.
They failed on my machine with `No space left on device` during image
extraction. CI covers them.

**How we stop this recurring:** container readiness waits in shared test
bases should use `TestConstants.ExtraLongTimeoutTimeSpan`, which
accounts for image pulls, instead of hand-picked minute values.

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 9fcb379)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-engineering-systems infrastructure helix infra engineering repo stuff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants