Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/cli-e2e-testing/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ ASPIRE_E2E_ARCHIVE=/tmp/aspire-localrelease.tar.gz \

These tests **skip** unless the CLI was installed from a `LocalHive` archive *and* that archive is
stable-shaped (no prerelease suffix). In default CI the archive is a prerelease `LocalArchive`, so they
skip and add zero CI cost — CI relies on `--ignore-exit-code 8` (set in `eng/Testing.props`
skip and add zero CI cost — CI relies on `--ignore-exit-code 8` (set in `eng/Testing.targets`
`MtpBaseArgs`) so an all-skipped class job still passes. The test also registers the hive as an
**ambient** NuGet source (`dotnet nuget add source`) because MSBuild resolves the apphost's
`Aspire.AppHost.Sdk` before restore from nuget.config sources only, ignoring `ASPIRE_CLI_PACKAGES`.
Expand Down
4 changes: 2 additions & 2 deletions .agents/skills/fix-flaky-test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -958,12 +958,12 @@ UPDATE todos SET status = 'done' WHERE id = 'cleanup';

### Build System Quarantine Filtering

`eng/Testing.props` auto-appends `--filter-not-trait "quarantined=true"` to test arguments via the `TestRunnerAdditionalArguments` MSBuild property. This property is evaluated during `dotnet test` even with `--no-build`, so it must be handled in both build and test commands:
`eng/Testing.targets` auto-appends `--filter-not-trait "quarantined=true"` to test arguments via the `TestRunnerAdditionalArguments` MSBuild property. The filter value is defined in `eng/Testing.props`, and the composed property is evaluated during `dotnet test` even with `--no-build`, so it must be handled in both build and test commands:

- **Reproduce workflow**: Overrides `_NonQuarantinedTestRunAdditionalArgs` to empty, removing the quarantine exclusion filter for all tests
- **Local reproduction**: Pass `/p:RunQuarantinedTests=true` to both `dotnet build` and `dotnet test`

`Testing.props` also adds `--ignore-exit-code 8`, which masks zero-test runs as successes. The workflow and `run-test-repeatedly` scripts detect this by checking test output for the `Total:` count indicator.
`Testing.targets` also adds `--ignore-exit-code 8` via `MtpBaseArgs`, which masks zero-test runs as successes. The workflow and `run-test-repeatedly` scripts detect this by checking test output for the `Total:` count indicator.

### test-reproduce.yml Architecture

Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/fix-flaky-test/run-test-repeatedly.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ for ($iter = 1; $iter -le $Iterations; $iter++) {
Remove-Item $stderrFile -ErrorAction SilentlyContinue
}

# Detect zero-test runs (exit code 8 is masked by --ignore-exit-code 8 in Testing.props)
# Detect zero-test runs (exit code 8 is masked by --ignore-exit-code 8 in Testing.targets)
if ($exitCode -eq 0) {
$iterContent = Get-Content $iterLog -Raw -ErrorAction SilentlyContinue
if (-not ($iterContent -match 'Total:\s*[1-9]')) {
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/fix-flaky-test/run-test-repeatedly.sh
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ for i in $(seq 1 "$ITERATIONS"); do
run_with_timeout 180 "${TEST_CMD[@]}" >> "$ITER_LOG" 2>&1
EXIT_CODE=$?

# Detect zero-test runs (exit code 8 is masked by --ignore-exit-code 8 in Testing.props)
# Detect zero-test runs (exit code 8 is masked by --ignore-exit-code 8 in Testing.targets)
# Match case-insensitively: MTP's native runner emits a lowercase "total: N" summary,
# while VSTest emits "Total: N". A case-sensitive match misses MTP and mis-flags every
# passing MTP run as a zero-test failure.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deployment-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -583,7 +583,7 @@ jobs:
# --ignore-exit-code 8: MTP returns exit code 8 ("zero tests ran") when the single
# matrix test is dynamically skipped (e.g. on a transient Azure regional-capacity error),
# which would otherwise be treated as a failure below. Treating 8 as success — the repo
# convention baked into MtpBaseArgs (eng/Testing.props) — keeps a skipped deploy green.
# convention baked into MtpBaseArgs (eng/Testing.targets) — keeps a skipped deploy green.
./dotnet.sh test --project tests/Aspire.Deployment.EndToEnd.Tests/Aspire.Deployment.EndToEnd.Tests.csproj \
-c Release \
--results-directory ${{ github.workspace }}/testresults \
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/reproduce-flaky-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ jobs:
RESULT=${PIPESTATUS[0]}
set -e

# Detect zero-test runs (--ignore-exit-code 8 in Testing.props masks exit code 8)
# Detect zero-test runs (--ignore-exit-code 8 in Testing.targets masks exit code 8)
if [ "$RESULT" -eq 0 ]; then
if ! grep -qiE "total:\s*[1-9]" "$ITER_OUTPUT" 2>/dev/null; then
echo "::error::Iteration $i: Zero tests executed. Verify TEST_FILTER and quarantine settings."
Expand Down Expand Up @@ -346,7 +346,7 @@ jobs:
$result = $LASTEXITCODE
Get-Content $iterOutput

# Detect zero-test runs (--ignore-exit-code 8 in Testing.props masks exit code 8)
# Detect zero-test runs (--ignore-exit-code 8 in Testing.targets masks exit code 8)
if ($result -eq 0) {
$content = Get-Content $iterOutput -Raw -ErrorAction SilentlyContinue
if (-not ($content -match '(?i)total:\s*[1-9]')) {
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:
required: false
type: string
# MTP diagnostic arguments (hangdump, crashdump, timeouts).
# Single source of truth is eng/Testing.props; the runsheet pipeline forwards per-project values.
# Single source of truth is eng/Testing.targets; the runsheet pipeline forwards per-project values.
mtpBaseArgs:
required: false
type: string
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ kill <pid>
* Do not leave newly-added tests commented out. All added tests should be building and passing.
* Do not use Directory.SetCurrentDirectory in tests as it can cause side effects when tests execute concurrently.
* Prefer using shared test service implementations (e.g., project-level `TestServices/` or `Helpers/` directories, or the cross-project `tests/Shared/` folder) rather than creating private implementation classes within individual test files. Reusing existing test fakes and helpers keeps tests consistent, reduces duplication, and makes maintenance easier. Do not create private test classes when a shared one already exists or can be extended.
* MTP diagnostic args (hang dump, crash dump, exit code handling) are defined in `eng/Testing.props` via `MtpBaseArgs`. Do not hardcode these args in workflow YAML. See [docs/ci/mtp-args-pipeline.md](docs/ci/mtp-args-pipeline.md) for details.
* MTP diagnostic args (hang dump, crash dump, exit code handling) are defined in `eng/Testing.targets` via `MtpBaseArgs`. Do not hardcode these args in workflow YAML. See [docs/ci/mtp-args-pipeline.md](docs/ci/mtp-args-pipeline.md) for details.
* Use `Verify` (snapshot testing) for generated artifacts (files, serialized output, structured text). Prefer `await Verify(value, "ext")` over `Assert.Contains` / `Assert.DoesNotContain` / `Assert.Equal` on the same value. Run the test once to generate the `.received.` file, review it, then rename it to `.verified.` to accept it.
* Avoid `Assert.DoesNotContain` as it is a weak assertion that easily goes out of date — it only proves something is absent without verifying what *is* present. Prefer `Assert.Equal` to check the entire string value, or `Assert.Collection` to verify the complete contents of a collection.

Expand Down
13 changes: 8 additions & 5 deletions docs/ci/mtp-args-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ MTP diagnostic arguments (hang dump, crash dump, exit code handling, timeouts) a
┌───────────────────────────────────────────────────────────────────────────┐
│ MSBuild (build time) │
│ │
│ eng/Testing.props │
│ eng/Testing.targets │
│ MtpBaseArgs = "<diagnostic flags> --hangdump-timeout <T> --timeout <S>"│
│ _BlameArgs = MtpBaseArgs │
│ TestRunnerAdditionalArguments = ... + _BlameArgs (Arcade/Helix path) │
Expand Down Expand Up @@ -42,10 +42,13 @@ MTP diagnostic arguments (hang dump, crash dump, exit code handling, timeouts) a

## Properties

### `MtpBaseArgs` (defined in `eng/Testing.props`)
### `MtpBaseArgs` (defined in `eng/Testing.targets`)

The single source of truth for all MTP diagnostic and timeout arguments. Contains everything that should be passed to every test execution. Timeouts are baked into this property so they flow as one resolved string.

The property is composed in a targets file, after project evaluation, so timeout
overrides declared in an individual test project are reflected in the final arguments.

**Default value:** `--ignore-exit-code 8 --crashdump --hangdump --hangdump-type none --hangdump-timeout 10m --timeout 20m`

| Arg | Purpose |
Expand Down Expand Up @@ -87,7 +90,7 @@ and TRX reporting options. xUnit v4 uses `--report-xunit-xml` and
`--report-xunit` options fail argument parsing before any tests execute.
`MicrosoftTestingPlatformTests` checks these options against the current test host.

Assembled in `eng/Testing.props` from:
Assembled in `eng/Testing.targets` from:
- `--filter-not-trait "category=failing"`
- Filter args (quarantine/outerloop exclusions)
- `_BlameArgs` (which equals `MtpBaseArgs` — diagnostic flags + timeouts)
Expand All @@ -106,7 +109,7 @@ Set in:

## How to add a new MTP diagnostic arg

1. Edit `eng/Testing.props` — append the arg to `MtpBaseArgs` (or to `_MtpDiagnosticFlags` if it's timeout-independent)
1. Edit `eng/Testing.targets` — append the arg to `MtpBaseArgs` (or to `_MtpDiagnosticFlags` if it's timeout-independent)
2. That's it. The arg flows automatically through both paths:
- **Arcade/Helix**: via `_BlameArgs` → `TestRunnerAdditionalArguments`
- **GitHub Actions**: via runsheet → `run-tests.yml` `mtpBaseArgs` input
Expand Down Expand Up @@ -140,4 +143,4 @@ MTP MSBuild integration injects `TestingPlatformCommandLineArguments` automatica

## Backward compatibility

The `run-tests.yml` `mtpBaseArgs` input has a default value that includes diagnostic flags (crashdump, hangdump, exit-code handling) but does not include timeout arguments. Timeout values are baked into `mtpBaseArgs` at build time by MSBuild (via `eng/Testing.props`) and flow through the test matrix metadata. Callers that bypass the matrix and don't pass `mtpBaseArgs` should include the timeout arguments explicitly if needed.
The `run-tests.yml` `mtpBaseArgs` input has a default value that includes diagnostic flags (crashdump, hangdump, exit-code handling) but does not include timeout arguments. Timeout values are baked into `mtpBaseArgs` at build time by MSBuild (via `eng/Testing.targets`) and flow through the test matrix metadata. Callers that bypass the matrix and don't pass `mtpBaseArgs` should include the timeout arguments explicitly if needed.
2 changes: 1 addition & 1 deletion docs/specs/cli-identity-sidecar.md
Original file line number Diff line number Diff line change
Expand Up @@ -466,4 +466,4 @@ The resolver and call-site migration land together; the test plan reflects that.

- `EmulatedReleasedBuildTests` (C# + TS): emulate the latest **shipped stable**; assert no `NuGet.config` is dropped (stable packages live on nuget.org) and the AppHost SDK is pinned stable-shaped.
- `EmulatedStagingBuildTests` (C# + TS): emulate the latest **staging** build (discovered darc commit); assert a `NuGet.config` mapping `Aspire*` to the SHA-specific darc feed is dropped and `aspire add` resolves the staging version.
- `EmulatedLocalReleaseBuildTests` (C# + TS): the **all-local "future release"** row. Build a stable-shaped archive with `localhive --version <X.Y.Z> --archive`, emulate `stable` with `ASPIRE_CLI_PACKAGES` pointed at the extracted hive, and assert `aspire new`/`aspire add` resolve the future-only version (e.g. `13.5.0`) **from the local hive** — proving the resolution fix (override honored under any emulated channel name). Registers the hive as an ambient NuGet source for buildability. Skips unless the CLI was installed from a stable-shaped `LocalHive` archive, so it adds zero default-CI cost (relying on `--ignore-exit-code 8` in `eng/Testing.props` for the all-skipped class job).
- `EmulatedLocalReleaseBuildTests` (C# + TS): the **all-local "future release"** row. Build a stable-shaped archive with `localhive --version <X.Y.Z> --archive`, emulate `stable` with `ASPIRE_CLI_PACKAGES` pointed at the extracted hive, and assert `aspire new`/`aspire add` resolve the future-only version (e.g. `13.5.0`) **from the local hive** — proving the resolution fix (override honored under any emulated channel name). Registers the hive as an ambient NuGet source for buildability. Skips unless the CLI was installed from a stable-shaped `LocalHive` archive, so it adds zero default-CI cost (relying on `--ignore-exit-code 8` in `eng/Testing.targets` for the all-skipped class job).
22 changes: 0 additions & 22 deletions eng/Testing.props
Original file line number Diff line number Diff line change
Expand Up @@ -31,26 +31,4 @@
<_ShortName Condition="'$(_ShortName)' == ''">$([System.IO.Path]::GetFileNameWithoutExtension('$(MSBuildProjectName)').Replace('Aspire.', '').Replace('.Tests', ''))</_ShortName>
</PropertyGroup>

<PropertyGroup>
<!--
MtpBaseArgs: The single source of truth for MTP diagnostic arguments, including
per-project timeouts. Shared by the Arcade/Helix path (TestRunnerAdditionalArguments)
and the GitHub Actions path (run-tests.yml via runsheet pipeline).
See docs/ci/mtp-args-pipeline.md for details.
-->
<_MtpDiagnosticFlags>--ignore-exit-code 8 --crashdump --hangdump --hangdump-type none</_MtpDiagnosticFlags>
<MtpBaseArgs Condition="'$(MtpBaseArgs)' == ''">$(_MtpDiagnosticFlags) --hangdump-timeout $(TestHangTimeout) --timeout $(TestSessionTimeout)</MtpBaseArgs>

<!-- Variant with uncollected-bucket timeouts, used for split-test entries that don't match a named collection -->
<_UncollectedMtpBaseArgs>$(_MtpDiagnosticFlags) --hangdump-timeout $(UncollectedTestsHangTimeout) --timeout $(UncollectedTestsSessionTimeout)</_UncollectedMtpBaseArgs>

<_BlameArgs>$(MtpBaseArgs)</_BlameArgs>

<TestRunnerAdditionalArguments>$(TestRunnerAdditionalArguments) --filter-not-trait &quot;category=failing&quot;</TestRunnerAdditionalArguments>
<!-- ignore exit code 8 (zero tests ran) is already in MtpBaseArgs / _BlameArgs -->

<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' != 'true' and '$(RunOuterloopTests)' != 'true'">$(TestRunnerAdditionalArguments) $(_NonQuarantinedTestRunAdditionalArgs) $(_NonOuterloopTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' == 'true'">$(TestRunnerAdditionalArguments) $(_QuarantinedTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
<TestRunnerAdditionalArguments Condition="'$(RunOuterloopTests)' == 'true'">$(TestRunnerAdditionalArguments) $(_OuterloopTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
</PropertyGroup>
</Project>
20 changes: 20 additions & 0 deletions eng/Testing.targets
Original file line number Diff line number Diff line change
@@ -1,4 +1,24 @@
<Project>
<PropertyGroup>
<!--
Compose these arguments after project evaluation so test projects can override
their timeout properties in the project file.
-->
<_MtpDiagnosticFlags>--ignore-exit-code 8 --crashdump --hangdump --hangdump-type none</_MtpDiagnosticFlags>
<MtpBaseArgs Condition="'$(MtpBaseArgs)' == ''">$(_MtpDiagnosticFlags) --hangdump-timeout $(TestHangTimeout) --timeout $(TestSessionTimeout)</MtpBaseArgs>

<!-- Variant with uncollected-bucket timeouts, used for split-test entries that don't match a named collection -->
<_UncollectedMtpBaseArgs>$(_MtpDiagnosticFlags) --hangdump-timeout $(UncollectedTestsHangTimeout) --timeout $(UncollectedTestsSessionTimeout)</_UncollectedMtpBaseArgs>

<_BlameArgs>$(MtpBaseArgs)</_BlameArgs>

<TestRunnerAdditionalArguments>$(TestRunnerAdditionalArguments) --filter-not-trait &quot;category=failing&quot;</TestRunnerAdditionalArguments>
<!-- ignore exit code 8 (zero tests ran) is already in MtpBaseArgs / _BlameArgs -->

<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' != 'true' and '$(RunOuterloopTests)' != 'true'">$(TestRunnerAdditionalArguments) $(_NonQuarantinedTestRunAdditionalArgs) $(_NonOuterloopTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
<TestRunnerAdditionalArguments Condition="'$(RunQuarantinedTests)' == 'true'">$(TestRunnerAdditionalArguments) $(_QuarantinedTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
<TestRunnerAdditionalArguments Condition="'$(RunOuterloopTests)' == 'true'">$(TestRunnerAdditionalArguments) $(_OuterloopTestRunAdditionalArgs) $(_BlameArgs)</TestRunnerAdditionalArguments>
</PropertyGroup>

<!--
Environment variables:
Expand Down
2 changes: 1 addition & 1 deletion eng/pipelines/azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -419,7 +419,7 @@ extends:
jobs:
- job: TemplateTests
displayName: Template Tests
timeoutInMinutes: 30
timeoutInMinutes: 45
pool:
name: NetCore1ESPool-Internal
image: windows.vs2026preview.scout.amd64
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

<!-- Timeout configuration -->
<TestSessionTimeout>20m</TestSessionTimeout>
<TestSessionTimeout Condition="'$(RunOnlyBasicBuildTemplateTests)' == 'true'">30m</TestSessionTimeout>
<TestHangTimeout>15m</TestHangTimeout>

<!--
Expand Down
2 changes: 2 additions & 0 deletions tests/Aspire.Templates.Tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ Each class runs 51 cases across the SDK/target-framework matrix. Keeping them se
lets CI shard all 153 cases without tripling a single job's runtime. The framework
tests check generated package names, versions, and executable output where required
before building, so update these expectations when changing starter test dependencies.
The internal pipeline runs all `basic-build` cases in one process, so that invocation
uses a 30-minute session timeout while the class-sharded runs retain the 20-minute timeout.

- The sdk+workload is never updated automatically. In other words, once installed the workload packs don't get overwritten even when the source binaries changes in `artifacts`. This may change in future.

Expand Down
Loading
Loading