Repository navigation
Description
[automated] This tracks automatic recovery for failed GitHub main CI runs. Measurement, evaluation duration, and health-report changes will be managed separately outside this issue.
Is there an existing issue for this?
- I have searched the existing issues
Is your feature request related to a problem? Please describe the problem.
Pull request CI automatically reruns failed jobs up to three times, but failed GitHub push runs on main do not have the same recovery path. A transient test, runner, network, container, or service failure can therefore leave main red until another commit happens to trigger a successful run.
CI runs use one concurrency group per ref and do not cancel an in-progress main run. A retry must be suppressed when its SHA has already been superseded; otherwise a stale retry can consume the pending slot or delay validation of newer code.
Describe the solution you'd like
Give failed GitHub main CI runs the same bounded retry cadence as pull requests:
- Rerun failed jobs unconditionally, up to three reruns/four total attempts.
- Immediately before requesting a rerun, verify that the source run SHA is still the current
mainSHA, no newermainCI run supersedes it, the source attempt has not advanced, and the attempt remains within the cap. - Reuse the existing failed-job rerun infrastructure rather than introduce a second retry implementation.
- Preserve the existing
mainconcurrency behavior. Reruns retain the originalGITHUB_REF, so they remain in theCI-refs/heads/mainconcurrency group; a newer build waits behind an already-started retry. - Keep the event-driven
Analyze CI Failureworkflow andmemory/ci-failure-analysisbranch as the durable failure/cause history. Report rerun decisions and skip reasons in the retry workflow's logs and summary. - Continue creating or updating failure-cause issues. Do not assign or delegate remediation work automatically.
Additional context
This work should be tracked under #18036. Durable retry-decision history for external reporting is tracked separately in #20409.
Out of scope:
- Azure DevOps/internal official-build retries;
- CI Shepherd lifecycle or assignment behavior;
- automatic Copilot delegation or PR shepherding;
- public CI-health report changes;
- a new metrics storage service or dashboard.
Acceptance criteria:
- Current-
mainfailed jobs are rerun automatically up to the configured attempt cap. - A superseded SHA cannot request a stale rerun.
- A stale analysis cannot request an additional attempt after the run has advanced.
- Existing
mainconcurrency behavior is preserved and covered by focused tests. - Rerun decisions and skip reasons are reported in the retry workflow's logs and summary.
Activity
joperezr commentedon Sep 23, 2026
🤖 Aspire triage pass — 2026-09-23 02:01 UTC
Area:
area-engineering-systemsKind: feature-request
Repro: none — the issue defines a missing CI recovery path and concrete acceptance criteria
Affected version(s): repository CI infrastructure (latest Aspire release: 13.5.4)
Severity: S3 — transient failures can leave the main branch red until a later successful run
Urgency: 🟡 vNext — recommend milestone 13.6
Reason: concrete, actionable CI reliability work for the active release train; it does not require servicing
Suggested owner(s): James Newton-King (@JamesNK) (score 54), backup Adam Ratzman (@adamint) (score 40)
Evidence:
area-engineering-systemsaspire doctortaking 15 secs to run #19016, Use resolveddotnetpaths inaspire doctor#19003, [13.5] Trace logging onaspire doctorcauses weird indentation #19017 as completed inarea-engineering-systemsarea-engineering-systemswithin the last 30 daysConfidence: medium
Needs human?: no
fix(ci): Run current-main retries in the CI auto-rerun workflow
fix(ci): File main failure issues after the final retry
fix(ci): Retry failed current-main jobs through auto-rerun
fix(ci): File main breakage issues only after the final retry
fix(ci): Revalidate rerun attempts before execution