docs(problems): add error recovery and retry mechanics to cross-run-memory

[Benkapner]

Summary

• Adds a section to cross-run-memory.md covering the immediate question of what happens when an agent fails, before any learning occurs
• Distinguishes retry budgets (total attempts and cost across runs) from per-run limits (max_turns, max_cost_usd already enforced in feat: add functional test framework for agent pipelines #1682)
• Describes escalation thresholds: after N identical failures, after budget exhaustion, on novel failure types, on regression
• Explains the interaction between error recovery and cross-run memory: recovery determines what to do now, memory records what happened for future runs

The cross-run-memory doc addresses how agents learn from prior failures. This addition addresses the complementary question: what does the system do immediately after a failure? The two concerns are related but distinct: memory is about learning, recovery is about acting.

Links to flapping-convergence.md (when merged) for the systemic convergence question.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

docs(problems): document error recovery, retry budgets, and escalation thresholds

📝 Documentation 🕐 10-20 Minutes

AI Description

• Document how the system should respond immediately after agent failures (retry vs. escalate).
• Define retry budgets across runs, distinct from per-run max_turns/max_cost_usd limits.
• Outline escalation thresholds and how recovery signals interact with cross-run memory.

Diagram

graph TD
  CRM[["cross-run-memory.md"]] --> ERR["Error recovery"] --> RETRY["Retry vs escalate"]
  ERR --> BUD["Retry budgets"] --> LIMITS["Per-run limits"]
  ERR --> ESC["Escalation thresholds"]
  ERR --> MEM["Recovery vs memory"]
  CRM --> FLAP[["flapping-convergence.md"]]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Create a dedicated error-recovery doc and link from cross-run-memory

• ➕ Keeps cross-run-memory focused strictly on learning/feedback mechanisms
• ➕ Makes it easier to expand recovery into a fuller spec (budgets, policies, runbooks) without bloating the memory doc
• ➖ Adds navigation overhead (readers must hop between docs)
• ➖ Requires deciding where shared concepts (budgets/escalation) live long-term

2. Add a concrete decision table/flowchart for escalation thresholds

• ➕ Reduces ambiguity for implementers and operators
• ➕ Makes “N identical failures / novel failure types / regression” actionable and testable
• ➖ Harder to keep accurate as policy evolves
• ➖ May imply stronger commitments than intended for a problems/ideas doc

Recommendation: Current approach (a clearly scoped section inside cross-run-memory that separates recovery vs. learning) is a good fit for answering the immediate “what happens now?” question while preserving the doc’s core framing. Consider splitting into a standalone error-recovery doc only if this section grows into a policy/spec with operational details or implementation requirements.

Files changed (1) +43 / -0

Documentation (1) +43 / -0

cross-run-memory.mdAdd error recovery, retry budgets, and escalation thresholds section +43/-0

Add error recovery, retry budgets, and escalation thresholds section

• Introduces a new section that distinguishes immediate error recovery decisions from cross-run learning. Defines retry vs. escalate framing (transient vs. structural failures), formalizes cross-run retry budgets distinct from per-run limits, and lists escalation thresholds including regression and novel failure types. Clarifies how recovery signals should (and should not) feed into cross-run memory and links to flapping/convergence guidance.

docs/problems/cross-run-memory.md

[github-actions]

Site preview

Preview: https://2ddd7887-site.fullsend-ai.workers.dev

Commit: 4a04f137169ba69a063bcc6d3bc9f429ea8db080

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. Broken flapping link ✗ Dismissed 🐞 Bug ≡ Correctness

Description

The new section links to flapping-convergence.md, but that file is not present under
docs/problems/, so the reference is broken for readers. This creates a dead-end exactly where the
doc suggests readers go for circuit-breaking details.

Code

docs/problems/cross-run-memory.md[R120-122]

+### Relationship to flapping
+
+Retry loops can become flapping when the system does not converge. See [flapping-convergence.md](flapping-convergence.md) for detection and circuit-breaking mechanisms. Error recovery addresses the single-task question (when to retry vs. escalate); flapping addresses the systemic question (when to stop the system from oscillating).

Relevance

⭐⭐⭐ High

Broken-doc-link fixes are routinely accepted; repo added markdown link linter and fixed many links
in PR #783.

PR-#783
PR-#601
PR-#226

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The link is introduced in cross-run-memory.md, but there is no corresponding
docs/problems/flapping-convergence.md file in the repo, so the link target cannot resolve.

docs/problems/cross-run-memory.md[120-122]
PR-#601

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/problems/cross-run-memory.md` links to `flapping-convergence.md`, but the file does not exist in the repository, resulting in a broken internal documentation link.

## Issue Context
The new “Relationship to flapping” section instructs readers to click through for detection/circuit-breaking mechanisms; currently that click target is missing.

## Fix Focus Areas
- docs/problems/cross-run-memory.md[120-122]

## Suggested fix options
1) Add the missing `docs/problems/flapping-convergence.md` file in this PR (preferred if content is ready).
2) If the doc will land later, change the link to plain text (no hyperlink) and optionally reference the PR/commit where it will be added.
3) If the doc exists elsewhere, update the relative path to the correct location.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Ambiguous retry-budget wording ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The sentence "loop indefinitely within their per-run limits" is ambiguous because per-run limits
(max_turns, max_cost_usd) are explicitly finite; the intended meaning is indefinite retries
across runs without a cumulative budget. This wording blurs the distinction the section is trying to
establish between per-run limits and cross-run retry budgets.

Code

docs/problems/cross-run-memory.md[R93-96]

+Without an explicit retry budget, agents can loop indefinitely within their per-run limits. A retry budget is distinct from per-run limits:
+
+- **Per-run limit:** max turns and max cost for a single attempt (already enforced via `max_turns` and `max_cost_usd` in the functional test framework, PR #1682)
+- **Retry budget:** max total attempts across runs for the same task, and max total cost across all attempts

Relevance

⭐⭐ Medium

No direct precedent on this exact ambiguity; team sometimes accepts doc clarity edits (e.g., PR
#770) but rejects many wording nits.

PR-#770
PR-#2009
PR-#7

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new text claims indefinite looping “within per-run limits,” while other docs define per-run
limits as specific finite thresholds (max_turns, max_cost_usd) for a single attempt.

docs/problems/cross-run-memory.md[91-98]
docs/testing/functional-tests.md[146-156]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The doc currently says agents can “loop indefinitely within their per-run limits,” which reads ambiguously given that per-run limits are bounded.

## Issue Context
This section is making a useful distinction between per-run limits and cross-run retry budgets; tightening the wording will prevent reader confusion.

## Fix Focus Areas
- docs/problems/cross-run-memory.md[93-98]

## Suggested rewrite
Replace:
- “Without an explicit retry budget, agents can loop indefinitely within their per-run limits.”

With something like:
- “Without an explicit retry budget, agents can keep retrying across runs indefinitely, even though each individual run is bounded by per-run limits.”

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Single recovery approach presented 📘 Rule violation ⌂ Architecture

Description

The new Error recovery and retry mechanics section describes a single implied approach (retry
budgets + escalation thresholds) without presenting at least two distinct options and clearly
labeled trade-offs. This fails the requirement for problem docs to document alternatives with
pros/cons rather than a single prescribed path.

Code

docs/problems/cross-run-memory.md[R81-122]

+## Error recovery and retry mechanics
+
+Cross-run memory is one piece of a broader question: what happens when an agent fails? The memory problem asks how agents learn from prior failures. The recovery problem asks what the system does immediately after a failure, before any learning occurs.
+
+### Retry vs. escalate
+
+When a run fails (CI failure, merge conflict, model refusal, timeout), the system must decide: retry the same approach, try a different approach, or escalate to a human. Today this decision is implicit. Agents retry until they hit a turn or cost limit, then stop. There is no structured decision about whether retrying is likely to produce a different outcome.
+
+The key distinction: **transient failures** (network timeout, flaky test, rate limit) benefit from retry. **Structural failures** (wrong approach, missing context, incompatible change) do not. Retrying a structural failure wastes resources and can make things worse if each attempt introduces new problems.
+
+### Retry budgets
+
+Without an explicit retry budget, agents can loop indefinitely within their per-run limits. A retry budget is distinct from per-run limits:
+
+- **Per-run limit:** max turns and max cost for a single attempt (already enforced via `max_turns` and `max_cost_usd` in the functional test framework, PR #1682)
+- **Retry budget:** max total attempts across runs for the same task, and max total cost across all attempts
+
+A task might allow 3 retries with a total budget of $10. Each individual run stays within its per-run limits, but the system tracks cumulative spend and attempt count.
+
+### Escalation thresholds
+
+When should the system stop retrying and ask a human?
+
+- **After N identical failures.** If the same error occurs on consecutive attempts, retrying is unlikely to help. Escalate with context: "failed 3 times with the same lint error on line 42."
+- **After budget exhaustion.** When the retry budget is consumed, escalate regardless of failure type.
+- **On novel failure types.** If each retry fails for a different reason, the task may be beyond the agent's current capability. Escalate after 2-3 distinct failure types.
+- **On regression.** If a retry makes things worse (introduces new test failures that the previous attempt did not have), stop immediately.
+
+### Interaction with cross-run memory
+
+Error recovery and memory are complementary:
+
+- Memory records *what happened* for future runs to learn from.
+- Recovery determines *what to do right now* in response to a failure.
+- A retry that succeeds generates a positive memory signal ("the second attempt with approach B worked after approach A failed").
+- A retry that fails generates a negative signal ("this task has failed 3 times; do not auto-assign to agents until the underlying issue is resolved").
+
+The risk: if recovery decisions feed into memory without review, a poorly-chosen retry strategy can poison future run context. For example, recording "approach B works" when it only worked due to a transient condition creates a false lesson.
+
+### Relationship to flapping
+
+Retry loops can become flapping when the system does not converge. See [flapping-convergence.md](flapping-convergence.md) for detection and circuit-breaking mechanisms. Error recovery addresses the single-task question (when to retry vs. escalate); flapping addresses the systemic question (when to stop the system from oscillating).

Relevance

⭐ Low

Similar “must present two options/trade-offs” restructure was rejected in PR #2549; requirement not
consistently enforced.

PR-#2549
PR-#2009

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1062035 requires at least two distinct options with clearly labeled trade-offs in
modified files under docs/problems/. The added section introduces retry budgets and escalation
thresholds but does not provide multiple alternative approaches with pros/cons.

Rule 1062035: Problem docs must present multiple options with trade-offs, not a single prescribed solution
docs/problems/cross-run-memory.md[81-122]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/problems/cross-run-memory.md` adds an error recovery section that frames one approach (retry budgets + escalation thresholds) but does not document at least two distinct solution options with explicit trade-offs.

## Issue Context
Problem documents under `docs/problems/` must present multiple options/approaches and include clearly labeled trade-offs (pros/cons/risks) for each; the document may recommend one option, but alternatives must still be analyzed.

## Fix Focus Areas
- docs/problems/cross-run-memory.md[81-122]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[Benkapner]

about the Single recovery approach presented: this section is an addition to an existing problem doc (cross-run-memory), not a standalone problem doc. it adds context on retry mechanics rather than proposing a complete solution. the approaches with trade-offs for the broader recovery problem are in the companion flapping-convergence doc (#3064).