Skip to content

GH-3518: Solo-mode health-check loop no longer leaks an unbounded OTel trace (5.x backport) - #4123

Merged
jeremydmiller merged 1 commit into
5.0from
gh-3518/solo-mode-trace-leak-5.0
Aug 25, 2026
Merged

jeremydmiller merged 1 commit into
5.0from
gh-3518/solo-mode-trace-leak-5.0

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Backport of main's a0772b571 (#3522) to the 5.x maintenance line. Found while auditing what OTel work main had that 5.x didn't during #4117; this was the remaining gap.

The bug

In DurabilityMode.Solo, StartSoloModeAsync scheduled the recurring health-check loop with a fire-and-forget Task.Run inside the startup wolverine_node_assignments activity's using block. Task.Run captures the ambient ExecutionContext — including the AsyncLocal behind Activity.Current — so every loop iteration, and every DB call underneath it, reparented itself to that one stopped-but-still-ambient activity for the entire process lifetime. The result is a single OTel trace that grows without bound for as long as the host runs.

Because ShouldTraceHealthCheck() was also only evaluated once at startup, NodeAssignmentHealthCheckTraceSamplingPeriod was a no-op in Solo mode.

The fix

Three parts, identical to main:

  1. Close the startup activity's scope before the recurring loop is scheduled.
  2. Fork the loop inside ExecutionContext.SuppressFlow() so Activity.Current is never captured into it.
  3. Start a fresh, bounded activity per tick, gated by ShouldTraceHealthCheck() — so each tick is its own root trace and the sampling period is finally honored.

Deliberately out of scope

Main's copy of startAllAgentsAsync also carries the GH-3519 per-agent start isolation (each agent's start wrapped in its own try/catch so one wedged agent doesn't skip its healthy siblings). That is a separate behavior change from a different commit and is not included here — this PR is GH-3518 only. Happy to do it as its own backport if wanted.

Testing

  • New solo_mode_health_check_tracing (2 tests, ported from main). Red baseline verified: with the test in place and the source fix reverted, each_recurring_tick_is_its_own_bounded_root_trace fails on the per-tick activity count, and passes again once the fix is restored.
  • Worth being precise about the second test: sampling_period_throttles_solo_mode_recurring_checks passes vacuously against the unfixed code (no per-tick activities exist to throttle). It is meaningful only paired with the first test, which establishes that ticks trace at all — same as on main.
  • Full CoreTests: 1691 passed, 0 failed on net9.0.
  • Full wolverine.slnx builds clean pinned to net9.0 (Release).

🤖 Generated with Claude Code

…l trace (5.x backport)

Backport of main's a0772b5 (#3522) to the 5.x maintenance line.

In DurabilityMode.Solo, StartSoloModeAsync scheduled the recurring
health-check loop with a fire-and-forget Task.Run *inside* the startup
wolverine_node_assignments activity's using block. Task.Run captures the
ambient ExecutionContext -- including the AsyncLocal behind
Activity.Current -- so every loop iteration (and every DB call underneath
it) reparented itself to that one stopped-but-still-ambient activity for
the entire process lifetime, producing a single OTel trace that grew
without bound.

Fix: close the startup activity scope before scheduling the loop,
suppress ExecutionContext flow when forking the background task so
Activity.Current is not captured, and start a fresh bounded activity per
tick gated by ShouldTraceHealthCheck(). Each tick is now its own root
trace, and NodeAssignmentHealthCheckTraceSamplingPeriod is finally
honored in Solo mode.

Scoped to GH-3518 only. Main's copy of startAllAgentsAsync also carries
the GH-3519 per-agent start isolation, which is a separate behavior
change and is deliberately NOT included here.

Red baseline verified: with the test in place and the source fix
reverted, each_recurring_tick_is_its_own_bounded_root_trace fails on the
per-tick activity count. Full CoreTests (1691) pass on net9.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit b847db6 into 5.0 Aug 25, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant