Repository navigation
GH-3518: Solo-mode health-check loop no longer leaks an unbounded OTel trace (5.x backport) - #4123
Merged
Merged
Conversation
…l trace (5.x backport) Backport of main's a0772b5 (#3522) to the 5.x maintenance line. In DurabilityMode.Solo, StartSoloModeAsync scheduled the recurring health-check loop with a fire-and-forget Task.Run *inside* the startup wolverine_node_assignments activity's using block. Task.Run captures the ambient ExecutionContext -- including the AsyncLocal behind Activity.Current -- so every loop iteration (and every DB call underneath it) reparented itself to that one stopped-but-still-ambient activity for the entire process lifetime, producing a single OTel trace that grew without bound. Fix: close the startup activity scope before scheduling the loop, suppress ExecutionContext flow when forking the background task so Activity.Current is not captured, and start a fresh bounded activity per tick gated by ShouldTraceHealthCheck(). Each tick is now its own root trace, and NodeAssignmentHealthCheckTraceSamplingPeriod is finally honored in Solo mode. Scoped to GH-3518 only. Main's copy of startAllAgentsAsync also carries the GH-3519 per-agent start isolation, which is a separate behavior change and is deliberately NOT included here. Red baseline verified: with the test in place and the source fix reverted, each_recurring_tick_is_its_own_bounded_root_trace fails on the per-tick activity count. Full CoreTests (1691) pass on net9.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of main's
a0772b571(#3522) to the 5.x maintenance line. Found while auditing what OTel work main had that 5.x didn't during #4117; this was the remaining gap.The bug
In
DurabilityMode.Solo,StartSoloModeAsyncscheduled the recurring health-check loop with a fire-and-forgetTask.Runinside the startupwolverine_node_assignmentsactivity'susingblock.Task.Runcaptures the ambientExecutionContext— including theAsyncLocalbehindActivity.Current— so every loop iteration, and every DB call underneath it, reparented itself to that one stopped-but-still-ambient activity for the entire process lifetime. The result is a single OTel trace that grows without bound for as long as the host runs.Because
ShouldTraceHealthCheck()was also only evaluated once at startup,NodeAssignmentHealthCheckTraceSamplingPeriodwas a no-op in Solo mode.The fix
Three parts, identical to main:
ExecutionContext.SuppressFlow()soActivity.Currentis never captured into it.ShouldTraceHealthCheck()— so each tick is its own root trace and the sampling period is finally honored.Deliberately out of scope
Main's copy of
startAllAgentsAsyncalso carries the GH-3519 per-agent start isolation (each agent's start wrapped in its own try/catch so one wedged agent doesn't skip its healthy siblings). That is a separate behavior change from a different commit and is not included here — this PR is GH-3518 only. Happy to do it as its own backport if wanted.Testing
solo_mode_health_check_tracing(2 tests, ported from main). Red baseline verified: with the test in place and the source fix reverted,each_recurring_tick_is_its_own_bounded_root_tracefails on the per-tick activity count, and passes again once the fix is restored.sampling_period_throttles_solo_mode_recurring_checkspasses vacuously against the unfixed code (no per-tick activities exist to throttle). It is meaningful only paired with the first test, which establishes that ticks trace at all — same as on main.wolverine.slnxbuilds clean pinned to net9.0 (Release).🤖 Generated with Claude Code