Skip to content

fix: forced health check and re-check storm on failure bursts - #2984

Open
IsoLeyN wants to merge 3 commits into
MetaCubeX:Metafrom
IsoLeyN:Meta
Open

fix: forced health check and re-check storm on failure bursts#2984
IsoLeyN wants to merge 3 commits into
MetaCubeX:Metafrom
IsoLeyN:Meta

Conversation

@IsoLeyN

@IsoLeyN IsoLeyN commented Jul 17, 2026

Copy link
Copy Markdown

This PR fixes two related bugs in outboundgroup (fallback / url-test group logic) that affect setups using empty-fallback: REJECT.


Problem 1: fallback/url-test never recovers from an empty group

When a url-test or fallback group has no alive members, it resolves to its empty-fallback proxy (in this config, REJECT). The issue is that dialing through REJECT is not treated as a failure by the group: REJECT's DialContext returns a no-op connection with a nil error (it never actually connects anywhere), so onDialFailed is never called, the group's failure counter never increments, and the forced health check that would normally recover the group is never triggered.

In practice, this meant that if a url-test group's underlying provider was temporarily empty or unreachable at startup, the group would get stuck silently serving REJECT forever, only recovering at the next scheduled interval (up to 15 minutes in this config), instead of noticing the problem and re-checking its members.

Fix: added forcedHealthCheckNeeded(), which detects when the proxy actually being used to dial is REJECT/REJECT-DROP or is marked dead. When that's the case, DialContext/ListenPacketContext in both fallback.go and urltest.go now proactively kick off an async health check, so the group re-evaluates its members instead of waiting for the next scheduled interval.

Video Demonstration: https://youtu.be/watch?v=LoVrG-ICByg.


Problem 2: failure-triggered health checks can storm the network

GroupBase.onDialFailed already had logic to trigger a forced health check after N consecutive failed dials (maxFailedTimes). However, there was no cooldown between these forced checks. With a group containing ~200 proxies, a burst of failed dials (e.g. during a flaky connection) could re-trigger a full health check of every proxy in the group every few seconds, causing repeated storms of health-check traffic against all providers - visible in logs as "because failed multiple times, activate health check" firing every 6-30 seconds.
You might have noticed that in url-test groups, the ping fluctuates SEVERAL times right before your eyes. This is precisely the issue I'm referring to.

Fix: added a 30-second cooldown (forcedHealthCheckCooldown) around GroupBasegered path, tracked via lastForcedCheck (atomic.TypedValue[time.Time]).Scheduled health checks (via interval config) are unaffected - only the failure-triggered path is throttled. Also reset the failure counter after a forced check fires, so it doesn't keep re-triggering warnings on every subsequent failed dial within the same window.


Testing

Added forcedcheck_test.go with three tests (written test-first, confirmed fafore the fix):

  • TestForcedHealthCheckCooldown - verifies a second failure-triggered health check within the cooldown window is suppressed.
  • TestFallbackRejectDialTriggersHealthCheck - verifies dialing through REJECs a health check.
  • TestURLTestRejectDialTriggersHealthCheck - same, for url-test groups.

@IsoLeyN
IsoLeyN marked this pull request as draft July 17, 2026 10:48
@IsoLeyN
IsoLeyN marked this pull request as ready for review July 17, 2026 13:30
@IsoLeyN

IsoLeyN commented Jul 17, 2026

Copy link
Copy Markdown
Author

Problem 1

Today I noticed different behavior in the program. It turns out that the GUI I'm using doesn't automatically populate the URL for url-test groups. I manually set it, and tests started running automatically when I launch the core. However, this doesn't solve the issue of having to wait for the fallback interval.

In the video, I demonstrated the core's operation both without my code and with it. In the second part of the video, you can see how fallback immediately selects a group that has at least some content.

Demonstration video: https://youtu.be/MRBBIPpImy8.


Problem 2

Demonstration video: https://youtu.be/Vkm76fPFLiE.

I placed a stopwatch nearby to track the time elapsed between tests. In my config, the interval is set to 900 seconds. In the first example, you can see that the servers start being tested in less than 2 minutes, which is incorrect. In the second example, I show the corrected code where the proper 900-second wait time is enforced.

olicesx

This comment was marked as low quality.

@olicesx olicesx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is a high-quality fix. Both bugs are addressed correctly: I verified the Unwrap chain in resolvesToReject terminates (via Base.Unwrap returning nil at adapter/outbound/base.go:142), that scheduled health checks go through the provider's own ticker (adapter/provider/healthcheck.go:45) and are not affected by the new cooldown, and that fallback.go's ListenPacketContext got the same forced-check treatment as DialContext. Just a few minor suggestions below, none blocking.

Detailed review

minor

M1 - forcedHealthCheckNeeded takes a testUrl parameter it never uses
groupbase.go: func forcedHealthCheckNeeded(proxy C.Proxy, testUrl string) bool { return resolvesToReject(proxy) }. The testUrl argument is dead, but all 4 call sites (urltest.go DialContext/ListenPacketContext, fallback.go DialContext/ListenPacketContext) pass u.testUrl/f.testUrl. Consider dropping the parameter to forcedHealthCheckNeeded(proxy C.Proxy) bool, or note in the comment that it is reserved for future use.

M2 - The forced-check snippet is duplicated across 4 sites
The same if forcedHealthCheckNeeded(proxy, xx.testUrl) { go xx.healthCheck() } block appears in urltest.go (DialContext L57, ListenPacketContext L84) and fallback.go (DialContext L33, ListenPacketContext L59). Since both *URLTest and *Fallback embed *GroupBase, this could collapse to a single method on GroupBase, e.g. func (gb *GroupBase) maybeForceHealthCheck(proxy C.Proxy) { if forcedHealthCheckNeeded(proxy) { go gb.healthCheck() } }, which also keeps the testUrl removal from M1 consistent. Acceptable as-is, but future edits will need to touch 4 places.

nit

N1 - forcedHealthCheckNeeded is currently a thin wrapper around resolvesToReject
Single-line return resolvesToReject(proxy). The comment explains "plain dead member deliberately excluded", which implies room for future expansion. Fine to keep as a seam; merge into resolvesToReject if no expansion is planned.

Test gap (non-blocking)

T1 - gb.failedTimes = 0 reset in onDialFailed is not directly covered
Existing tests cover cooldown suppression and the REJECT-dial trigger, but not the new "reset failedTimes after firing so subsequent bursts in the same window don't re-emit Warnln" behavior. Logic is simple and low regression risk, but a small unit test would lock it in.

Points I verified (no issue)

  • resolvesToReject's for p := proxy; p != nil; p = p.Unwrap(nil, false) relies on Base.Unwrap returning nil to terminate (adapter/outbound/base.go:142) - the chain is finite, no infinite loop.
  • Selector/URLTest/Fallback Unwrap implementations do not dereference metadata, so passing nil is safe.
  • Scheduled health checks use the provider's own ticker (adapter/provider/healthcheck.go:45 time.NewTicker(hc.interval)) and do not route through gb.healthCheck(), so the new cooldown does not throttle scheduled checks - consistent with the PR description.
  • fallback.go's ListenPacketContext includes the same forced-check as DialContext.
  • The 30s floor applies only when no interval is configured; when interval is set it respects the user's configured value.

@IsoLeyN

IsoLeyN commented Jul 31, 2026

Copy link
Copy Markdown
Author

@olicesx Thanks for the review. Both findings are fixed, but digging into them I found that the first half of this PR didn't actually work, so I've removed it. The diff looks pretty different now.

What went wrong. healthCheck() ends up in HealthCheck.check() in adapter/provider/healthcheck.go, and the very first thing that function does is return early when it has nothing to test:

func (hc *HealthCheck) check() {
	if len(hc.proxies) == 0 {
		return
	}

A group only serves its empty-fallback when GetProxies came back empty - either the provider has no proxies, or the group's filters excluded all of them. In the first case the forced check returns right away; in the second it runs but the filters still exclude everything. Either way nothing recovers, because that's Update()'s job, not HealthCheck()'s.

So the forced-check-on-REJECT machinery cost a tree walk and a goroutine per dial and did nothing. What actually fixed things in my demo video was the other half - findAliveProxy skipping members that resolve to REJECT so the group picks a sibling with real capacity. That part stays, along with the cooldown from Problem 2. Pointing the trigger at provider.Update() would genuinely fix the empty-provider case, but that's a network fetch from the dial path, so I left it for a separate change.

Cooldown. Tying it to interval was also wrong. Scheduled checks already run on the provider's ticker every interval, so a forced check gated by the same value can never fire more often than the scheduled one. And the interval you get isn't the one you'd expect: in ParseProxyGroup, the Interval == 0 -> 300 default sits inside the len(groupOption.Proxies) != 0 branch, the one that builds a compatible provider for an inline list. A use:-only group never enters it, keeps Interval == 0, and silently fell through to the 30s floor instead - so two groups that look identical in the config got different cooldowns. Now it's one flat failureRecheckCooldown = 5 * time.Minute and interval isn't involved. The tradeoff: upstream called fn() with no throttle at all on the connection refused branch, and a group with use: and no url: gets no scheduled ticker either, so for that config the forced check is the only recovery and now waits up to 5 minutes.

resolvesToReject no longer uses Unwrap. You checked that the chain terminates, and it does, but it isn't side-effect free: Fallback.Unwrap runs findAliveProxy, which clears f.selected. So a probe everything treats as read-only was quietly dropping the user's pin - and my change made it worse, since a member that's alive but resolving to REJECT now fails the usability check too, clearing the pin in exactly the transient state this feature was written for. It walks the member list via the existing ProxyGroup interface now: a group counts as blackholing only if every member does. Conservative, and it mutates nothing.

Two bugs in my own patch, caught while re-testing:

  • URLTest.fast() could return a dead node - my replacement step took the first non-REJECT member and ignored aliveness, so [empty-group, dead, live] gave you the dead one.
  • I'd reordered findAliveProxy so resolvesToReject ran before AliveForTestUrl, which killed the short-circuit and made every dead member get fully walked. ~6x worse on nested fallbacks.

On M2 - just so it doesn't come up again elsewhere: a helper on GroupBase calling gb.healthCheck() wouldn't work, since URLTest overrides healthCheck to reset fastSingle and Go has no virtual dispatch. It'd silently call the embedded one. Has to take the check func as an argument.

Concurrency: failedTesting was a check-then-set, now a CAS with a deferred reset. healthCheck() had an unsynchronized failedTimes = 0 racing onDialFailed's increment - removed, onDialFailed already does it under the mutex. The cooldown gate has its own mutex and releases it before running the check, since onDialFailed holds failedTestMux there.

12 tests, green under -race. Stubbing resolvesToReject to always return false fails 6 of them.

Not covered here:

  • load-balance - none of the strategies use the usability check and all fall back to proxies[0], which can be the REJECT. So it still blackholes a share of traffic where fallback/url-test no longer do.
  • URLTest.fastNode and Fallback.selected are both written from concurrent dial paths with no synchronization, and a torn read of either two-word value is memory-unsafe. Root cause is singledo.Reset() clearing s.call while fn is still running, so a later Do starts a second concurrent fn. All pre-existing; I only removed the extra pressure my change was adding.

@IsoLeyN
IsoLeyN requested a review from olicesx July 31, 2026 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants