Escalate a tunnel failure that retrying cannot fix - #7
Merged
Conversation
#4 suggested three fixes. Two of them landed with #3 — doctor now compares the CLI's project against the API key's, and the gateway authenticates a CLI config of its own. This is the third, which did not. `no connection found matching filter` will never come right by being retried. The supervisor treated it exactly like a network blip: warning level, doubling backoff, nothing new said each time. That is how "the gateway is receiving no events" stays invisible while everything above the tunnel lines reads healthy. Three consecutive runs too short to count as healthy now log once at error level, with the CLI's own last lines and, for the failures we recognise, the likely cause and the command that confirms it. Once per streak, not per restart, so the log does not fill up; a healthy run re-arms it, so a second outage is not silent. The output tail is reset per run — carrying it over would name a cause that had since been fixed, which is worse than naming none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-lands #6, which merged into its stacked base branch (
tests/tools-coverage) rather thanmain— #5 landed 14 seconds earlier, so GitHub had not retargeted the base yet.hookdeck/tunnel.pyis untouched onmainsince v0.1.1, so none of this shipped.Same single commit (
d09de3d), cherry-picked onto currentmain. No conflicts.What it does
no connection found matching filterwill never come right by being retried. The supervisor treated it exactly like a network blip — warning level, doubling backoff, nothing new said on each pass. That is how "the gateway is receiving no events" stays invisible while everything above the tunnel lines reads healthy.Three consecutive runs too short to be healthy now produce one error naming the likely cause and the command that confirms it. Once per streak rather than per restart; a healthy run re-arms it, so a second outage is not silent. Unrecognised output escalates without a cause rather than guessing one.
This was the third of #4's three suggestions — the other two landed with #3.
Verification
299 tests pass, ruff clean. Mutation-checked: disabling the escalation fails two tests. The tunnel tests replay the exact CLI output from #4.