Skip to content

Escalate a tunnel failure that retrying cannot fix - #6

Merged
garethx merged 1 commit into
tests/tools-coveragefrom
fix/cli-project-mismatch
Aug 12, 2026
Merged

Escalate a tunnel failure that retrying cannot fix#6
garethx merged 1 commit into
tests/tools-coveragefrom
fix/cli-project-mismatch

Conversation

@garethx

@garethx garethx commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Rewritten after #3 merged. #4 suggested three fixes; two of them landed with #3, so this PR is now only the third.

#4's suggestion Status
1. doctor compares the CLI's project against the API key's Landed in #3 (_check_cli_project) — dropped from this PR
3. Point hookdeck listen at a gateway-owned config Landed in #3, and further: authenticate() mints the session too — dropped from this PR
2. Escalate a deterministic tunnel failure Not implemented. This PR.

Stacked on #5, which it shares tunnel tests with.

What is left to fix

no connection found matching filter will never come right by being retried. The supervisor treated it exactly like a network blip — warning level, doubling backoff, nothing new said on each pass. That is how "the gateway is receiving no events" stays invisible while everything above the tunnel lines reads healthy:

[hookdeck] mode=cli listening on 127.0.0.1:3579/hookdeck — ack_mode=async_retry …
[hookdeck cli] no connection found matching filter "livetest" for source "hermes-livetest"
[hookdeck] CLI tunnel exited after 1s — restarting in 2s
[hookdeck] CLI tunnel exited after 1s — restarting in 4s

Three consecutive runs too short to be healthy now produce one error:

[hookdeck] CLI tunnel has failed to stay up 3 times in a row (last run 1s). No
events are reaching the gateway. Likely cause: the CLI is forwarding from a
different Hookdeck project than the API key manages… Run `hermes hookdeck
doctor` — it compares the two. Last output: no connection found matching filter
"livetest" for source "hermes-livetest"

Once per streak rather than per restart, so the log does not fill up. A healthy run re-arms it, so a second outage is not silent. Unrecognised output escalates without a cause rather than guessing one — sending an operator after the wrong thing is worse than saying less.

The output tail resets per run: carrying it across would name a cause that had since been fixed, which is the worst kind of wrong because it looks specific.

Verification

299 tests pass, ruff clean. The escalation is mutation-checked — disabling it fails two tests. The tunnel tests replay the exact CLI output from #4.

Also included, since it is the same file and #5's rebase surfaced it: authenticate() arrived in #3 with no tests and is load-bearing (the adapter refuses to start when it returns False). That coverage is in #5, taking tunnel.py from 81% to 96%.

@garethx
garethx marked this pull request as ready for review August 12, 2026 10:16
@garethx
garethx force-pushed the tests/tools-coverage branch from db29562 to 609e626 Compare August 12, 2026 10:20
#4 suggested three fixes. Two of them landed with #3 — doctor now compares the
CLI's project against the API key's, and the gateway authenticates a CLI config
of its own. This is the third, which did not.

`no connection found matching filter` will never come right by being retried.
The supervisor treated it exactly like a network blip: warning level, doubling
backoff, nothing new said each time. That is how "the gateway is receiving no
events" stays invisible while everything above the tunnel lines reads healthy.

Three consecutive runs too short to count as healthy now log once at error
level, with the CLI's own last lines and, for the failures we recognise, the
likely cause and the command that confirms it. Once per streak, not per
restart, so the log does not fill up; a healthy run re-arms it, so a second
outage is not silent.

The output tail is reset per run — carrying it over would name a cause that had
since been fixed, which is worse than naming none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@garethx
garethx force-pushed the fix/cli-project-mismatch branch from bb7a68b to d09de3d Compare August 12, 2026 10:23
@garethx garethx changed the title Make a CLI/API-key project mismatch diagnosable instead of silent Escalate a tunnel failure that retrying cannot fix Aug 12, 2026
@garethx
garethx merged commit c1d29d4 into tests/tools-coverage Aug 12, 2026
8 checks passed
@garethx
garethx deleted the fix/cli-project-mismatch branch August 12, 2026 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant