Harden Envoy backend: pin image digest, disable admin interface by ChrisJBurns · Pull Request #5949 · stacklok/toolhive

ChrisJBurns · 2026-07-23T17:47:37Z

Summary

Two hardening cleanups for the Envoy network-proxy backend before wider rollout.

Changes

Image digest pinning: defaultEnvoyImage was tag-pinned (v1.32.3). Tag mutation is a supply-chain attack vector — a registry push can silently swap the image behind the same tag. The constant is now tag@digest, combining human readability with cryptographic integrity. Users with stricter image policies can still substitute via TOOLHIVE_ENVOY_IMAGE.

Admin interface disabled: The admin block bound to 127.0.0.1:9901 on every Envoy container. Omitting the admin block entirely causes Envoy to skip the admin server — port 9901 never opens. The admin API is not used by ToolHive at runtime; it existed only as a debugging convenience. The test is updated from asserting loopback-only binding to asserting the admin block is absent.

File	Change
`pkg/container/docker/envoy.go`	Digest-pin image constant; remove `Admin` field from `envoyBootstrap` struct and construction
`pkg/container/docker/envoy_test.go`	Remove `Admin` from test bootstraps; rename `TestEnvoyAdmin_LoopbackOnly` → `TestEnvoyAdmin_Absent`

Type of change

New feature

Test plan

task build passes
task lint-fix passes
Non-Docker unit tests (TestEnvoyAdmin_Absent, TestWriteEnvoyBootstrap_FileMode, TestGetEnvoyImage) pass
TestEnvoyBootstrap_ValidatesAgainstRealEnvoy validates admin-less bootstrap against real Envoy (runs in CI with Docker)

Special notes for reviewers

The digest sha256:375aab0d80b3c0e1b42a776b4cb1743ed79012032051d2da19cbc93ea884fb81 was obtained via crane digest envoyproxy/envoy-distroless:v1.32.3 and matches the manifest index for the v1.32.3 tag. When upgrading Envoy, update both the tag and the digest together.

Generated with Claude Code

Pin defaultEnvoyImage to tag+digest for supply-chain integrity. Add a comment documenting the TOOLHIVE_ENVOY_IMAGE override for users with image-policy requirements. Remove the admin interface block from the bootstrap. Envoy does not start an admin server when the field is absent, eliminating the 9901 port surface area with no functional impact on the proxy. Update the test from asserting loopback-only binding to asserting the admin block is entirely absent. Closes #5903 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

The middleware returns HTTP 200 with a JSON-RPC error body when the client accepts JSON (absent Accept header → clientAcceptsJSON = true). This is correct per the MCP streamable-HTTP spec: application-level errors ride in a 200 response, not a 400. The test added in #5934 was asserting 400, which never matched the production code path. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

codecov · 2026-07-23T20:56:53Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.81%. Comparing base (c007e6a) to head (af377fd).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5949      +/-   ##
==========================================
+ Coverage   71.80%   71.81%   +0.01%     
==========================================
  Files         708      708              
  Lines       72767    72790      +23     
==========================================
+ Hits        52250    52277      +27     
+ Misses      16778    16774       -4     
  Partials     3739     3739

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

glageju

Multi-Agent Consensus Review

Agents consulted: security-specialist, go-correctness-specialist, general-quality-specialist, codex (gpt-5.5)

Consensus Summary

#	Finding	Consensus	Severity	Action
1	`PullImage` cannot handle `tag@digest` references — breaks Envoy setup on any non-cached host	10/10	HIGH	Fix
2	No test exercises the real image-pull path with a digest-pinned reference	9/10	HIGH	Fix
3	Unrelated streamable-HTTP test fix bundled without disclosure	8/10	MEDIUM	Discuss
4	PR body's Changes table omits the third modified file	7/10	MEDIUM	Fix
5	Test plan doesn't cover the streamable dispatcher change	7/10	MEDIUM	Discuss
6	"Type of change" checkbox ("New feature") doesn't match hardening/cleanup content	6/10	LOW	Discuss
7	`TestEnvoyAdmin_Absent` uses substring checks instead of structural JSON assertions	6/10	LOW	Discuss

Overall

The intent here is sound and both tasks in #5903 are nominally addressed: defaultEnvoyImage is pinned by digest (verified against crane digest envoyproxy/envoy-distroless:v1.32.3 — exact match), and the admin interface is removed outright rather than merely bound to loopback, which is the stronger of the two options the issue proposed. The admin-removal half is clean, complete, and validated against a real Envoy binary via TestEnvoyBootstrap_ValidatesAgainstRealEnvoy.

The digest-pinning half has a functional regression, though. RegistryImageManager.PullImage (pkg/container/images/registry.go) converts the parsed reference to a name.Tag for the daemon.Write call, with a fallback to name.NewTag(ref.String()) when the reference isn't already a tag. For a repo:tag@digest string, name.ParseReference correctly resolves to a name.Digest, so the fallback fires — and name.NewTag rejects the string because the @ makes the "repository" portion invalid (repository can only contain the characters ...). I reproduced this directly inside pkg/container/images against the version of go-containerregistry pinned in go.mod (v0.21.2), so this isn't tool-specific or a local environment quirk — it's the exact code path PullImage runs. The practical effect: PullImage errors on this exact image string every time, and SetupIngress's fallback (ImageExists) only recovers if the image is already cached in the local Docker daemon. On a fresh install or CI runner, that's not the case, so the Envoy proxy — and the workload depending on it — fails to start. None of the existing tests catch this, since TestGetEnvoyImage and the orchestration tests use fakeImageManager (always succeeds), and TestEnvoyBootstrap_ValidatesAgainstRealEnvoy calls docker run directly, bypassing ToolHive's own pull path entirely.

Separately, the PR bundles a second, unrelated commit that fixes a test assertion in dispatcher_standalone_sse_integration_test.go (a different package, streamable-HTTP transport, addressing a bug introduced in #5934). The fix itself looks correct — it now matches the established pattern elsewhere in the same file for how JSON-RPC application errors ride in HTTP 200 responses — but it's undisclosed in the PR title, summary, and Changes table, which only account for the two Envoy files.

Generated with Claude Code

glageju · 2026-07-23T21:33:36Z

-	defaultEnvoyImage = "envoyproxy/envoy-distroless:v1.32.3"
+	// defaultEnvoyImage is pinned by tag+digest for supply-chain integrity.
+	// Override with TOOLHIVE_ENVOY_IMAGE (accepts any docker pull reference).
+	defaultEnvoyImage = "envoyproxy/envoy-distroless:v1.32.3@sha256:375aab0d80b3c0e1b42a776b4cb1743ed79012032051d2da19cbc93ea884fb81"


[HIGH, 10/10] PullImage cannot handle this tag@digest reference — breaks Envoy setup on any non-cached host

name.ParseReference resolves this string to a name.Digest, not a name.Tag. RegistryImageManager.PullImage (pkg/container/images/registry.go:100-108) then falls back to name.NewTag(ref.String()), which errors (repository can only contain the characters ...) because the @sha256:... suffix makes the "repository" portion invalid for a tag. I reproduced this directly inside pkg/container/images against the go-containerregistry v0.21.2 version pinned in go.mod — PullImage errors on every call with this exact image string, regardless of registry availability.

In SetupIngress (envoy.go ~line 919), the failure path calls ImageExists, which only checks the local Docker daemon cache. On a fresh install or CI runner where this image isn't already cached, that returns false, and SetupIngress returns an error — the Envoy proxy (and the workload depending on it) fails to start.

Recommendation: fix RegistryImageManager.PullImage's tag-conversion fallback to handle name.Digest references without requiring a name.Tag for the daemon.Write call (e.g. derive a synthetic tag, or use a daemon-write path that accepts a name.Reference directly). This should block merge as-is — shipping this would break Envoy backend startup for most users on first run.

Raised by: security specialist, independently verified.

glageju · 2026-07-23T21:33:36Z

-	defaultEnvoyImage = "envoyproxy/envoy-distroless:v1.32.3"
+	// defaultEnvoyImage is pinned by tag+digest for supply-chain integrity.
+	// Override with TOOLHIVE_ENVOY_IMAGE (accepts any docker pull reference).
+	defaultEnvoyImage = "envoyproxy/envoy-distroless:v1.32.3@sha256:375aab0d80b3c0e1b42a776b4cb1743ed79012032051d2da19cbc93ea884fb81"


[HIGH, 9/10] No test exercises the real image-pull path with a digest-pinned reference

TestGetEnvoyImage and the SetupIngress orchestration tests use fakeImageManager, which always succeeds and never touches go-containerregistry's reference-parsing/pull logic. TestEnvoyBootstrap_ValidatesAgainstRealEnvoy calls docker run directly, bypassing ToolHive's own PullImage/ImageExists entirely. As a result, the PullImage regression above shipped without any failing test.

Recommendation: add a test (can be gated behind Docker availability like the existing real-Envoy test) that calls RegistryImageManager.PullImage/ImageExists with a real tag@digest reference, or an end-to-end check that starts the Envoy backend on a clean image cache.

Raised by: security specialist.

glageju · 2026-07-23T21:33:36Z

+	assert.NotContains(t, s, `"admin"`,
+		"bootstrap must not contain an admin block")
+	assert.NotContains(t, s, "9901",
+		"bootstrap must not reference the admin port")


[LOW, 6/10] TestEnvoyAdmin_Absent uses substring checks instead of structural JSON assertions

Checking that the marshaled JSON doesn't contain the substrings "admin" and 9901 is adequate for the current struct shape, but weaker than a structural check — a future field value that happens to contain "admin" as a substring could produce a false negative.

Recommendation: consider unmarshaling into map[string]any and asserting the admin key is absent (_, ok := m["admin"]; assert.False(t, ok)), which is precise regardless of what else the config contains. The 9901 check is redundant once the admin key's absence is confirmed structurally.

Raised by: go-correctness specialist, codex.

glageju · 2026-07-23T21:33:36Z

+	// The middleware returns HTTP 200 with a JSON-RPC error body when the
+	// client accepts JSON (no Accept header → clientAcceptsJSON returns true).
+	// This is correct MCP streamable HTTP behaviour: application-level errors
+	// ride in 200 responses carrying a JSON-RPC error object.
 	callBody := `{"jsonrpc":"2.0","id":"call-1","method":"tools/call","params":{"name":"filtered_tool"}}`
 	callResp, err := http.Post(endpoint, "application/json", strings.NewReader(callBody)) //nolint:noctx // test-only
 	require.NoError(t, err)
 	t.Cleanup(func() { _ = callResp.Body.Close() })
-	assert.Equal(t, http.StatusBadRequest, callResp.StatusCode)
+	callRespBody, err := io.ReadAll(callResp.Body)
+	require.NoError(t, err)
+	assert.Equal(t, http.StatusOK, callResp.StatusCode)
+	assert.Contains(t, string(callRespBody), `"error"`, "response must contain a JSON-RPC error for the filtered tool")
+	assert.NotContains(t, string(callRespBody), "filtered_tool", "error must not leak the filtered tool name")


[MEDIUM, 8/10] Unrelated test fix bundled into an Envoy-hardening PR without disclosure

This PR's title, summary, and Changes table are entirely about Envoy digest pinning and admin-interface removal, but this hunk fixes an assertion in a different subsystem (streamable-HTTP transport dispatcher), addressing a bug from a different PR (#5934). The fix itself is technically correct — it matches the established pattern elsewhere in this file for how JSON-RPC application errors ride in HTTP 200 responses — but it's undisclosed here, so reviewers scanning the title/description have no reason to look at this file, and reverting the Envoy hardening would also silently revert this fix.

Recommendation: split this into its own PR with its own description, or at minimum update this PR's title/summary/Changes table and test plan to disclose and justify it.

Raised by: go-correctness specialist, general-quality specialist, codex (3/3 agents).

ChrisJBurns requested review from JAORMX, amirejaz, blkt, jhrozek and rdimitrov as code owners July 23, 2026 17:47

github-actions Bot added the size/XS Extra small PR: < 100 lines changed label Jul 23, 2026

github-actions Bot added size/XS Extra small PR: < 100 lines changed and removed size/XS Extra small PR: < 100 lines changed labels Jul 23, 2026

glageju reviewed Jul 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Harden Envoy backend: pin image digest, disable admin interface#5949

Harden Envoy backend: pin image digest, disable admin interface#5949
ChrisJBurns wants to merge 2 commits into
mainfrom
cburns/envoy-harden-5903

ChrisJBurns commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026

Uh oh!

glageju left a comment

Uh oh!

glageju Jul 23, 2026

Uh oh!

glageju Jul 23, 2026

Uh oh!

glageju Jul 23, 2026

Uh oh!

glageju Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ChrisJBurns commented Jul 23, 2026

Summary

Type of change

Test plan

Special notes for reviewers

Uh oh!

codecov Bot commented Jul 23, 2026

Codecov Report

Uh oh!

glageju left a comment

Choose a reason for hiding this comment

Multi-Agent Consensus Review

Consensus Summary

Overall

Uh oh!

glageju Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

glageju Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

glageju Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

glageju Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants