Skip to content

feat: support configurable proxy host and user-provided aiohttp client#5632

Open
Incheonkirin wants to merge 1 commit into
bentoml:mainfrom
Incheonkirin:feat/proxy-host-and-user-client
Open

feat: support configurable proxy host and user-provided aiohttp client#5632
Incheonkirin wants to merge 1 commit into
bentoml:mainfrom
Incheonkirin:feat/proxy-host-and-user-client

Conversation

@Incheonkirin

Copy link
Copy Markdown

Summary

create_proxy_app hardcoded localhost as the proxy host and could not use a user-provided aiohttp.ClientSession (set via self.client). Closes #5568.

The user's session has no base_url — they cannot know BentoML's internal proxy_port at client-creation time — so the existing code (which relied on base_url and built relative URLs) broke for user-provided clients. The host was also hardcoded to localhost, so requests could not be forwarded to a remote/external upstream server.

Changes

  • Configurable proxy host: add optional http.proxy_host (default localhost) so requests can be forwarded to a remote/external ML server.
  • Remote-safe process management: when proxy_host is remote, should_start_process is False — BentoML cannot spawn the upstream command on another machine.
  • Absolute URLs: build full absolute URLs in reverse_proxy and the startup health check instead of relying on the session's base_url, so a user-provided client works as-is (custom auth/SSL/retries supported).
  • Config registration: register proxy_host in the v2 config schema, the SDK HTTPSchema, and the default configuration.

Testing

  • py_compile passes for all changed Python files; default config YAML parses.
  • Verified URL construction (local / remote / multi-segment path / query string) and the remote-host classification (localhost/127.0.0.1/::1 → local; others → remote).

Notes

Backward compatible: when proxy_host is unset it defaults to localhost, preserving current behavior. No new dependencies.

`create_proxy_app` hardcoded `localhost` as the proxy host and could not use a
user-provided `aiohttp.ClientSession` (set via `self.client`), because it relied
on `base_url` being configured on the session — which a user cannot set, since
BentoML's internal proxy port is unknown at client-creation time.

- Add an optional `http.proxy_host` config (default `localhost`) so requests can
  be forwarded to a remote/external upstream server.
- When the proxy host is remote, do not spawn the upstream command locally
  (should_start_process = False).
- Build absolute URLs in the reverse proxy and the health check instead of
  relying on the session's base_url, so a user-provided client works as-is.
- Register `proxy_host` in the v2 config schema, the SDK HTTPSchema, and the
  default configuration.

Closes bentoml#5568

Signed-off-by: Mingi Jeong <incheonkirin@users.noreply.github.com>
@Incheonkirin Incheonkirin requested a review from a team as a code owner June 5, 2026 07:45
@Incheonkirin Incheonkirin requested review from ssheng and removed request for a team June 5, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support user-provided aiohttp.ClientSession and configurable proxy host in create_proxy_app

1 participant