diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 5bc04beb3..d9b9672a5 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -296,11 +296,9 @@ openshell logs | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | -| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | -| HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | -| HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | - -## Reporting +| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and a `gatewayApi` resource is enabled | +| TLSRoute remains unaccepted or port 443 is not exposed | The referenced Gateway has no compatible TLS passthrough listener, or the Gateway API controller does not support TLSRoute | Check `kubectl describe tlsroute -n ` and `kubectl describe gateway -n `; with a chart-created Gateway, verify both `gatewayApi.gateway.create=true` and `gatewayApi.routes.tls.enabled=true` | +| TLS passthrough connection fails during the handshake | The gateway server is running plaintext or its certificate does not cover the external SNI hostname | Keep `server.disableTls=false`, verify the server certificate SAN, and configure `pkiInitJob.serverDnsNames` or `certManager.serverDnsNames` before issuing the certificate | When handing results back to the user, include: diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index f3a86f884..b77437bb8 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -53,7 +53,8 @@ The `dev` tags are intended for testing changes ahead of a release. Production d See [`values.yaml`](values.yaml) for source defaults. Selected overlays: -- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration +- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - Gateway API configuration +- [`ci/values-gateway-tls.yaml`](ci/values-gateway-tls.yaml) - Gateway API TLS passthrough configuration - [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration - [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration - [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing @@ -147,16 +148,14 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | -| grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | -| grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. | -| grpcRoute.gateway.listener.allowedRoutes | string | `"Same"` | "Same" restricts attached routes to the release namespace; "All" allows any namespace. | -| grpcRoute.gateway.listener.port | int | `80` | Listener port for the generated Gateway resource. Use 443 with protocol HTTPS. | -| grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource: HTTP or HTTPS. HTTPS terminates TLS at the Envoy Gateway listener; pair it with server.disableTls=true so Envoy forwards plaintext to the gateway pod, and use OIDC for client identity (the gateway never sees the client cert). | -| grpcRoute.gateway.listener.tls.certificateRefs | list | `[]` | certificateRefs for the HTTPS listener. Required when protocol is HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret in the Gateway's namespace. May reference a cert-manager-issued Secret or the existing openshell-server-tls Secret (its SANs must include the external hostname). | -| grpcRoute.gateway.name | string | `""` | Name of the Gateway resource. Defaults to the chart fullname. | -| grpcRoute.gateway.namespace | string | `""` | Namespace of the Gateway referenced by the GRPCRoute parentRef. Defaults to the release namespace. | -| grpcRoute.hostnames | list | `[]` | Hostnames the GRPCRoute matches on. Leave empty to match all hosts. | +| gatewayApi.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | +| gatewayApi.gateway.create | bool | `false` | When true, a Gateway resource is created. Set to false and provide name/namespace to attach to a pre-existing Gateway. | +| gatewayApi.gateway.name | string | `""` | Name of the Gateway resource. Defaults to the chart fullname. | +| gatewayApi.gateway.namespace | string | `""` | Namespace of a pre-existing Gateway referenced by routes. Defaults to the release namespace. Chart-created Gateways always use the release namespace. | +| gatewayApi.routes.grpc.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | +| gatewayApi.routes.grpc.hostnames | list | `[]` | Hostnames the GRPCRoute matches on. Leave empty to match all hosts. | +| gatewayApi.routes.tls.enabled | bool | `false` | Create a Gateway API TLSRoute that passes encrypted traffic through to the gateway service. When gatewayApi.gateway.create is true, the generated Gateway also gets a TLS listener on port 443 in Passthrough mode. | +| gatewayApi.routes.tls.hostnames | list | `[]` | SNI hostnames the TLSRoute matches on. Leave empty to match all hosts. | | image.pullPolicy | string | `"IfNotPresent"` | Gateway image pull policy. | | image.repository | string | `"ghcr.io/nvidia/openshell/gateway"` | Gateway image repository. | | image.tag | string | `""` | Gateway image tag. Defaults to the chart appVersion when empty. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index e246ca67b..2d5d2bf4f 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -53,7 +53,8 @@ The `dev` tags are intended for testing changes ahead of a release. Production d See [`values.yaml`](values.yaml) for source defaults. Selected overlays: -- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration +- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - Gateway API configuration +- [`ci/values-gateway-tls.yaml`](ci/values-gateway-tls.yaml) - Gateway API TLS passthrough configuration - [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration - [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration - [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing diff --git a/deploy/helm/openshell/ci/values-gateway-tls.yaml b/deploy/helm/openshell/ci/values-gateway-tls.yaml index a77676014..f8ea60b66 100644 --- a/deploy/helm/openshell/ci/values-gateway-tls.yaml +++ b/deploy/helm/openshell/ci/values-gateway-tls.yaml @@ -1,33 +1,20 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Gateway API overlay with TLS termination at the Envoy Gateway listener. -# -# Exercises the HTTPS listener branch of templates/gateway.yaml for -# `helm template`/lint coverage. Envoy Gateway terminates TLS using the -# referenced kubernetes.io/tls Secret and forwards plaintext to the gateway pod, -# so the gateway runs with TLS disabled and uses OIDC for client identity. -# -# The certificate Secret (openshell-ingress-tls) and the OIDC issuer below are -# placeholders for render coverage; a real deployment must provide a valid TLS -# Secret in the release namespace and a reachable OIDC issuer. +# Gateway API overlay with TLS passthrough to the OpenShell gateway server. +# Envoy routes by SNI without terminating TLS, preserving server TLS and mTLS. -grpcRoute: - enabled: true +gatewayApi: gateway: create: true className: "eg" - listener: - port: 443 - protocol: HTTPS - tls: - certificateRefs: - - name: openshell-ingress-tls - hostnames: [] + routes: + tls: + enabled: true + hostnames: + - openshell.example.com -server: - # Envoy terminates TLS at the edge; the gateway listens plaintext behind it. - disableTls: true - oidc: - issuer: "https://keycloak.example.com/realms/openshell" - audience: "openshell-cli" +# Include the external hostname in certificates generated for fresh installs. +pkiInitJob: + serverDnsNames: + - openshell.example.com diff --git a/deploy/helm/openshell/ci/values-gateway.yaml b/deploy/helm/openshell/ci/values-gateway.yaml index 196192213..c22066350 100644 --- a/deploy/helm/openshell/ci/values-gateway.yaml +++ b/deploy/helm/openshell/ci/values-gateway.yaml @@ -15,12 +15,14 @@ # kubectl -n openshell port-forward svc/ 8080:80 # # then: grpcurl -plaintext localhost:8080 ... -grpcRoute: - enabled: true +gatewayApi: gateway: create: true className: "eg" - # Set one or more hostnames to scope the route, e.g.: - # hostnames: - # - openshell.example.com - hostnames: [] + routes: + grpc: + enabled: true + # Set one or more hostnames to scope the route, e.g.: + # hostnames: + # - openshell.example.com + hostnames: [] diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml index d571c3bd9..0cda95433 100644 --- a/deploy/helm/openshell/skaffold.yaml +++ b/deploy/helm/openshell/skaffold.yaml @@ -71,14 +71,14 @@ deploy: # crds.enabled: true # Envoy Gateway — Kubernetes Gateway API implementation. # Installs the Gateway API CRDs and the "eg" GatewayClass. - # Required when grpcRoute.enabled is true in the openshell release. + # Required when Gateway API routes are enabled in the openshell release. #- name: envoy-gateway # remoteChart: oci://docker.io/envoyproxy/gateway-helm - # version: v1.7.2 + # version: v1.8.1 # namespace: envoy-gateway-system # createNamespace: true # # wait ensures Gateway API CRDs are registered before the openshell - # # release attempts to create Gateway and HTTPRoute resources. + # # release attempts to create Gateway and route resources. # wait: true # SPIRE — installs SPIRE Server, Agent, Controller Manager, CSI Driver, # and OIDC Discovery Provider using the SPIFFE hardened charts. @@ -114,8 +114,10 @@ deploy: # setup task first, then uncomment the line below: # mise run keycloak:k8s:setup #- ci/values-keycloak.yaml - # To enable the Gateway API HTTPRoute (requires Envoy Gateway above): + # To enable the Gateway API Gateway and GRPCRoute (requires Envoy Gateway above): #- ci/values-gateway.yaml + # To enable TLS passthrough instead (requires Envoy Gateway above): + #- ci/values-gateway-tls.yaml # To enable SPIFFE/SPIRE provider token grants (requires the # spire-crds and spire releases above): #- ci/values-spire.yaml diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 5872dc404..12d6d80e1 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -109,6 +109,26 @@ Name of the Secret holding gateway-minted sandbox JWT signing material. {{- .Values.server.sandboxJwt.signingSecretName | default (printf "%s-jwt-keys" (include "openshell.fullname" .)) -}} {{- end }} +{{/* +Name of the Kubernetes Gateway API Gateway resource created or referenced by +the chart. +*/}} +{{- define "openshell.gatewayApiGatewayName" -}} +{{- .Values.gatewayApi.gateway.name | default (include "openshell.fullname" .) -}} +{{- end }} + +{{/* +Namespace of the Kubernetes Gateway API Gateway resource created or referenced +by the chart. +*/}} +{{- define "openshell.gatewayApiGatewayNamespace" -}} +{{- if .Values.gatewayApi.gateway.create -}} +{{- .Release.Namespace -}} +{{- else -}} +{{- .Values.gatewayApi.gateway.namespace | default .Release.Namespace -}} +{{- end -}} +{{- end }} + {{/* gRPC endpoint sandbox pods use to call back into the gateway. An explicit .Values.server.grpcEndpoint is used verbatim. Otherwise it is derived from @@ -178,6 +198,12 @@ database requires persistent per-pod storage. Validate chart values that Helm would otherwise accept silently. */}} {{- define "openshell.validateValues" -}} +{{- if hasKey .Values "grpcRoute" -}} +{{- fail "grpcRoute values were replaced by gatewayApi; configure Gateway creation under gatewayApi.gateway and GRPCRoute creation under gatewayApi.routes.grpc." -}} +{{- end -}} +{{- if and .Values.gatewayApi.routes.tls.enabled .Values.server.disableTls -}} +{{- fail "gatewayApi.routes.tls.enabled requires server.disableTls=false because TLS passthrough forwards the encrypted connection to the gateway server." -}} +{{- end -}} {{- $workloadKind := include "openshell.workloadKind" . -}} {{- $workload := .Values.workload | default dict -}} {{- $replicaCount := int (default 1 .Values.replicaCount) -}} diff --git a/deploy/helm/openshell/templates/gateway.yaml b/deploy/helm/openshell/templates/gateway.yaml index f431ffbbd..e3c30c28a 100644 --- a/deploy/helm/openshell/templates/gateway.yaml +++ b/deploy/helm/openshell/templates/gateway.yaml @@ -1,33 +1,25 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -{{- if and .Values.grpcRoute.enabled .Values.grpcRoute.gateway.create }} +{{- if .Values.gatewayApi.gateway.create }} apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: - name: {{ default (include "openshell.fullname" .) .Values.grpcRoute.gateway.name }} - namespace: {{ .Release.Namespace }} + name: {{ include "openshell.gatewayApiGatewayName" . }} + namespace: {{ include "openshell.gatewayApiGatewayNamespace" . }} labels: {{- include "openshell.labels" . | nindent 4 }} spec: - gatewayClassName: {{ .Values.grpcRoute.gateway.className }} + gatewayClassName: {{ .Values.gatewayApi.gateway.className }} listeners: - - name: {{ ternary "https" "http" (eq .Values.grpcRoute.gateway.listener.protocol "HTTPS") }} - port: {{ .Values.grpcRoute.gateway.listener.port }} - protocol: {{ .Values.grpcRoute.gateway.listener.protocol }} - {{- if eq .Values.grpcRoute.gateway.listener.protocol "HTTPS" }} - {{- if not .Values.grpcRoute.gateway.listener.tls.certificateRefs }} - {{- fail "grpcRoute.gateway.listener.tls.certificateRefs is required when grpcRoute.gateway.listener.protocol is HTTPS" }} - {{- end }} - {{- if not .Values.server.disableTls }} - {{- fail "grpcRoute.gateway.listener.protocol=HTTPS terminates TLS at Envoy Gateway, which forwards plaintext gRPC to the gateway pod; set server.disableTls=true so the pod listens plaintext (this chart does not render a BackendTLSPolicy for re-encryption to a TLS backend)" }} - {{- end }} + - name: http + port: 80 + protocol: HTTP + {{- if .Values.gatewayApi.routes.tls.enabled }} + - name: tls + port: 443 + protocol: TLS tls: - mode: Terminate - certificateRefs: - {{- toYaml .Values.grpcRoute.gateway.listener.tls.certificateRefs | nindent 10 }} - {{- end }} - allowedRoutes: - namespaces: - from: {{ .Values.grpcRoute.gateway.listener.allowedRoutes }} + mode: Passthrough + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/templates/grpcroute.yaml b/deploy/helm/openshell/templates/grpcroute.yaml index 8fde5458c..6effeca0f 100644 --- a/deploy/helm/openshell/templates/grpcroute.yaml +++ b/deploy/helm/openshell/templates/grpcroute.yaml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -{{- if .Values.grpcRoute.enabled }} +{{- if .Values.gatewayApi.routes.grpc.enabled }} apiVersion: gateway.networking.k8s.io/v1 kind: GRPCRoute metadata: @@ -11,11 +11,11 @@ metadata: {{- include "openshell.labels" . | nindent 4 }} spec: parentRefs: - - name: {{ default (include "openshell.fullname" .) .Values.grpcRoute.gateway.name }} - namespace: {{ default .Release.Namespace .Values.grpcRoute.gateway.namespace }} - {{- if .Values.grpcRoute.hostnames }} + - name: {{ include "openshell.gatewayApiGatewayName" . }} + namespace: {{ include "openshell.gatewayApiGatewayNamespace" . }} + {{- if .Values.gatewayApi.routes.grpc.hostnames }} hostnames: - {{- toYaml .Values.grpcRoute.hostnames | nindent 4 }} + {{- toYaml .Values.gatewayApi.routes.grpc.hostnames | nindent 4 }} {{- end }} rules: - backendRefs: diff --git a/deploy/helm/openshell/templates/tlsroute.yaml b/deploy/helm/openshell/templates/tlsroute.yaml new file mode 100644 index 000000000..3da5f3ec3 --- /dev/null +++ b/deploy/helm/openshell/templates/tlsroute.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.gatewayApi.routes.tls.enabled }} +apiVersion: gateway.networking.k8s.io/v1 +kind: TLSRoute +metadata: + name: {{ include "openshell.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + parentRefs: + - name: {{ include "openshell.gatewayApiGatewayName" . }} + namespace: {{ include "openshell.gatewayApiGatewayNamespace" . }} + {{- if .Values.gatewayApi.routes.tls.hostnames }} + hostnames: + {{- toYaml .Values.gatewayApi.routes.tls.hostnames | nindent 4 }} + {{- end }} + rules: + - backendRefs: + - name: {{ include "openshell.fullname" . }} + port: {{ .Values.service.port }} +{{- end }} diff --git a/deploy/helm/openshell/tests/gateway_api_test.yaml b/deploy/helm/openshell/tests/gateway_api_test.yaml new file mode 100644 index 000000000..448afbf6a --- /dev/null +++ b/deploy/helm/openshell/tests/gateway_api_test.yaml @@ -0,0 +1,203 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: Gateway API resources +templates: + - templates/gateway-config.yaml + - templates/gateway.yaml + - templates/grpcroute.yaml + - templates/statefulset.yaml + - templates/tlsroute.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: does not render Gateway API resources by default + template: templates/gateway.yaml + asserts: + - hasDocuments: + count: 0 + + - it: does not render a GRPCRoute by default + template: templates/grpcroute.yaml + asserts: + - hasDocuments: + count: 0 + + - it: does not render a TLSRoute by default + template: templates/tlsroute.yaml + asserts: + - hasDocuments: + count: 0 + + - it: renders a Gateway independently with a fixed HTTP listener + template: templates/gateway.yaml + set: + gatewayApi.gateway.create: true + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell + - equal: + path: metadata.namespace + value: my-namespace + - equal: + path: spec.gatewayClassName + value: eg + - equal: + path: spec.listeners[0].name + value: http + - equal: + path: spec.listeners[0].port + value: 80 + - equal: + path: spec.listeners[0].protocol + value: HTTP + - notExists: + path: spec.listeners[0].allowedRoutes + - notExists: + path: spec.listeners[0].tls + + - it: does not render a Gateway when only TLSRoute is enabled + template: templates/gateway.yaml + set: + gatewayApi.routes.tls.enabled: true + asserts: + - hasDocuments: + count: 0 + + - it: adds a fixed TLS passthrough listener to a chart-created Gateway + template: templates/gateway.yaml + set: + gatewayApi.gateway.create: true + gatewayApi.routes.tls.enabled: true + asserts: + - hasDocuments: + count: 1 + - equal: + path: spec.listeners[0].name + value: http + - equal: + path: spec.listeners[1].name + value: tls + - equal: + path: spec.listeners[1].port + value: 443 + - equal: + path: spec.listeners[1].protocol + value: TLS + - equal: + path: spec.listeners[1].tls.mode + value: Passthrough + - notExists: + path: spec.listeners[1].hostname + - notExists: + path: spec.listeners[1].tls.certificateRefs + + - it: renders a GRPCRoute independently for a pre-existing Gateway + template: templates/grpcroute.yaml + set: + gatewayApi.routes.grpc.enabled: true + gatewayApi.gateway.name: external-gateway + gatewayApi.gateway.namespace: ingress-system + gatewayApi.routes.grpc.hostnames: + - openshell.example.com + asserts: + - hasDocuments: + count: 1 + - equal: + path: spec.parentRefs[0].name + value: external-gateway + - equal: + path: spec.parentRefs[0].namespace + value: ingress-system + - equal: + path: spec.hostnames[0] + value: openshell.example.com + - equal: + path: spec.rules[0].backendRefs[0].name + value: openshell + - equal: + path: spec.rules[0].backendRefs[0].port + value: 8080 + + - it: renders a TLSRoute independently for a pre-existing Gateway + template: templates/tlsroute.yaml + set: + gatewayApi.routes.tls.enabled: true + gatewayApi.gateway.name: external-gateway + gatewayApi.gateway.namespace: ingress-system + gatewayApi.routes.tls.hostnames: + - openshell.example.com + asserts: + - hasDocuments: + count: 1 + - equal: + path: apiVersion + value: gateway.networking.k8s.io/v1 + - equal: + path: kind + value: TLSRoute + - equal: + path: spec.parentRefs[0].name + value: external-gateway + - equal: + path: spec.parentRefs[0].namespace + value: ingress-system + - equal: + path: spec.hostnames[0] + value: openshell.example.com + - equal: + path: spec.rules[0].backendRefs[0].name + value: openshell + - equal: + path: spec.rules[0].backendRefs[0].port + value: 8080 + + - it: renders the Gateway and GRPCRoute together using shared identity + template: templates/gateway.yaml + set: + gatewayApi.gateway.create: true + gatewayApi.gateway.name: shared-gateway + gatewayApi.routes.grpc.enabled: true + asserts: + - equal: + path: metadata.name + value: shared-gateway + - equal: + path: metadata.namespace + value: my-namespace + + - it: uses shared Gateway identity when both resources are enabled + template: templates/grpcroute.yaml + set: + gatewayApi.gateway.create: true + gatewayApi.gateway.name: shared-gateway + gatewayApi.routes.grpc.enabled: true + asserts: + - equal: + path: spec.parentRefs[0].name + value: shared-gateway + - equal: + path: spec.parentRefs[0].namespace + value: my-namespace + + - it: fails with guidance when legacy grpcRoute values are supplied + template: templates/statefulset.yaml + set: + grpcRoute.enabled: true + asserts: + - failedTemplate: + errorPattern: "grpcRoute values were replaced by gatewayApi" + + - it: rejects TLS passthrough when gateway server TLS is disabled + template: templates/statefulset.yaml + set: + gatewayApi.routes.tls.enabled: true + server.disableTls: true + asserts: + - failedTemplate: + errorPattern: "gatewayApi.routes.tls.enabled requires server.disableTls=false" diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index fd73dd071..df09cc93a 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -352,44 +352,33 @@ certManager: serverIpAddresses: - 127.0.0.1 -# Kubernetes Gateway API - HTTPRoute and Gateway resources. +# Kubernetes Gateway API resources. # Requires a Gateway API controller in the cluster. Install Envoy Gateway via # the skaffold.yaml releases or independently: # helm install eg oci://docker.io/envoyproxy/gateway-helm \ -# --version v1.4.1 -n envoy-gateway-system --create-namespace -grpcRoute: - # -- Create a Gateway API GRPCRoute for the gateway service. - enabled: false - # -- Hostnames the GRPCRoute matches on. Leave empty to match all hosts. - hostnames: [] +# --version v1.8.1 -n envoy-gateway-system --create-namespace +gatewayApi: gateway: - # -- When true, a Gateway resource is created in the release namespace. + # -- When true, a Gateway resource is created. # Set to false and provide name/namespace to attach to a pre-existing Gateway. create: false # -- GatewayClass to reference. Envoy Gateway installs one named "eg". className: "eg" # -- Name of the Gateway resource. Defaults to the chart fullname. name: "" - # -- Namespace of the Gateway referenced by the GRPCRoute parentRef. - # Defaults to the release namespace. + # -- Namespace of a pre-existing Gateway referenced by routes. Defaults to + # the release namespace. Chart-created Gateways always use the release namespace. namespace: "" - # Listener settings (only used when gateway.create is true). - listener: - # -- Listener port for the generated Gateway resource. Use 443 with protocol HTTPS. - port: 80 - # -- Listener protocol for the generated Gateway resource: HTTP or HTTPS. - # HTTPS terminates TLS at the Envoy Gateway listener; pair it with - # server.disableTls=true so Envoy forwards plaintext to the gateway pod, - # and use OIDC for client identity (the gateway never sees the client cert). - protocol: HTTP - # -- "Same" restricts attached routes to the release namespace; "All" allows any namespace. - allowedRoutes: Same - # TLS settings for the listener. Used only when protocol is HTTPS - # (mode is always Terminate). - tls: - # -- certificateRefs for the HTTPS listener. Required when protocol is - # HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret - # in the Gateway's namespace. May reference a cert-manager-issued Secret - # or the existing openshell-server-tls Secret (its SANs must include the - # external hostname). - certificateRefs: [] + routes: + grpc: + # -- Create a Gateway API GRPCRoute for the gateway service. + enabled: false + # -- Hostnames the GRPCRoute matches on. Leave empty to match all hosts. + hostnames: [] + tls: + # -- Create a Gateway API TLSRoute that passes encrypted traffic through + # to the gateway service. When gatewayApi.gateway.create is true, the + # generated Gateway also gets a TLS listener on port 443 in Passthrough mode. + enabled: false + # -- SNI hostnames the TLSRoute matches on. Leave empty to match all hosts. + hostnames: [] diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 844167246..667d0ca1c 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -3,14 +3,14 @@ # SPDX-License-Identifier: Apache-2.0 title: "Ingress" sidebar-title: "Ingress" -description: "Expose the OpenShell gateway externally using the Kubernetes Gateway API and a GRPCRoute." -keywords: "Generative AI, Cybersecurity, Kubernetes, Gateway API, Envoy Gateway, GRPCRoute, Ingress, External Access" +description: "Expose the OpenShell gateway externally using Kubernetes Gateway API routes." +keywords: "Generative AI, Cybersecurity, Kubernetes, Gateway API, Envoy Gateway, GRPCRoute, TLSRoute, TLS Passthrough, Ingress, External Access" position: 3 --- By default, the OpenShell gateway is only reachable inside the cluster. To let CLI clients connect without a `kubectl port-forward`, expose the gateway through an ingress. -OpenShell uses the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) for ingress. The chart creates a `GRPCRoute` that routes inbound gRPC traffic to the gateway pod. You need a Gateway API implementation installed on your cluster to fulfill the `GRPCRoute`. This page uses [Envoy Gateway](https://gateway.envoyproxy.io), which the chart is tested with. +OpenShell uses the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) for ingress. The chart can create a `GRPCRoute` for plaintext traffic or a `TLSRoute` for TLS passthrough to the gateway pod. You need a Gateway API implementation installed on your cluster to fulfill these routes. This page uses [Envoy Gateway](https://gateway.envoyproxy.io), which the chart is tested with. ## Install Envoy Gateway @@ -57,9 +57,9 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set grpcRoute.enabled=true \ - --set grpcRoute.gateway.create=true \ - --set grpcRoute.gateway.className=eg + --set gatewayApi.routes.grpc.enabled=true \ + --set gatewayApi.gateway.create=true \ + --set gatewayApi.gateway.className=eg ``` ## Get the external address @@ -77,63 +77,39 @@ openshell gateway add http:// --name production openshell status ``` -This setup is plaintext end-to-end and is intended for development. For external access, terminate TLS at the gateway as shown below. +This setup is plaintext end-to-end and is intended for development. -## HTTPS (TLS termination) +## Enable TLS passthrough -Envoy Gateway can terminate TLS at the listener and forward plaintext to the OpenShell gateway pod: +TLS passthrough keeps the connection encrypted from the client to the OpenShell gateway server. Envoy Gateway reads the TLS Server Name Indication (SNI) to select the route but does not terminate TLS: ```text -client → HTTPS → Envoy Gateway (terminate TLS) → plaintext → openshell gateway pod +client → TLS → Envoy Gateway (SNI routing) → TLS → OpenShell gateway ``` -Envoy Gateway only terminates TLS here — it does not perform OIDC. Do not enable an Envoy Gateway OIDC `SecurityPolicy` in front of the gateway: that flow relies on browser redirects and cookies and cannot work with the OpenShell CLI or headless agents. Instead, the OpenShell gateway validates an OIDC bearer token that the client sends in the gRPC `authorization` metadata, which Envoy forwards untouched. - -Because Envoy terminates TLS, the OpenShell gateway never sees a client certificate, so client mTLS cannot provide identity on this path. Use OIDC bearer tokens for client identity instead. - -For headless agents and CI, the CLI obtains the token via the OAuth2 client-credentials grant (no browser): set `OPENSHELL_OIDC_CLIENT_SECRET` to the OAuth client secret before running `openshell gateway add`. The client id comes from `--oidc-client-id` (default `openshell-cli`); pass it explicitly if your IdP client uses a different id. Interactive human users get the browser-based Authorization Code + PKCE flow by default. - -### Provide a TLS certificate - -Create a `kubernetes.io/tls` Secret in the `openshell` namespace with the certificate for your external hostname: - -```shell -kubectl -n openshell create secret tls openshell-ingress-tls \ - --cert=tls.crt --key=tls.key -``` - -The Secret may also be issued by cert-manager, or you can reference the chart's existing `openshell-server-tls` Secret if its SANs include the external hostname. - -### Install with HTTPS termination - -Enable an HTTPS listener, point it at the Secret, disable gateway-pod TLS so Envoy forwards plaintext, and configure an OIDC issuer for client identity: +Enable TLSRoute and let the chart add a port 443 TLS listener to its Gateway: ```shell helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set grpcRoute.enabled=true \ - --set grpcRoute.gateway.create=true \ - --set grpcRoute.gateway.className=eg \ - --set grpcRoute.gateway.listener.protocol=HTTPS \ - --set grpcRoute.gateway.listener.port=443 \ - --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \ - --set server.disableTls=true \ - --set server.oidc.issuer=https:// \ - --set 'grpcRoute.hostnames[0]=' + --set gatewayApi.gateway.create=true \ + --set gatewayApi.gateway.className=eg \ + --set gatewayApi.routes.tls.enabled=true \ + --set 'gatewayApi.routes.tls.hostnames[0]=openshell.example.com' \ + --set 'pkiInitJob.serverDnsNames[0]=openshell.example.com' ``` -Keep the certificate Secret in the release namespace. Referencing a Secret in another namespace requires a `ReferenceGrant`. +The generated Gateway listener uses protocol `TLS` and mode `Passthrough`. It does not reference a certificate Secret because the OpenShell gateway server completes the TLS handshake. -### Register over HTTPS +Configure `pkiInitJob.serverDnsNames` before the built-in certificate generator creates the server Secret. The generator leaves an existing Secret unchanged during upgrades. For an existing installation, ensure `server.tls.certSecretName` contains a certificate whose SAN covers the external hostname. When cert-manager owns the certificate, add the hostname to `certManager.serverDnsNames` instead. -```shell -openshell gateway add https:// --name production --oidc-issuer https:// -openshell status -``` +TLS passthrough preserves the gateway's existing authentication mode, including mTLS. Clients must have the corresponding client certificate when `server.tls.clientCaSecretName` is configured. To use OIDC without client certificates, set `server.tls.clientCaSecretName` to an empty string and configure the server OIDC settings. + +To create only TLSRoute for a pre-existing Gateway, leave `gatewayApi.gateway.create=false` and set `gatewayApi.gateway.name` and `gatewayApi.gateway.namespace`. The existing Gateway must provide a compatible TLS passthrough listener and allow routes from the OpenShell release namespace. -See [Authentication](/kubernetes/setup) for OIDC issuer, audience, and roles configuration. +After the route is accepted and the external DNS record resolves to the Envoy Gateway address, register the HTTPS endpoint using the authentication mode configured for the server. ## SSH Relay