Skip to navigation
Kubernetes

Ingress

View as Markdown

By default, the OpenShell gateway is only reachable inside the cluster. To let CLI clients connect without a kubectl port-forward, expose the gateway through an ingress.

OpenShell uses the Kubernetes Gateway API for ingress. The chart creates a GRPCRoute that routes inbound gRPC traffic to the gateway pod. You need a Gateway API implementation installed on your cluster to fulfill the GRPCRoute. The examples use Envoy Gateway, but you can select another implementation through grpcRoute.gateway.className.

Install Envoy Gateway

Envoy Gateway installs the Gateway API CRDs and controller:

helm install eg \
  oci://docker.io/envoyproxy/gateway-helm \
  --version v1.8.1 \
  --namespace envoy-gateway-system \
  --create-namespace \
  --wait

Create the GatewayClass

Create the eg GatewayClass that the OpenShell chart references:

kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

Verify the GatewayClass is accepted:

kubectl get gatewayclass eg

The ACCEPTED column should show True.

Install OpenShell with Gateway API enabled

Enable the GRPCRoute and let the chart create a Gateway resource in the openshell namespace:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.gateway.create=true \
  --set grpcRoute.gateway.className=eg

The generated Gateway defaults to the chart fullname.

Use another GatewayClass

Set grpcRoute.gateway.className to any GatewayClass installed by your Gateway API implementation:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.gateway.create=true \
  --set grpcRoute.gateway.className=<gateway-class> \
  --set grpcRoute.gateway.name=openshell-ingress

Some Gateway API implementations name their proxy Service after the Gateway. Set a distinct Gateway name, such as openshell-ingress, when the default would conflict with the OpenShell Service. For ingress, the OpenShell chart creates only the Gateway and GRPCRoute. Install and manage the selected controller, GatewayClass, and Gateway API CRDs separately.

Use an existing Gateway

To attach OpenShell to a platform-managed Gateway, disable Gateway creation and set the parent Gateway name and namespace. Set sectionName when the route should attach to a specific listener:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.gateway.create=false \
  --set grpcRoute.gateway.name=platform-gateway \
  --set grpcRoute.gateway.namespace=gateway-system \
  --set grpcRoute.gateway.sectionName=grpc

The existing Gateway listener must allow routes from the OpenShell release namespace.

Get the external address

With the Envoy Gateway example above, the controller creates a LoadBalancer service in the openshell namespace. Wait for it to get an external address:

kubectl -n openshell get svc -l gateway.envoyproxy.io/owning-gateway-name=openshell

For another Gateway API implementation or an existing Gateway, follow that implementation’s instructions to find the external address.

After the EXTERNAL-IP is assigned, register the gateway with the CLI:

openshell gateway add http://<external-ip> --name production
openshell status

This setup is plaintext end-to-end and is intended for development. For external access, terminate TLS at the gateway as shown below.

HTTPS (TLS termination)

Envoy Gateway can terminate TLS at the listener and forward plaintext to the OpenShell gateway pod:

client → HTTPS → Envoy Gateway (terminate TLS) → plaintext → openshell gateway pod

Envoy Gateway only terminates TLS here — it does not perform OIDC. Do not enable an Envoy Gateway OIDC SecurityPolicy in front of the gateway: that flow relies on browser redirects and cookies and cannot work with the OpenShell CLI or headless agents. Instead, the OpenShell gateway validates an OIDC bearer token that the client sends in the gRPC authorization metadata, which Envoy forwards untouched.

Because Envoy terminates TLS, the OpenShell gateway never sees a client certificate, so client mTLS cannot provide identity on this path. Use OIDC bearer tokens for client identity instead.

For an interactive login on a headless machine, set OPENSHELL_NO_BROWSER=1 before running openshell gateway add. The CLI uses the Device Authorization Grant with S256 PKCE and prompts the user to approve the login from another browser. For unattended agents and CI, set OPENSHELL_OIDC_CLIENT_SECRET to use the OAuth2 client-credentials grant instead. The client id comes from --oidc-client-id (default openshell-cli); pass it explicitly if your identity provider uses a different id. Interactive users with a local browser get the Authorization Code flow with PKCE by default.

Provide a TLS certificate

Create a kubernetes.io/tls Secret in the openshell namespace with the certificate for your external hostname:

kubectl -n openshell create secret tls openshell-ingress-tls \
  --cert=tls.crt --key=tls.key

The Secret may also be issued by cert-manager, or you can reference the chart’s existing openshell-server-tls Secret if its SANs include the external hostname.

Install with HTTPS termination

Enable an HTTPS listener, point it at the Secret, disable gateway-pod TLS so Envoy forwards plaintext, and configure an OIDC issuer for client identity:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set grpcRoute.enabled=true \
  --set grpcRoute.gateway.create=true \
  --set grpcRoute.gateway.className=eg \
  --set grpcRoute.gateway.listener.protocol=HTTPS \
  --set grpcRoute.gateway.listener.port=443 \
  --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \
  --set server.disableTls=true \
  --set 'gatewayConfig.openshell\.gateway\.oidc.issuer=https://keycloak.example.com/realms/openshell' \
  --set 'gatewayConfig.openshell\.gateway\.oidc.audience=openshell-cli' \
  --set 'grpcRoute.hostnames[0]=gateway.example.com'

Keep the certificate Secret in the release namespace. Referencing a Secret in another namespace requires a ReferenceGrant.

Register over HTTPS

openshell gateway add https://gateway.example.com \
  --name production \
  --oidc-issuer https://keycloak.example.com/realms/openshell \
  --oidc-client-id openshell-cli
openshell status

See Authentication for OIDC issuer, audience, and roles configuration.

End-to-end TLS (BackendTLSPolicy)

As an alternative to the plaintext backend path above, the chart can create a BackendTLSPolicy that tells the Gateway proxy to re-encrypt traffic when connecting to the OpenShell gateway pod:

client → HTTPS → Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod

This keeps TLS on the gateway pod. The Gateway proxy validates the backend’s certificate against a CA ConfigMap that the certgen hook auto-creates.

BackendTLSPolicy is a standard Gateway API resource. It is supported on OpenShift 4.22+ (via the OpenShift gateway controller) and on other platforms where the Gateway API implementation supports it (check your controller’s documentation).

Install with e2e TLS

The certgen hook automatically creates the backend CA ConfigMap when backendTLSPolicy is enabled:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set server.tls.enableMtls=false \
  --set grpcRoute.enabled=true \
  --set grpcRoute.gateway.create=true \
  --set grpcRoute.gateway.className=eg \
  --set grpcRoute.gateway.listener.protocol=HTTPS \
  --set grpcRoute.gateway.listener.port=443 \
  --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \
  --set grpcRoute.backendTLSPolicy.enabled=true \
  --set 'gatewayConfig.openshell\.gateway\.oidc.issuer=https://keycloak.example.com/realms/openshell' \
  --set 'gatewayConfig.openshell\.gateway\.oidc.audience=openshell-cli' \
  --set 'grpcRoute.hostnames[0]=gateway.example.com'

The gateway pod continues to serve TLS, while server.tls.enableMtls=false disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both grpcRoute.backendTLSPolicy.enabled=true and server.tls.enableMtls=true simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured in gatewayConfig.openshell.gateway.oidc).

The example above uses the default pkiInitJob for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (--set certManager.enabled=true), the Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (<release>-certgen-backend-ca) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. This means a single helm install is sufficient in most cases.

If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with --set pkiInitJob.timeoutSeconds=<seconds>. The hook polls for exactly this many seconds. For example, timeoutSeconds=180 polls for 180 seconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional.

Troubleshooting

If you see the error remote connection failure, transport failure reason: TLS error: Secret is not supplied by SDS when connecting through the Gateway:

  1. Check if the backend CA ConfigMap exists:

    kubectl get configmap <release-name>-backend-ca -n <namespace>

  2. If the ConfigMap is missing, verify the TLS secret exists:

    kubectl get secret <release-name>-server-tls -n <namespace>

  3. If the secret exists but the ConfigMap doesn’t, run helm upgrade to create it:

    helm upgrade <release-name> oci://ghcr.io/nvidia/openshell/helm-chart \
      --reuse-values --namespace <namespace>

This situation can occur if you set pkiInitJob.failOnTimeout=false and cert-manager issued the certificate after the hook timed out.

For OpenShift 4.22+, see OpenShift for platform-specific instructions including Gateway and GatewayClass setup.

Replica Routing

With more than one gateway replica, a sandbox’s SSH, port-forward, and exec connections can land on a replica that does not hold the sandbox’s supervisor session, and that replica relays them to the one that does. Enable replica routing to send those connections straight to the owning replica:

helm upgrade <release-name> oci://ghcr.io/nvidia/openshell/helm-chart \
  --reuse-values --namespace <namespace> \
  --set workload.kind=statefulset \
  --set workload.allowMultiReplicaStatefulSet=true \
  --set grpcRoute.replicaRouting.enabled=true

The chart renders one Service per replica and a GRPCRoute rule that matches the x-openshell-replica request header. The CLI sets that header from the owner the gateway reports and retries without it if the named replica is unavailable. Replica routing requires workload.kind=statefulset, workload.allowMultiReplicaStatefulSet=true, and at most 15 replicas.

If the release uses workload.kind=deployment, this upgrade replaces the Deployment with a StatefulSet, so every gateway pod restarts at once. The StatefulSet also requests a 1Gi PersistentVolumeClaim per replica, which stays Pending on clusters without a default StorageClass.

SSH Relay

Sandbox SSH uses the gateway endpoint registered with the CLI. No separate Helm SSH host or port values are required.

Next Steps

Return to Setup to complete the installation.