Kubernetes

Set Up OpenShell on Kubernetes

View as Markdown

Your cluster MUST use a CNI that enforces Kubernetes NetworkPolicy for both ingress and egress in every sandbox namespace. OpenShell creates the policies, but Kubernetes accepts them even when no CNI enforces them. Without enforcement, sandbox workloads may reach the network directly and bypass supervisor policy. Verify CNI support before installing OpenShell.

OpenShell on Kubernetes: the gateway provisions a Sandbox resource, separate supervisor and workload Pods, a boundary Service, NetworkPolicy, and a workspace PVC.

The gateway’s Kubernetes compute driver creates a Sandbox resource and a separate supervisor Pod. The Agent Sandbox controller creates the workload Pod from that resource. A Service gives the supervisor a stable address for the workload’s authenticated TLS boundary, and a PVC preserves /sandbox across Pod restarts. The gateway can run in the same namespace as these resources or in a separate namespace.

The supervisor opens the boundary connection and maintains an outbound session to the gateway. A NetworkPolicy allows only supervisor ingress to the workload’s boundary port and denies new workload-initiated connections. Agent network requests return on the established channel; the supervisor evaluates policy and opens approved upstream connections. For the trust boundaries and request flow, see Architecture.

Prerequisites

Make sure the following are in place before you install.

PrerequisiteRequiredNotes
Kubernetes 1.29+ with RBAC enabledYesNo additional notes.
CNI that enforces ingress and egress NetworkPolicy in sandbox namespacesYesVerify enforcement on your cluster.
Helm 3.xYesNo additional notes.
Agent Sandbox controller and CRDsYesInstall before the OpenShell chart. Refer to Install Agent Sandbox.
cert-managerNoRefer to Managing Certificates. Use cert-manager only if you prefer it over the built-in PKI job.
Kubernetes Gateway APINoRefer to Ingress. Use it only for external access without port-forwarding.

Install Agent Sandbox

OpenShell uses the Agent Sandbox Kubernetes SIG project to provision sandbox pods. Install the Agent Sandbox controller and its CRDs on your cluster before installing the OpenShell Helm chart.

Apply the latest release manifest:

kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml

This creates the agent-sandbox-system namespace, installs the sandboxes.agents.x-k8s.io CRD, and starts the controller.

The Helm chart checks for a supported Agent Sandbox API before it creates gateway resources. This preflight is enabled by default. Disable it only for offline helm template rendering, where Helm cannot discover cluster APIs:

helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --set agentSandbox.preflight.enabled=false

The chart does not install or upgrade the cluster-scoped Agent Sandbox CRDs or controller.

Air-gapped clusters: mirror the manifest above and the registry.k8s.io/agent-sandbox/agent-sandbox-controller image referenced inside it to your internal registry, then point the manifest’s image reference at your mirror before applying. Mirror the OpenShell gateway, supervisor, and trusted sandbox runtime images, then set global.image.registry to the mirror. Configure the separate default workload sandbox image with sandbox.image when needed.

Confirm the controller pod is running before proceeding:

kubectl -n agent-sandbox-system get pods

The controller pod should reach Running status within a few seconds. For cluster-specific setup instructions, including KinD and GKE walkthroughs, refer to the Agent Sandbox getting started guide.

Upgrade Agent Sandbox

OpenShell detects the served Agent Sandbox Sandbox API when the Kubernetes gateway first needs it and caches that choice for the gateway process. If you upgrade Agent Sandbox in place, restart the OpenShell gateway after the Agent Sandbox controller and CRD rollout completes so the gateway can detect the served API versions again. Existing sandboxes keep running during the upgrade, and the restarted gateway can continue managing them.

Install OpenShell

1

Create the namespace

kubectl create namespace openshell
2

Install the chart

Install from the OCI registry on GHCR. Replace <version> with the chart version you want to install.

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell

To use the latest development build instead of a stable release:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version 0.0.0-dev \
  --namespace openshell

The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required.

Split gateway and workspace releases

For a platform-managed namespace, install the gateway without namespace-scoped sandbox resources, then install the workspace chart in the sandbox namespace:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set workspaceResources.enabled=false \
  --set server.sandboxNamespace=app-a
helm upgrade --install openshell-workspace \
  oci://ghcr.io/nvidia/openshell/openshell-workspace \
  --version <version> \
  --namespace app-a \
  --set gateway.serviceAccount.name=openshell \
  --set gateway.serviceAccount.namespace=openshell

The workspace chart does not create the namespace or deploy a gateway. It owns only the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy in its release namespace. For one pre-provisioned namespace, keep server.drivers.kubernetes.workspaceMode=shared and set server.sandboxNamespace=app-a. To map multiple workspaces to separately provisioned namespaces, use workspaceMode=operator, configure exactly one of operatorNamespaceLabel or operatorNamespaceFile, and install the workspace chart in every allowlisted namespace. In operator mode, the workspace chart Role is the gateway’s only Secret permission grant in that namespace.

If you enable server.drivers.kubernetes.allowDriverConfig on the gateway, also set gateway.allowDriverConfig=true on every workspace release. This grants the namespace-scoped PVC metadata read needed to admit caller-selected PVC mounts. Leave both values disabled when callers do not need driver config.

Store provider credentials in Kubernetes Secrets

To store provider credentials as Kubernetes Secrets instead of in the gateway database, enable the Kubernetes Secrets credential driver with a dedicated namespace:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set server.credentialDrivers.kubernetesSecrets.enabled=true \
  --set server.credentialDrivers.kubernetesSecrets.namespace=openshell-credentials \
  --set server.credentialDrivers.kubernetesSecrets.createNamespace=true

The driver stores every credential in that namespace, in every workspace mode. The gateway receives Secret permissions through a Role in that namespace only. For the full set of options, refer to the gateway configuration reference.

3

Wait for the gateway to be ready

kubectl -n openshell rollout status statefulset/openshell

If you set workload.kind=deployment, wait on the Deployment instead:

kubectl -n openshell rollout status deployment/openshell
4

Connect to the gateway

For local evaluation, use a port-forward:

kubectl -n openshell port-forward svc/openshell 8080:8080

The port-forward is for local evaluation only. For shared environments, expose the gateway through your ingress controller or access proxy. Refer to Ingress for an external access option.

5

Install the TLS client bundle

The chart generates an mTLS bundle for transport security. Kubernetes deployments do not use that bundle as user authentication; configure OIDC or a trusted access proxy as described in Access Control. For local port-forwarded access, copy the generated bundle so the CLI can verify the gateway certificate:

mkdir -p ~/.config/openshell/gateways/k8s/mtls
kubectl -n openshell get secret openshell-client-tls \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/k8s/mtls/ca.crt
kubectl -n openshell get secret openshell-client-tls \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/k8s/mtls/tls.crt
kubectl -n openshell get secret openshell-client-tls \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/k8s/mtls/tls.key

The server certificate SANs include localhost and 127.0.0.1, so hostname verification passes over the port-forward without extra flags.

6

Register with the CLI

In another terminal, register the gateway with the user authentication mode you configured and verify it is reachable. For example, with OIDC:

openshell gateway add https://127.0.0.1:8080 --local --name k8s \
  --oidc-issuer https://your-idp.example.com/realms/openshell \
  --oidc-client-id openshell-cli
openshell status

Configure Chart Values

The most commonly changed values are:

ValuePurpose
global.image.registry / global.image.tag / global.image.pullPolicyShared registry, tag, and pull policy for the gateway, supervisor, and trusted sandbox runtime. Individual image settings take precedence.
gateway.image.registry / gateway.image.repository / gateway.image.tag / gateway.image.digestGateway container image. Defaults to ghcr.io/nvidia/openshell/gateway:latest.
replicaCountNumber of gateway replicas. Values above 1 require shared PostgreSQL through server.externalDbSecret.
workload.kindGateway workload controller. Use statefulset for SQLite or deployment with server.externalDbSecret.
workload.allowMultiReplicaStatefulSetAllow replicaCount > 1 with workload.kind=statefulset. Prefer Deployment for external database-backed multi-replica gateways.
server.sandboxNamespaceNamespace where sandbox pods are created. Defaults to the Helm release namespace when left empty.
workspaceResources.enabledCreate namespace-scoped sandbox prerequisites from the gateway chart. Disable when installing the workspace chart separately.
server.externalDbSecretSecret containing a PostgreSQL connection URI in the uri key. Use when the database is managed outside the chart.
server.telemetryEnabledEnable anonymous OpenShell telemetry from the gateway and its sandbox supervisors. Set to false to opt out.
sandbox.image.repository / sandbox.image.tag / sandbox.image.digestDefault sandbox image used when a sandbox does not specify one.
sandboxRuntime.image.registry / sandboxRuntime.image.repository / sandboxRuntime.image.tag / sandboxRuntime.image.digestTrusted workload-side image that provides the openshell-sandbox binary. A digest takes precedence over the tag.
supervisor.image.registry / supervisor.image.repository / supervisor.image.tag / supervisor.image.digestTrusted control-side supervisor image.
server.sandboxImagePullSecretsImage pull secrets attached to sandbox pods. Referenced Secrets must exist in the sandbox namespace.
server.grpcEndpointEndpoint that sandbox supervisors use to call back to the gateway. Must be reachable from inside the cluster.
server.disableTlsRun the gateway over plaintext HTTP. Use only behind a trusted transport.
server.auth.allowUnauthenticatedUsersAccept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy.
server.enableLoopbackServiceHttpEnable local plaintext HTTP for loopback sandbox service URLs. Defaults to true.
pkiInitJob.serverDnsNames / certManager.serverDnsNamesAdditional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain.
supervisor.sandboxRuntime.boundaryPortNon-privileged TLS port used between paired supervisor and sandbox Pods.
upstreamProxyOperator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to Configure a Corporate Upstream Proxy.

Use a values file for repeatable deployments:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --values my-values.yaml

To use private sandbox images, create a kubernetes.io/dockerconfigjson Secret in the sandbox namespace and reference its name:

kubectl -n openshell create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username="$REGISTRY_USER" \
  --docker-password="$REGISTRY_TOKEN"
server:
  sandboxImage: registry.example.com/team/openshell-sandbox:latest
  sandboxImagePullSecrets:
    - name: regcred

Configure a Corporate Upstream Proxy

Configure a corporate forward proxy when sandbox TLS egress cannot dial the Internet directly. OpenShell evaluates policy and SSRF checks before it opens an HTTP CONNECT tunnel through the proxy. The proxy URL is operator-owned configuration. Sandbox environment variables cannot select, replace, or bypass it.

Create the credential Secret in the sandbox namespace when the proxy requires Basic authentication. The Secret value uses the user:pass form.

kubectl -n openshell create secret generic corporate-proxy-auth \
  --from-literal=credentials="$PROXY_USER:$PROXY_PASSWORD"

Add the proxy settings to your Helm values file. Replace the DNS suffixes and CIDRs in noProxy with values for your cluster. noProxy bypasses only the corporate proxy. OpenShell policy evaluation still applies.

upstreamProxy:
  url: http://proxy.corp.example:8080
  noProxy: .svc,.svc.cluster.local,10.96.0.0/12,10.244.0.0/16
  authSecret:
    name: corporate-proxy-auth
    key: credentials
  authAllowInsecure: true

Use authAllowInsecure: true only when you accept that Basic authentication is cleartext on the connection to an http:// proxy. The initial release supports http:// proxy endpoints and TLS CONNECT egress. It does not support HTTPS-to-proxy, custom corporate CA bundles, or forwarding plain HTTP egress through the proxy.

The credential mounts only in the separately scheduled supervisor Pod. The sandbox workload cannot read it through its environment or volumes.

RBAC

The chart creates the following RBAC resources in the release namespace:

ResourceScopeName
ServiceAccountNamespaceopenshell
ServiceAccountNamespaceopenshell-sandbox (for sandbox pods)
Role + RoleBindingNamespaceopenshell-sandbox
ClusterRole + ClusterRoleBindingClusteropenshell-node-reader

When the Kubernetes Secrets credential driver is enabled, the chart also creates an openshell-credential-secrets Role and RoleBinding in the credential namespace. That Role grants get, create, patch, and delete on all Secrets in the credential namespace, and is the gateway’s only permission grant for provider credential Secrets.

In managed and operator workspace modes, the chart creates an openshell-workspace-secret-source Role and RoleBinding in the sandbox namespace. That Role grants get on only the gateway client TLS Secret and, in managed mode, the configured image-pull Secrets. The gateway stages their contents into each sandbox runtime generation’s Secrets in the workspace namespace.

The namespaced Role covers sandbox lifecycle and identity:

API GroupResourceVerbs
agents.x-k8s.iosandboxes, sandboxes/statuscreate, delete, get, list, patch, update, watch
""eventsget, list, watch
""podsget

The ClusterRole grants node inspection and token validation:

API GroupResourceVerbs
authentication.k8s.iotokenreviewscreate
""nodesget, list, watch

To use an existing ServiceAccount instead of creating one, set serviceAccount.create=false and supply its name:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set serviceAccount.create=false \
  --set serviceAccount.name=my-existing-sa

The ServiceAccount must already have the Role and ClusterRole bindings described above.

Probes

The gateway exposes /healthz for process liveness and /readyz for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes:

  • startupProbe and livenessProbe use /healthz.
  • readinessProbe uses /readyz, which reflects the latest result of an in-process background database check.

Next Steps