High Availability

Run two or more gateway replicas when the OpenShell control plane must remain available during a gateway pod failure or rolling update. Every replica can serve API requests. The replicas share persistent state through PostgreSQL and relay session-bound requests to the replica connected to each sandbox supervisor.

Requirements

An HA gateway deployment requires:

• Two or more schedulable Kubernetes nodes or failure domains.
• PostgreSQL reachable from every gateway pod.
• A Kubernetes Secret in the OpenShell namespace with the PostgreSQL connection URI in a key named uri.
• workload.kind: deployment. The default StatefulSet and per-pod SQLite database are intended for a single gateway replica.
• An ingress or load balancer that routes clients to the gateway Service. Refer to Ingress for a Gateway API configuration.
• PostgreSQL connection capacity for every replica, including pods that are starting or terminating during a rollout. Refer to Size PostgreSQL Connections.

The Helm chart rejects replicaCount values above 1 unless server.externalDbSecret is set. It also rejects a multi-replica StatefulSet unless workload.allowMultiReplicaStatefulSet is explicitly enabled. Use a Deployment for the standard HA configuration.

Create the PostgreSQL Secret

Provision PostgreSQL outside the OpenShell chart, then create the connection Secret. The chart does not install PostgreSQL.

kubectl create namespace openshell
kubectl -n openshell create secret generic openshell-postgres \
  --from-literal=uri='postgresql://openshell:<password>@<postgres-host>:5432/openshell'

Use your PostgreSQL provider’s required TLS parameters in the connection URI. Keep the Secret in the same namespace as the gateway release.

Configure Gateway Replicas

Create values-ha.yaml:

replicaCount: 2
workload:
  kind: deployment
server:
  externalDbSecret: openshell-postgres
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: openshell
            app.kubernetes.io/instance: openshell
        topologyKey: kubernetes.io/hostname

The anti-affinity rule places the two replicas on different nodes. The app.kubernetes.io/instance value must match the Helm release name. Change the topology key to a zone label when replicas must span availability zones. A required rule leaves a replica Pending when the cluster does not have enough matching failure domains; use preferred anti-affinity when scheduling the replica is more important than strict separation.

Install or upgrade OpenShell:

helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --values values-ha.yaml \
  --wait

The chart shares its retained sandbox JWT signing material, TLS material, and default credential-storage encryption key across the gateway pods. Do not create different copies of those Secrets for individual replicas.

Verify the Deployment

Wait for all gateway replicas to become ready:

kubectl -n openshell rollout status deployment/openshell
kubectl -n openshell get pods \
  -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \
  -o wide

Confirm that the client-facing Service and headless peer Service have ready endpoints:

kubectl -n openshell get service openshell openshell-peer
kubectl -n openshell get endpointslice \
  -l app.kubernetes.io/service-name=openshell
kubectl -n openshell get endpointslice \
  -l app.kubernetes.io/service-name=openshell-peer

If you changed the chart’s generated names with nameOverride or fullnameOverride, use the rendered Service names instead.

After registering the gateway with the CLI, verify that requests succeed:

openshell status
openshell sandbox list

Protect Voluntary Disruptions

Add a PodDisruptionBudget so a voluntary disruption, such as node maintenance, does not evict every ready gateway pod at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: openshell
  namespace: openshell
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell

Apply the manifest separately from the OpenShell chart. Match the namespace and release labels to your installation. A PodDisruptionBudget controls voluntary evictions; it does not protect against an unavailable node or failure domain.

How Peer Routing Works

A sandbox supervisor keeps one active control stream to one gateway replica. That replica records ownership in PostgreSQL. When Kubernetes sends a client request to another replica, the receiving gateway relays exec, forwarding, file-sync, service, and status traffic to the owner through the headless peer Service.

Peer RPCs use a projected, pod-bound ServiceAccount token with the openshell-gateway-peer audience. The receiving gateway validates the token with the Kubernetes TokenReview API, verifies the live pod identity and release labels, and authorizes only peer methods. When gateway TLS is enabled, peer connections also use the chart’s TLS materials.

You do not need session affinity at the ingress or client-facing Service.

Failure and Rollout Behavior

When a non-owner gateway pod stops, Kubernetes removes it from ready Service endpoints and new requests go to another replica. Existing requests through the stopped pod disconnect and must be retried.

When an owner gateway pod stops gracefully, it re-reads gateway membership and redirects its connected supervisors to the replicas that now own them, including replicas that started moments earlier. After a crash, supervisors reconnect through the gateway Service. Either way, they publish new ownership. Other replicas re-read ownership while waiting to relay a request. A request fails if no fresh, reachable owner appears before its normal deadline.

A stopping owner waits up to five seconds for a redirected supervisor to publish its new session before it marks the sandbox not ready, so a sandbox stays Ready through a graceful handoff. Supervisors reconnect with exponential backoff that restarts after each accepted session, so repeated gateway restarts do not lengthen later reconnects.

Established streams do not move between gateway replicas. An interrupted exec, forwarding, or service stream must reconnect. File sync retries with a new SSH session instead of resuming the interrupted byte stream. Rolling updates can temporarily concentrate supervisor sessions on the replicas that stayed up; client requests remain routable through peer relay.

Monitor Capacity

Each gateway replica exposes Prometheus metrics on port 9090 (service.metricsPort). Scrape every gateway pod, not the Service, because these signals describe one replica. Refer to Gateway Metrics for the full catalog, scrape configuration, and access control.

These queries assume that Prometheus labels each series with namespace and pod:

# Relay capacity used on each replica. Usually near 0; graph it and alert
# on rejections instead.
max by (namespace, pod) (openshell_server_relay_pending / openshell_server_relay_pending_capacity)
# Relays rejected for capacity. Alert on any increase.
sum by (namespace, pod, reason) (increase(openshell_server_relay_rejected_total[5m]))
# 99th percentile time for supervisors to claim a relay on each replica.
histogram_quantile(0.99, sum by (namespace, pod, le) (rate(openshell_server_relay_claim_duration_seconds_bucket[5m])))
# Completed or cancelled relay setup attempts, by replica, relay kind, and route.
sum by (namespace, pod, relay_kind, route) (rate(openshell_server_routed_request_attempts_total{operation="relay"}[5m]))
# Session skew across replicas. 1 means balanced. Starting and stopping pods
# raise it during rollouts.
max(openshell_server_supervisor_sessions)
  / clamp_min(avg(openshell_server_supervisor_sessions), 1)
# Failed peer requests as a share of attempts, by operation.
sum by (operation) (rate(openshell_server_routed_request_attempts_total{route="peer",outcome!="success"}[5m]))
  / sum by (operation) (rate(openshell_server_routed_request_attempts_total{route="peer"}[5m]))
# 99th percentile peer request latency across the fleet.
histogram_quantile(0.99, sum by (le, operation) (rate(openshell_server_peer_request_duration_seconds_bucket{outcome="success"}[5m])))
# 99th percentile mutation lock wait by scope. Watch scope="sandbox" during rollouts.
histogram_quantile(0.99, sum by (le, scope) (rate(openshell_server_mutation_lock_wait_seconds_bucket[5m])))
# Mutation lock timeouts by scope. Alert on any increase.
sum by (scope) (increase(openshell_server_mutation_lock_timeouts_total[10m]))
# 99th percentile time a mutation lock is held, by replica and scope.
# Conflicting acquisitions time out after 10 seconds.
histogram_quantile(0.99, sum by (namespace, pod, le, scope) (rate(openshell_server_mutation_lock_hold_seconds_bucket[5m])))
# Mutation lock acquisitions that failed without timing out. Timeouts do not
# include them, so alert on any increase here as well.
sum by (namespace, pod, scope) (increase(openshell_server_mutation_lock_errors_total[10m]))
# Lock-pool connections in use on each replica, PostgreSQL only. Values that
# stay near 1 mean the pool is close to full.
max by (namespace, pod) (max_over_time(openshell_server_mutation_lock_connections_in_use[5m]) / openshell_server_mutation_lock_connections_capacity)

Relay capacity is used on the replica that owns the sandbox’s supervisor session, including relays that other replicas request through peers, so adding replicas does not relieve a busy owner. A relay whose client gave up stays counted for up to about 40 seconds, the 10-second claim timeout plus the 30-second cleanup interval. Routed request counts include retries during rollouts.

Each supervisor that reconnects takes one or two short sandbox-scoped locks on the replica that receives it. When a gateway pod stops during a rollout, its supervisors reconnect to the other replicas, so watch openshell_server_mutation_lock_wait_seconds and openshell_server_mutation_lock_timeouts_total with scope="sandbox" on the receiving replicas. Rising waits or any timeout mean that those replicas receive reconnects faster than they can absorb them, or that slow compute driver, credential backend, middleware, or provider profile source calls during sandbox and provider changes hold their locks longer. Guarded work has no time limit of its own, so such a call holds its lock until it returns: openshell_server_mutation_lock_hold_seconds shows how long, and the replica logs mutation lock held longer than the lock wait timeout when it releases a lock held longer than 10 seconds.

Scale the Gateway

Scale the gateway by changing a fixed replica count or by letting a HorizontalPodAutoscaler choose one. In both cases, keep at least two ready replicas when availability must survive one gateway pod failure, and size PostgreSQL and the cluster nodes for the largest replica count.

Scale Manually

Change replicaCount in values-ha.yaml, then apply the release again:

helm upgrade openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --values values-ha.yaml \
  --wait

Autoscale with a HorizontalPodAutoscaler

Set autoscaling.enabled to render an autoscaling/v2 HorizontalPodAutoscaler for the gateway Deployment, or for a StatefulSet with workload.allowMultiReplicaStatefulSet. Add these values to values-ha.yaml:

resources:
  requests:
    cpu: 500m
    memory: 512Mi
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 6
  targetCPUUtilizationPercentage: 70

With autoscaling enabled, the chart stops setting spec.replicas and ignores replicaCount. Kubernetes computes CPU and memory utilization against the container request, so those targets need a matching resources.requests entry. A resources.limits entry also works, because Kubernetes copies a limit into an unset request. The chart rejects an autoscaling.maxReplicas above 1 without server.externalDbSecret. Each scale-down hands the removed replica’s supervisor sessions to the remaining replicas, so the default behavior removes at most one pod every two minutes after a five-minute stabilization window. Override it with autoscaling.behavior, or set autoscaling.behavior.scaleDown to null to drop the default. New replicas do not take existing sessions.

Check the autoscaler after you apply the release:

kubectl -n openshell get hpa openshell
kubectl -n openshell describe hpa openshell

Autoscale on Custom Metrics

The HPA can also scale on a per-pod metric that a metrics adapter publishes. The chart passes autoscaling.metrics to the HPA unchanged and does not install Prometheus or an adapter. A custom metric needs these pieces:

• Prometheus scrapes every gateway pod and labels each series with namespace and pod.
• A metrics adapter, such as prometheus-adapter, maps those labels to Kubernetes resources and publishes a per-pod metric through the custom.metrics.k8s.io API.
• autoscaling.metrics references that metric as a Pods metric with an AverageValue target.

Scale on signals that fall when you add replicas, such as CPU utilization. Do not scale on supervisor sessions, relay utilization, relay claim latency, or peer request rate. New replicas do not take existing sessions, relay load stays on the replica that owns each session, and peer traffic grows with the replica count. Alert on those signals instead.

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 6
  targetCPUUtilizationPercentage: 70
  metrics:
    - type: Pods
      pods:
        metric:
          name: <metric-name>
        target:
          type: AverageValue
          averageValue: "100"

Replace <metric-name> with the name your adapter publishes, and set averageValue to the per-pod value at which you want another replica. Check that the adapter serves the metric for the gateway pods:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/openshell/pods/*/<metric-name>"

Size PostgreSQL Connections

With the default settings, each gateway pod opens up to 14 PostgreSQL connections on demand: 10 for data access (database_max_connections) and 4 for mutation locks (database_lock_max_connections), both set under gatewayConfig.openshell.gateway. A request cancelled during a lock wait can leave its lock session open for up to 10 seconds, so keep headroom. The chart uses the Kubernetes default rolling update strategy for each workload kind, so the number of pods that run during a rollout depends on the kind.

A Deployment rollout adds up to 25 percent of the replica count, rounded up, as surge pods. Kubernetes does not count terminating pods against that surge, and each replaced pod can keep its connections throughout its termination grace period. A rollout can therefore run up to twice the replica count at once, and an eviction or an autoscaler scale-down during the rollout adds more terminating pods. Set PostgreSQL max_connections to at least (2 × replicas + surge) × (data + lock connections), which leaves one surge of margin for those pods, plus headroom for your other clients and administration. Use autoscaling.maxReplicas as the replica count when autoscaling is enabled. With the defaults, that is (2 × replicas + surge) × 14:

With a Deployment, three or more replicas exceed what the PostgreSQL default of 100 leaves for the gateway, because PostgreSQL reserves 3 connections for superusers. Raise max_connections or use a larger managed instance. Let one rollout finish before you start another, because each overlapping rollout adds its own terminating pods.

If you run several replicas as a StatefulSet with workload.allowMultiReplicaStatefulSet, a rollout adds no surge pods. It replaces one pod at a time and creates each replacement only after the old pod exits. Set max_connections to at least replicas × (data + lock connections), 14 per replica with the defaults, using autoscaling.maxReplicas when autoscaling is enabled, plus headroom for your other clients and administration.

Raise database_lock_max_connections only when mutation lock timeouts on a replica come from a full lock pool: the gateway logs them with the detail waiting for a mutation lock connection, and openshell_server_mutation_lock_connections_in_use stays at openshell_server_mutation_lock_connections_capacity. Also check that the replica’s own acquisitions are not mostly waiting behind another holder: few of its timeouts carry the detail waiting for a PostgreSQL advisory lock or canceling statement due to lock timeout, and pg_locks shows the pod’s lock sessions granted rather than waiting. Otherwise, find the slow holder first, because a larger pool only queues more waiters behind it. Each extra lock connection adds 2 × replicas + surge to the minimum max_connections for a Deployment.

If a connection pooler sits between the gateway and PostgreSQL, use session pooling. The gateway holds session-level advisory locks, which transaction pooling breaks.

Mutations lock only what they change. Sandbox operations lock their own sandbox, provider and workspace-profile changes lock their workspace, and gateway-global policy, settings, and platform-profile changes lock the whole fleet. Sandbox start, stop, and delete lock only on the replica that runs them, and only a gateway-global change on that replica blocks them directly. Provider, workspace-profile, and other replicas’ gateway-global changes delay them only while another operation on the same sandbox waits behind the change. Sandbox create, start, restart, and delete also take one fleet-wide lock while they provision or remove the sandbox’s SSH host identity, so that step runs for one sandbox at a time across all replicas. For other mutations, a lock wait longer than 10 seconds returns UNAVAILABLE with the reason MUTATION_LOCK_TIMEOUT, and clients can retry the request. A request that carried a request_id leaves its admission unresolved, like any other error, so a retry with the same ID returns REQUEST_OUTCOME_UNCERTAIN. Observe resource state and reconcile effects, then send a new request with a new request_id. Refer to Durable Request Admission.

Upgrade from an Earlier Release

With the default settings, each gateway pod now opens up to 14 PostgreSQL connections instead of 10. The upgrade rollout already runs new pods at that count, so raise max_connections to the minimum in Size PostgreSQL Connections before you upgrade.

While gateways from the earlier release and the new release run together during the upgrade, the older replicas serialize every mutation across the fleet. Mutations can be slower and can briefly fail under load until the rollout finishes.

Next Steps

• To expose the gateway through a highly available data path, refer to Ingress.
• To configure automatic certificate renewal, refer to Managing Certificates.
• To configure user authentication and authorization, refer to Access Control.
• To scrape and alert on gateway metrics, refer to Gateway Metrics.