Control Plane Operations

This section provides runbooks for operating the self-hosted NVCF control plane, including encryption key rotation, service management, and upgrades.

Encryption Key Management

Self-hosted NVCF uses a two-tier encryption hierarchy to protect secrets stored in the Encrypted Secret Store (ESS):

When a user stores a secret through the NVCF API, ESS encrypts it with the active NEK for that namespace. The NEK itself is stored in Cassandra, encrypted by the MEK. To decrypt a secret, ESS retrieves the NEK from Cassandra, decrypts it using the MEK from OpenBAO, then decrypts the secret.

Key Rotation Runbooks

• control-plane-runbook-mek-rotation. Rotate the master encryption key stored in OpenBAO.

Basic Operations

Service Reference

The following table lists all NVCF control plane services with their namespace, resource name, and resource type. Use these values in the commands throughout this section.

Pod Annotations

Set pod annotations for a service in environments/<environment-name>.yaml, for example to opt its pods into a log or metrics agent. Each key is passed to the chart unchanged; unset keys render nothing.

# environments/<environment-name>.yaml
api:
  podAnnotations:
    example.com/scrape: "true"

Annotation values must be strings, so quote numbers and booleans ("8080", "true"), and do not set keys the charts manage themselves, such as checksum/config. A change rolls the Deployments and NATS. OpenBao uses the OnDelete update strategy, so its pods pick up new annotations only when they are restarted. Keys starting with nvcf.nvidia.com/nats-worker-auth are reserved for the auth callout’s restart on a worker auth change, and setting them fails the render.

Restarting a Service

Restarting a Deployment:

kubectl rollout restart deployment/<name> -n <namespace>
# Example: restart the NVCF API
kubectl rollout restart deployment/nvcf-api -n nvcf
# Verify the rollout completes
kubectl rollout status deployment/<name> -n <namespace> --timeout=120s

Restarting a StatefulSet:

StatefulSets using RollingUpdate restart one pod at a time in reverse ordinal order. Check the update strategy before using the commands below. OpenBao uses OnDelete, so kubectl rollout restart alone does not replace its pods. For OpenBao 2.6.2 to 2.6.3, use the OpenBao patch upgrade runbook.

kubectl rollout restart statefulset/<name> -n <namespace>
# Example: restart Cassandra
kubectl rollout restart statefulset/cassandra -n cassandra-system
# Verify the rollout completes
kubectl rollout status statefulset/<name> -n <namespace> --timeout=300s

kubectl exec -n vault-system openbao-server-0 -c openbao -- sh -c \
  'BAO_ADDR=http://127.0.0.1:8200 bao status'

Restarting all Deployments in a namespace:

kubectl rollout restart deployment -n <namespace>
# Example: restart all services in the nvcf namespace
kubectl rollout restart deployment -n nvcf

Checking Service Health

List pods and their status:

# All pods in a namespace
kubectl get pods -n <namespace>
# All NVCF control plane pods at a glance
for ns in nvcf sis api-keys ess nats-system vault-system cassandra-system; do
  echo "=== $ns ==="
  kubectl get pods -n $ns
done

Check logs for a service:

# Recent logs
kubectl logs -n <namespace> -l app.kubernetes.io/name=<name> --tail=50
# Follow logs in real time
kubectl logs -n <namespace> -l app.kubernetes.io/name=<name> -f

Describe a pod for events and conditions:

kubectl describe pod -n <namespace> -l app.kubernetes.io/name=<name>

Scaling a Service

To temporarily take a service offline (for example, during maintenance), scale it to zero, perform the work, then scale it back:

# Scale down
kubectl scale deployment/<name> -n <namespace> --replicas=0
# ... perform maintenance ...
# Scale back up
kubectl scale deployment/<name> -n <namespace> --replicas=1
# Verify
kubectl rollout status deployment/<name> -n <namespace>

Upgrading Services

For a stack upgrade, follow the staged 1.0.2-to-1.0.4 procedure. It covers the stack combination in docs editions 1.0.2 and 1.0.3 and the required hostname-policy, NVCA and control-plane ordering. Older source stacks must complete their intermediate migration steps first.

The commands below are for targeted fixes to individual services. A spot upgrade bypasses the stack’s coordinated versions and migrations. Review the specific chart’s compatibility and rollback limits before using this path.

When to Spot Upgrade

Pre-Upgrade Checklist

Before upgrading any chart:

1. Note the current chart version and app version:

   helm list -n <namespace>

2. Back up the current Helm values:

   helm get values <release> -n <namespace> -o yaml > <release>-values-backup.yaml

3. Review release notes for the target version. Check for breaking changes, required value changes, or new dependencies.

4. Verify the cluster is healthy before starting: all pods running, no pending operations.

Spot Upgrading a Helm Chart

The following commands work for any Deployment-based service. Replace the placeholders with values from the [Service Reference] table above.

# 1. Upgrade the chart
helm upgrade <release> \
  oci://<registry>/<repository>/<chart> \
  --version <new-version> \
  --namespace <namespace> \
  --wait --timeout 5m \
  -f <release>-values.yaml
# 2. Verify the rollout
kubectl rollout status deployment/<name> -n <namespace> --timeout=120s
# 3. Confirm the new chart version
helm list -n <namespace>

Example: upgrading the NVCF API chart:

helm upgrade api \
  oci://${REGISTRY}/${REPOSITORY}/helm-nvcf-api \
  --version 2.1.0 \
  --namespace nvcf \
  --wait --timeout 5m \
  -f nvcf-api-values.yaml
kubectl rollout status deployment/api -n nvcf --timeout=120s

Upgrading StatefulSet-Based Services

Cassandra, NATS, and OpenBao are deployed as StatefulSets. The helm upgrade command is the same, but rollout depends on the update strategy. The following rolling-update description does not apply to OpenBao’s OnDelete strategy:

• Rolling update: StatefulSets restart pods one at a time in reverse ordinal order, waiting for each pod to become ready before proceeding to the next.
• Quorum preserved: For 3-replica clusters, at most one pod is unavailable at a time, maintaining quorum throughout the upgrade.

helm upgrade <release> \
  oci://<registry>/<repository>/<chart> \
  --version <new-version> \
  --namespace <namespace> \
  --wait --timeout 10m \
  -f <release>-values.yaml
kubectl rollout status statefulset/<name> -n <namespace> --timeout=300s

Service-specific notes:

Rolling Back

The commands below are not a data-downgrade procedure for OpenBao. Use a separately validated recovery plan for stateful data services.

If an upgrade causes issues, roll back to the previous Helm revision:

# List revision history
helm history <release> -n <namespace>
# Roll back to the previous revision
helm rollback <release> -n <namespace>
# Verify
kubectl rollout status deployment/<name> -n <namespace> --timeout=120s
# Or for StatefulSets
kubectl rollout status statefulset/<name> -n <namespace> --timeout=300s

Disabling Admin Issuer Proxy

Admin Issuer Proxy is a bootstrap helper. POST /v1/admin/keys returns a 12-hour admin JWT, signed by OpenBao as the NVCF API’s in-cluster issuer, carrying function, secret, registry-credential, cluster-group listing and cluster-management scopes.

It does not authenticate callers. The stack relies on the gateway in front of it for that.

It exists so a new install with no identity provider can mint its first admin token, and nvcf-cli init uses it.

An install whose callers already get tokens from their own identity provider has no use for it, and can remove it.

$ nvcf-cli init
Error: failed to generate admin token: API error 404: 
Usage:
  nvcf-cli init [flags]
...

export NVCF_TOKEN="<token from your identity provider>"
# or, to persist it, add to ~/.nvcf-cli.yaml:
#   token: "<token from your identity provider>"

Configuration

# environments/<environment-name>.yaml
adminIssuerProxy:
  enabled: false

This value controls two things that must stay in step:

The NVCF UI health list drops its Admin Issuer Proxy entry at the same time, so the UI does not report a missing component.

A related value, adminIssuerProxy.gateway.enabled: false, removes only the api-keys.<domain>/v1/admin/keys route and keeps the service reachable inside the cluster. Use it when nvcf-cli init is still needed from inside the cluster but the endpoint should not be on the public edge.

Setting adminIssuerProxy.enabled: false removes the route as well, so the two are not combined.

Removing it from an existing install

Setting the value to false on a stack where the proxy is already installed does not remove anything. The release condition stops Helmfile managing the release, and the migration stops creating the OpenBao grant, but both already exist and stay as they are.

Removal is two steps, and both are required.

1. Uninstall the release. This removes everything the chart owns: the Deployment, Service, ServiceAccount, HTTPRoute and ReferenceGrant, plus the PodDisruptionBudget if adminIssuerProxy.podDisruptionBudget.enabled is set.

helm uninstall admin-issuer-proxy -n api-keys
# Verify: nothing left in the namespace, and no route on the shared Gateway
kubectl get pods,deploy,svc,sa,pdb -n api-keys | grep admin-token-issuer-proxy
kubectl get httproute,referencegrant -A | grep admin-token-issuer-proxy

1. Revoke the OpenBao grant. This is the step that is easy to miss.

OpenBao state is not owned by the Helm release, so uninstalling leaves the signing capability in place: any pod running as the admin-issuer-proxy service account in the api-keys namespace can still authenticate to OpenBao and sign admin-scoped tokens for the NVCF API issuer.

Until these three objects are deleted, removing the Deployment has reduced the exposed surface without withdrawing the capability.

Retrieve the root token as described in Troubleshooting, then:

# The JWT auth role bound to the proxy's service account
kubectl exec -it openbao-server-0 -c openbao -n vault-system -- \
  env BAO_TOKEN=$BAO_ROOT_TOKEN \
  bao delete auth/jwt/role/admin-issuer-proxy
# The policy granting it signing rights
kubectl exec -it openbao-server-0 -c openbao -n vault-system -- \
  env BAO_TOKEN=$BAO_ROOT_TOKEN \
  bao delete sys/policies/acl/services-admin-issuer-proxy-jwt-sign-admin-issuer-proxy-rw
# The JWT secret role that mints the admin-scoped token
kubectl exec -it openbao-server-0 -c openbao -n vault-system -- \
  env BAO_TOKEN=$BAO_ROOT_TOKEN \
  bao delete services/nvcf-api/jwt/roles/admin-issuer-proxy

Verify all three are gone:

kubectl exec -it openbao-server-0 -c openbao -n vault-system -- \
  env BAO_TOKEN=$BAO_ROOT_TOKEN \
  sh -c 'bao list auth/jwt/role; bao policy list; bao list services/nvcf-api/jwt/roles' \
  | grep admin-issuer-proxy

No output means the grant is withdrawn.

Verifying at the edge

The route’s hostname may have no DNS record, which does not by itself make it unreachable. A load balancer that forwards the Host header passes the request to the Gateway regardless. Check the edge, not only the cluster:

curl -s -o /dev/null -w '%{http_code}\n' \
  --resolve api-keys.<your-domain>:443:<load-balancer-address> \
  https://api-keys.<your-domain>/v1/admin/keys
echo "curl exit: $?"

--resolve rather than -H "Host: ...": the Host header is an HTTP header and does not set the TLS server name, so against a load-balancer IP the certificate would be checked against that IP and the request could fail before the Gateway ever saw it.

Read the two results together. Exit 0 with 404 means no route is attached, which is the outcome you want. Exit 0 with anything else means the route is still being served.

A non-zero curl exit means the request never reached the Gateway, so the check is inconclusive rather than a pass. The printed status will be 000.

Worker NATS Authentication

Every worker connects to NATS to receive its function’s requests. nats.workerAuth sets how it authenticates:

Use token when workers run outside the control-plane cluster, as in a multi-cluster install. In nkey mode, the shared key leaves the cluster on every worker launch.

Configuration

# environments/<environment-name>.yaml
nats:
  workerAuth: token

This value changes three releases:

The auth callout sends worker tokens to the NVCF API’s in-cluster Service, http://api.nvcf.svc.cluster.local:8080/v2/nvcf/webhook/nats-auth. If the API runs elsewhere, set the URL:

# environments/<environment-name>.yaml
nats:
  workerAuth: token
  authCallout:
    workerTokenAuth:
      webhookUrl: http://<api-service>.<api-namespace>.svc.cluster.local:8080/v2/nvcf/webhook/nats-auth

Any value other than nkey or token fails the deployment with nats.workerAuth must be nkey or token. In token mode, an empty webhook URL fails it too.

Switching an existing install

1. Set nats.workerAuth: token and sync the stack.
2. New workers authenticate with their worker tokens. Workers that were already running keep the shared NKey and stay connected until they’re replaced, for example by redeploying their functions.
3. Once no worker started before the switch is running, rotate the shared NATS NKey, because those workers held it.

Switching back to nkey restores the shared key for new workers and restarts the auth callout the same way.

Verifying

The auth callout logs successful logins, with the account and the user it assigned. Worker logins in token mode show the account APP and a user named after the function version:

kubectl logs -n nats-system -l app.kubernetes.io/instance=nats-auth-callout-service \
  --tail=-1 --prefix | grep 'Authentication successful' | grep 'worker-'

If new workers don’t appear there, look for refused logins:

kubectl logs -n nats-system -l app.kubernetes.io/instance=nats-auth-callout-service \
  --tail=-1 --prefix | grep -E 'Authentication (failed|denied)'

The auth callout samples its logs: per message, the first 100 lines a second, then 1 in 100, so a burst of worker logins can drop lines. While debugging, log every login and turn on debug logs, which include each webhook request to the NVCF API, in environments/<environment-name>.yaml:

natsAuthCalloutService:
  serviceConfig:
    logging:
      level: debug
      sampling:
        initial: 0

Only natsAuthCalloutService.serviceConfig.logging reaches the auth callout; the stack ignores other serviceConfig keys. A change restarts the auth callout.

With the settings above, if the auth callout logged nothing for a worker, its login didn’t reach the auth callout. Check the worker’s own logs for its NATS connection error.

Observability

For observability configuration and reference architecture, see self-hosted-observability.