OpenBao#

Overview#

NVIDIA Mission Control uses OpenBao as the cluster’s secrets backend and internal certificate authority. It holds the KV data that platform components read at runtime and signs the certificate chain that cert-manager issues leaves from. OpenBao is delivered as an Argo CD Application. Refer to Installing GitOps-Managed Components for the deploy flow.

How it works#

OpenBao runs the upstream OpenBao Helm chart on integrated Raft storage, single-node at bootstrap and scalable to a high-availability quorum. It comes up in two passes: a one-time self-initialization the first time the server starts on empty storage, and a configure Job that runs after each Argo CD sync to load the data the components consume.

%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#888888', 'edgeLabelBackground': '#e8e8e8'}}}%% flowchart TB Seal[("openbao-seal key")] Argo["Argo CD"] subgraph ob["OpenBao server"] Pod["OpenBao pod"] Init["Self-init<br/>mounts, CA, policies, nmc-admin"] Cfg["Configure Job<br/>KV seed, issuing role"] end Down["Other components"] Seal -.->|"auto-unseal, every pod start"| Pod Pod -->|"first start on empty storage"| Init Argo -.->|"each sync"| Cfg Cfg -->|"downstream components gate on this"| Down style ob fill:#ede7f6,stroke:#5e35b1,stroke-width:2px style Seal fill:#fff3e0,stroke:#e65100,color:#000 style Argo fill:#e3f2fd,stroke:#1565c0,color:#000 style Pod fill:#ef7b4d,stroke:#c5410a,color:#fff style Init fill:#e8f5e9,stroke:#2e7d32,color:#000 style Cfg fill:#e8f5e9,stroke:#2e7d32,color:#000 style Down fill:#f5f5f5,stroke:#9e9e9e,color:#000 linkStyle default stroke-width:1.5px

Auto-unseal. OpenBao unseals itself on every pod restart from a static 32-byte key you supply in the openbao-seal Secret. You own that key. Keep a backup outside the cluster, as Break-glass and backup describes.

Warning

The seal key is permanent. Losing it while the Raft volumes still exist leaves the store unrecoverable. Never rotate it while Raft storage persists.

Internal certificate authority. OpenBao builds a two-tier certificate authority, a root and an intermediate, entirely inside the server. No private key ever leaves it. cert-manager signs leaf certificates from the intermediate (refer to Certificates).

Self-initialization. On first start with empty storage, OpenBao enables its secret and auth mounts and builds the CA. It creates a least-privilege policy and Kubernetes auth role for each component, so each reads only its own secrets. It also creates nmc-admin, the Day-2 operator login. nmc-admin manages KV data, the component policies and roles, and the intermediate CA. It cannot touch the seal or the root CA, which are break-glass operations. Self-init runs only on empty storage. Day-2 changes go through nmc-admin, not a re-init.

Configure Job. After each Argo CD sync of the OpenBao Application, NVIDIA Mission Control’s OpenBao configurator runs a PostSync Job. It loads the KV data the components read, sourced from the bootstrap Secrets the seed step created. This KV load is create-only, so a re-sync never overwrites a value already in use. The Job also applies the cert-manager issuing role. OpenBao syncs first. Every other component waits for this Job to succeed before it authenticates.

For the exact mounts, policies, auth roles, and PKI paths, refer to the OpenBao self-initialization configuration in the component template tree (apps/openbao/values.yaml.j2).

Configuration#

Add these settings under the openbao key in values.yaml. The key already carries the pinned version, configuratorVersion, and syncWave from Installing GitOps-Managed Components. The following fields are the ones you set. A field shown with a value uses that value as its default when you omit it.

openbao:
  enabled: true            # optional, deploy the OpenBao Application
  bootstrapInit: true      # optional, gate self-init. True on first bring-up, false after
  replicas: 1              # optional, HA Raft replicas. Bootstrap at 1, then scale to an odd count
  storageSize: 100Gi       # optional, size of each server's Raft data volume
  injectorEnabled: true    # optional, enable the agent injector Keycloak and Launchpad use
  csiEnabled: false        # optional, enable the OpenBao CSI provider

  • enabled – deploy the OpenBao Application. Default true.

  • bootstrapInit – gates self-initialization. Keep it true for first bring-up. Set it false once the cluster is bootstrapped so re-syncs only retry_join. Default true.

  • replicas – HA Raft replica count. Bootstrap with 1 so only node-0 self-inits, then scale out. For a multi-node quorum use an odd count so Raft can elect a leader. Default 1.

  • storageSize – size of each server’s Raft data volume. Default 100Gi.

  • injectorEnabled – enables the agent injector that Keycloak and Launchpad use to receive their secrets at runtime. Default true.

  • csiEnabled – enables the OpenBao CSI provider. Default false.

Note

replicas and bootstrapInit work together. Bring the cluster up at replicas: 1 with bootstrapInit: true so a single node self-inits, then scale to an odd replicas count with bootstrapInit: false so the joining nodes only retry_join instead of re-initializing.

Secrets#

OpenBao owns two bootstrap secrets, seeded into the openbao namespace before it deploys. Leave either empty to auto-generate it. Add them to secrets.yaml:

k8s_bootstrap_secrets_seal_key_b64: ""    # static auto-unseal key, base64 of 32 bytes. Leave empty to auto-generate
k8s_bootstrap_secrets_admin_password: ""  # nmc-admin login. Leave empty to auto-generate

  • k8s_bootstrap_secrets_seal_key_b64 becomes the openbao-seal Secret, the static 32-byte auto-unseal key read on every pod start. Its lifecycle is permanent. Refer to the break-glass section for backup.

  • k8s_bootstrap_secrets_admin_password becomes the nmc-admin userpass login, read once by self-init. A generated value is 24 characters. Its lifecycle is operator-managed: back it up outside the cluster, then remove the openbao-admin-credentials Secret once bootstrap is complete (refer to Clean Up the Bootstrap Secrets).

Operations#

Log in as the nmc-admin operator#

Day-2 changes and credential lookups run through the bao command-line client as the nmc-admin operator.

Get the password. If you set k8s_bootstrap_secrets_admin_password in secrets.yaml, that value is the password. If you let NVIDIA Mission Control generate it, read it from the openbao-admin-credentials Secret while it still exists. Cleanup removes the Secret (refer to Clean Up the Bootstrap Secrets), so back up this value first and keep it as your break-glass login.

kubectl -n openbao get secret openbao-admin-credentials \
  -o jsonpath='{.data.password}' | base64 -d

Log in. Run the client from a temporary pod you create in the openbao namespace, rather than the server pods that Argo CD manages. Reuse the deployed OpenBao image so the pod carries a matching bao client. bao reads any @file argument from the pod, so stage the files a command needs into the pod before you log in:

IMAGE=$(kubectl -n openbao get statefulset openbao \
  -o jsonpath='{.spec.template.spec.containers[0].image}')

kubectl -n openbao run bao-admin --image="$IMAGE" --restart=Never \
  --env=BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200 \
  --command -- sleep 3600

kubectl -n openbao wait --for=condition=Ready pod/bao-admin --timeout=60s   # wait until the pod is Ready before copying in

kubectl -n openbao cp <file> bao-admin:/tmp/<file>   # stage any file a command reads with @/tmp/<file>
kubectl -n openbao exec -it bao-admin -- sh

# In the pod shell:
bao login -method=userpass username=nmc-admin      # prompts for the nmc-admin password

Delete the pod when you finish. It holds your session and any staged files:

kubectl -n openbao delete pod bao-admin

Read and write KV. As nmc-admin you read and write the nmc KV store. Read a seeded value with bao kv get, for example the generated Keycloak admin password:

bao kv get nmc/keycloak/realm/master/users/admin

A bao kv put that writes file material reads the file from the pod, so stage it first and reference its staged path with @, for example <key>=@/tmp/<file>.

Intermediate-CA rotation#

Rotate the intermediate entirely inside OpenBao under nmc-admin, with no offline ceremony: pki_int/intermediate/generate/internal, then pki/root/sign-intermediate, then set-signed. Leaf certificates need no action. cert-manager auto-renews them from pki_int. Rotating the root CA is a break-glass procedure.

Adding a consumer (Day-2)#

A new consumer needs a policy, an auth role, and a KV seed if it reads stored data. Self-init does not re-run on a non-empty store, so on an existing cluster you, as nmc-admin, create the policy and auth role with bao write and seed the KV path directly. The configure Job’s create-only KV load then skips the already-populated path. To remove an NVIDIA Mission Control-shipped consumer, delete it from the chart values, then run bao policy delete <name> and bao delete auth/kubernetes/role/<name>.

Secret rotation#

Rotate KV data with a direct bao kv put under nmc-admin. The configure Job’s KV load is create-only and will not overwrite. A full rotation also requires updating the consumer’s persisted state, which is component-specific. Rotate the nmc-admin password itself with bao write against auth/userpass/users/nmc-admin while you still hold the current password. If the password is lost, reset it through the break-glass procedure.

Failure modes#

OpenBao keeps serving traffic unless both the seal key and the Raft volumes are lost.

  • A failed PostSync configure Job is usually transient. Re-run it with argocd app sync <app>. Argo CD deletes the failed Job and runs a fresh one. If the old Job is stuck, delete it first.

  • If self-init failed partway through or storage is in a bad first-run state, wipe and restart (refer to the following procedure). A failed configure Job is re-run, not wiped.

  • If the seal key is lost but the volumes are intact, restore the openbao-seal backup. With no backup, OpenBao cannot be unsealed.

  • If the nmc-admin password is lost and recovery shares exist, mint a temporary root and reset the password (refer to the following break-glass procedure). If no recovery shares were ever generated, the only path is wipe and restart, because root cannot be minted without recovery shares and they cannot be created without an admin token.

Wipe and restart#

Use this only when self-init failed partway through or storage is in a bad first-run state. In a partial-bootstrap state, no operator-managed material has been backed up yet, so deleting the whole openbao namespace is safe. Argo CD recreates chart-managed resources and the seed step recreates operator-managed ones. Suspend the Application’s sync policy, delete the namespace, re-run the seed step so the seal and admin Secrets are recreated, then resume the sync policy and sync. OpenBao regenerates the CA on the next self-init. Do not attempt to repair a partial bootstrap by editing OpenBao state by hand.

Break-glass and backup#

OpenBao auto-recovers from pod restarts, but not from a lost seal key or lost data. Back up the following items outside the cluster. The openbao-seal key matters most: OpenBao cannot unseal without it, and it is not part of a Raft snapshot, so its loss is unrecoverable. A Raft snapshot is the full data backup. Recovery shares are separate, optional material that authorize generate-root. That procedure mints a temporary root token to reset a lost nmc-admin password or run a root-only operation. The server has no recovery shares by default.

Back up the following:

  • The openbao-seal key, the static auto-unseal key. Keep an authoritative copy in your secrets manager, KMS, or HSM, or in sealed offline storage, with access limited to break-glass operators. The in-cluster Secret is only the runtime copy.

  • A Raft snapshot, which captures the root key and the rest of the store.

  • The nmc-admin password, your Day-2 login.

  • The recovery shares, once you generate them (refer to Generate recovery shares). They authorize generate-root.

Generate recovery shares#

Generate the shares while nmc-admin is still available, so they exist before you ever need them.

  1. Run bao write -format=json sys/rotate/recovery/init secret_shares=<n> secret_threshold=<m> require_verification=true. This returns the shares and a verification nonce but does not commit them.

  2. Back the shares up offline.

  3. Submit the threshold of shares to sys/rotate/recovery/verify with the nonce to commit them. require_verification catches a mis-copied share here, before the commit takes effect. Once committed, shares cannot be regenerated without the current ones.

Mint a root token (generate-root)#

This procedure uses the recovery shares (refer to Generate recovery shares). Use it to reset a lost nmc-admin password, or for a root-only operation such as resealing, editing the audit configuration, or adding a mount.

  1. Suspend Argo CD auto-sync for the OpenBao Application so self-heal does not revert the listener edit mid-procedure.

  2. Set disable_unauthed_generate_root_endpoints = false in the listener stanza and do a controlled StatefulSet rollout, restarting pods one at a time. The static seal auto-unseals each pod. With three replicas the service stays up through leader failover.

  3. Run bao operator generate-root with the recovery shares to mint a temporary root token.

  4. Perform the recovery action: reset nmc-admin or run the root-only operation.

  5. Revoke the temporary root token.

  6. Revert the listener to the default, do another controlled rollout, and resume Argo CD auto-sync.

Verify#

Confirm OpenBao is up:

  • Confirm the server is unsealed and ready: query /v1/sys/health?standbyok=true on the OpenBao service, which is the path the readiness and liveness probes use.

  • Confirm self-init completed: the pki, pki_int, and nmc mounts exist and the CA chain is present at pki_int/cert/ca_chain.

  • Confirm the configure Job succeeded at PostSync. Downstream Applications gate on it.

  • Confirm nmc-admin can log in via userpass.