OpenBao#

Overview#

NVIDIA Mission Control uses OpenBao as the cluster’s secrets backend and internal certificate authority. It holds the KV data that platform components read at runtime and signs the certificate chain that cert-manager issues leaves from. OpenBao is delivered as an Argo CD Application. Refer to Installing GitOps-Managed Components for the deploy flow.

How it works#

OpenBao runs the upstream OpenBao Helm chart on integrated Raft storage, single-node at bootstrap and scalable to a high-availability quorum. It comes up in two passes: a one-time self-initialization the first time the server starts on empty storage, and a configure Job that runs after each Argo CD sync to load the data the components consume.

%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#888888', 'edgeLabelBackground': '#e8e8e8'}}}%% flowchart TB Seal[("openbao-seal key")] Argo["Argo CD"] subgraph ob["OpenBao server"] Pod["OpenBao pod"] Init["Self-init<br/>mounts, CA, policies, nmc-admin"] Cfg["Configure Job<br/>KV seed, issuing role"] end Down["Other components"] Seal -.->|"auto-unseal, every pod start"| Pod Pod -->|"first start on empty storage"| Init Argo -.->|"each sync"| Cfg Cfg -->|"downstream components gate on this"| Down style ob fill:#ede7f6,stroke:#5e35b1,stroke-width:2px style Seal fill:#fff3e0,stroke:#e65100,color:#000 style Argo fill:#e3f2fd,stroke:#1565c0,color:#000 style Pod fill:#ef7b4d,stroke:#c5410a,color:#fff style Init fill:#e8f5e9,stroke:#2e7d32,color:#000 style Cfg fill:#e8f5e9,stroke:#2e7d32,color:#000 style Down fill:#f5f5f5,stroke:#9e9e9e,color:#000 linkStyle default stroke-width:1.5px

Auto-unseal. OpenBao unseals itself on every pod restart from a static 32-byte key you supply in the openbao-seal Secret. You own that key. Keep a backup outside the cluster, as Break-glass and backup describes.

Warning

The seal key is permanent. Losing it while the Raft volumes still exist leaves the store unrecoverable. Never rotate it while Raft storage persists.

Internal certificate authority. OpenBao builds a two-tier certificate authority, a root and an intermediate, entirely inside the server. No private key ever leaves it. cert-manager signs leaf certificates from the intermediate (refer to Certificates).

Self-initialization. On first start with empty storage, OpenBao enables its secret and auth mounts and builds the CA. It creates a least-privilege policy and Kubernetes auth role for each component, so each reads only its own secrets. It also creates nmc-admin, the Day-2 operator login. nmc-admin manages KV data, the component policies and roles, and the intermediate CA. It cannot touch the seal or the root CA, which are break-glass operations. Self-init runs only on empty storage. Day-2 changes go through nmc-admin, not a re-init.

Configure Job. After each Argo CD sync of the OpenBao Application, NVIDIA Mission Control’s OpenBao configurator runs a PostSync Job. It loads the KV data the components read, sourced from the bootstrap Secrets the seed step created. This KV load is create-only, so a re-sync never overwrites a value already in use. The Job also applies the cert-manager issuing role. OpenBao syncs first. Every other component waits for this Job to succeed before it authenticates.

For the exact mounts, policies, auth roles, and PKI paths, refer to the OpenBao self-initialization configuration in the component template tree (apps/openbao/values.yaml.j2).

Configuration#

Add these settings under the openbao key in values.yaml. The key already carries the pinned version, configuratorVersion, and syncWave from Installing GitOps-Managed Components. The following fields are the ones you set. A field shown with a value uses that value as its default when you omit it.

openbao:
  enabled: true            # optional, deploy the OpenBao Application
  bootstrapInit: true      # optional, gate self-init. True on first bring-up, false after
  replicas: 1              # optional, HA Raft replicas. Bootstrap at 1, then scale to an odd count
  storageSize: 100Gi       # optional, size of each server's Raft data volume
  injectorEnabled: true    # optional, enable the agent injector Keycloak and Launchpad use
  csiEnabled: false        # optional, enable the OpenBao CSI provider

  • enabled – deploy the OpenBao Application. Default true.

  • bootstrapInit – gates self-initialization. Keep it true for first bring-up. Set it false once the cluster is bootstrapped so re-syncs only retry_join. Default true.

  • replicas – HA Raft replica count. Bootstrap with 1 so only node-0 self-inits, then scale out. For a multi-node quorum use an odd count so Raft can elect a leader. Default 1.

  • storageSize – size of each server’s Raft data volume. Default 100Gi.

  • injectorEnabled – enables the agent injector that Keycloak and Launchpad use to receive their secrets at runtime. Default true.

  • csiEnabled – enables the OpenBao CSI provider. Default false.

Note

replicas and bootstrapInit work together. Bring the cluster up at replicas: 1 with bootstrapInit: true so a single node self-inits, then scale to an odd replicas count with bootstrapInit: false so the joining nodes only retry_join instead of re-initializing.

Secrets#

OpenBao owns two bootstrap secrets, seeded into the openbao namespace before it deploys. Leave either empty to auto-generate it. Add them to secrets.yaml:

k8s_bootstrap_secrets_seal_key_b64: ""    # static auto-unseal key, base64 of 32 bytes. Leave empty to auto-generate
k8s_bootstrap_secrets_admin_password: ""  # nmc-admin login. Leave empty to auto-generate

  • k8s_bootstrap_secrets_seal_key_b64 becomes the openbao-seal Secret, the static 32-byte auto-unseal key read on every pod start. Its lifecycle is permanent. Refer to the break-glass section for backup.

  • k8s_bootstrap_secrets_admin_password becomes the nmc-admin userpass login, read once by self-init. A generated value is 24 characters. Its lifecycle is operator-managed: back it up outside the cluster, then remove the openbao-admin-credentials Secret once bootstrap is complete (refer to Clean Up the Bootstrap Secrets).

Operations#

Log in as the nmc-admin operator#

Day-2 changes and credential lookups run through the bao command-line client as the nmc-admin operator.

Get the password. If you set k8s_bootstrap_secrets_admin_password in secrets.yaml, that value is the password. If you let NVIDIA Mission Control generate it, read it from the openbao-admin-credentials Secret while it still exists. Cleanup removes the Secret (refer to Clean Up the Bootstrap Secrets), so back up this value first and keep it as your break-glass login.

kubectl -n openbao get secret openbao-admin-credentials \
  -o jsonpath='{.data.password}' | base64 -d

Log in. On the active BCM head node, create a temporary client pod in the openbao namespace. Reuse the deployed OpenBao image so the pod carries a matching bao client:

IMAGE=$(kubectl -n openbao get statefulset openbao \
  -o jsonpath='{.spec.template.spec.containers[0].image}')

kubectl -n openbao run bao-admin --image="$IMAGE" --restart=Never \
  --env=BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200 \
  --command -- sleep 3600

kubectl -n openbao wait --for=condition=Ready pod/bao-admin --timeout=60s

If a command needs a file, copy it into the pod before you open the shell. bao reads @file arguments from inside the pod. Skip this step when you only need to log in:

kubectl -n openbao cp <file> bao-admin:/tmp/<file>

Open a shell in the client pod:

kubectl -n openbao exec -it bao-admin -- sh

In the pod shell, log in:

bao login -method=userpass username=nmc-admin      # prompts for the nmc-admin password

Read and write KV. As nmc-admin you read and write the nmc KV store. In the pod shell, read a seeded value with bao kv get, for example the generated Keycloak admin password:

bao kv get nmc/keycloak/realm/master/users/admin

A bao kv put that writes file material reads the file from the pod, so stage it first and reference its staged path with @, for example <key>=@/tmp/<file>.

When you finish, exit the pod shell:

exit

Back on the BCM head node, delete the pod to remove its local credential files and any staged files:

kubectl -n openbao delete pod bao-admin

Intermediate-CA rotation#

Rotate the intermediate entirely inside OpenBao under nmc-admin, with no offline ceremony: pki_int/intermediate/generate/internal, then pki/root/sign-intermediate, then set-signed. Leaf certificates need no action. cert-manager auto-renews them from pki_int. Rotating the root CA is a break-glass procedure.

Adding a consumer (Day-2)#

A new consumer needs a policy, an auth role, and a KV seed if it reads stored data. Self-init does not re-run on a non-empty store, so on an existing cluster you, as nmc-admin, create the policy and auth role with bao write and seed the KV path directly. The configure Job’s create-only KV load then skips the already-populated path. To remove an NVIDIA Mission Control-shipped consumer, delete it from the chart values, then run bao policy delete <name> and bao delete auth/kubernetes/role/<name>.

Secret rotation#

Rotate KV data with a direct bao kv put under nmc-admin. The configure Job’s KV load is create-only and will not overwrite. A full rotation also requires updating the consumer’s persisted state, which is component-specific. Rotate the nmc-admin password itself with bao write against auth/userpass/users/nmc-admin while you still hold the current password. If the password is lost, follow Reset a Lost Administrator Password.

Failure modes#

Use the failure type to choose the recovery action:

  • A failed PostSync configure Job is usually transient. Re-run it with argocd app sync <app>. Argo CD deletes the failed Job and runs a fresh one. If the old Job is stuck, delete it first.

  • If self-init failed partway through or storage is in a bad first-run state, wipe and restart (refer to the following procedure). A failed configure Job is re-run, not wiped.

  • If the seal key is lost but the volumes are intact, restore the openbao-seal backup. With no backup, OpenBao cannot be unsealed.

  • If the nmc-admin password is lost, follow Reset a Lost Administrator Password. You need recovery shares committed before you lost access; creating them requires an authenticated admin token.

Wipe and restart#

Use this only when self-init failed partway through or storage is in a bad first-run state. In a partial-bootstrap state, no operator-managed material has been backed up yet, so deleting the whole openbao namespace is safe. Argo CD recreates chart-managed resources and the seed step recreates operator-managed ones. Suspend the Application’s sync policy, delete the namespace, re-run the seed step so the seal and admin Secrets are recreated, then resume the sync policy and sync. OpenBao regenerates the CA on the next self-init. Do not attempt to repair a partial bootstrap by editing OpenBao state by hand.

Break-glass and backup#

OpenBao auto-recovers from pod restarts, but not from a lost seal key or lost data. Back up the following items outside the cluster. The openbao-seal key matters most: OpenBao cannot unseal without it, and it is not part of a Raft snapshot, so its loss is unrecoverable. A Raft snapshot is the full data backup. Recovery shares are separate, optional material that authorize generate-root. That procedure mints a temporary root token to reset a lost nmc-admin password or run a root-only operation. The server has no recovery shares by default.

Back up the following:

  • The openbao-seal key, the static auto-unseal key. Keep an authoritative copy in your secrets manager, KMS, or HSM, or in sealed offline storage, with access limited to break-glass operators. The in-cluster Secret is only the runtime copy.

  • A Raft snapshot, which captures the root key and the rest of the store.

  • The nmc-admin password, your Day-2 login.

  • The recovery shares, once you generate them (refer to Generate Recovery Shares).

Generate Recovery Shares#

Generate the shares while nmc-admin is still available, so they exist before you ever need them.

  1. Run bao write -format=json sys/rotate/recovery/init secret_shares=<n> secret_threshold=<m> require_verification=true. This returns the shares and a verification nonce but does not commit them.

  2. Back the shares up offline.

  3. Submit the threshold of shares to sys/rotate/recovery/verify with the nonce to commit them. require_verification catches a mis-copied share here, before the commit takes effect. Once committed, shares cannot be regenerated without the current ones.

Generate a Root Token#

For a root-only operation while you can still log in as nmc-admin, use the upstream generate-root procedure with the recovery shares. Revoke the temporary root token when you finish. Keep the legacy listener endpoints disabled.

Reset a Lost Administrator Password#

Use the committed shares from Generate Recovery Shares to reset a lost nmc-admin password. The bao operator generate-root command requires authentication, including when it decodes a token. This procedure uses the legacy recovery API and a small offline script to decode the token.

You need Kubernetes administrator access, Bash, curl, jq, and Python 3. Run the procedure in two Bash sessions on the active BCM head node, with kubectl configured for k8s-admin in both. If a command fails, stop and follow the cleanup instructions at the end of this procedure.

  1. Record the auto-sync settings for the root and k8s-admin-openbao Applications. Disable auto-sync on root and wait for its current sync to finish. Then disable auto-sync on k8s-admin-openbao and wait for its current sync to finish. Confirm that auto-sync is disabled on both before continuing; a running parent sync can restore the child’s auto-sync setting. Do not sync either Application until you finish recovery.

  2. Edit openbao-config in the openbao namespace. In its TCP listener stanza, temporarily set disable_unauthed_generate_root_endpoints = false.

    kubectl -n openbao edit configmap openbao-config

  3. Replace the OpenBao server pods one at a time to load the listener change. The chart uses OnDelete, so a StatefulSet restart alone does not replace them. For HA, replace standbys first and the active pod last. Before replacing the next pod, wait for the replacement to be Ready, initialized, and unsealed. Its Raft applied index must catch up to its committed index. Confirm that the cluster has an active server. Stop if a pod does not recover. A single-replica deployment is unavailable during its restart.

    kubectl -n openbao get pods -l component=server -L openbao-active
kubectl -n openbao delete pod <pod-name>

    After the replacement pod appears, check its readiness and local status:

    kubectl -n openbao wait --for=condition=Ready pod/<pod-name> --timeout=180s
kubectl -n openbao exec <pod-name> -- bao status

  4. After all servers recover, forward a local port to the active service in a separate terminal. Keep this command running. If the active server changes, restart the port-forward before continuing.

    kubectl -n openbao port-forward --address 127.0.0.1 \
  service/openbao-active 18200:8200

  5. In your recovery terminal, create a private working directory. Confirm that sys/health succeeds with initialized: true, sealed: false, and standby: false. Check generation status and continue only when started is false. If another attempt exists, coordinate with its owner before canceling it.

    set +x
recovery_umask=$(umask)
umask 077
recovery_dir=$(mktemp -d)
cd "$recovery_dir"
BAO_ADDR=http://127.0.0.1:18200
curl --fail --silent --show-error "$BAO_ADDR/v1/sys/health"
curl --fail --silent --show-error "$BAO_ADDR/v1/sys/generate-root/attempt"

  6. Start an attempt and save its nonce and one-time password (OTP). The server returns the OTP only in this response. Keep attempt.json private.

    curl --fail --silent --show-error \
  --header 'Content-Type: application/json' --data '{}' \
  "$BAO_ADDR/v1/sys/generate-root/attempt" > attempt.json

  7. Submit one distinct recovery share at a time, using the same attempt nonce. Repeat this step until complete is true. The final response contains the encoded token. The prompt hides the share from the terminal and shell history.

    nonce=$(jq -er '.nonce' attempt.json)
IFS= read -r -s -p 'Recovery share: ' recovery_share
printf '\n'
printf '%s' "$recovery_share" |
  jq -Rs --arg nonce "$nonce" '{nonce: $nonce, key: .}' > share.json
unset recovery_share
curl --fail --silent --show-error \
  --header 'Content-Type: application/json' --data-binary @share.json \
  "$BAO_ADDR/v1/sys/generate-root/update" > update.json
rm share.json
jq '{progress, required, complete}' update.json

  8. Copy and run the following block from the private working directory containing attempt.json and update.json. It creates and runs the decoder script, which uses only the Python standard library and makes no network requests. The script writes root-header without displaying the root token. Keep these files until you revoke the token.

    Create and Run the Token Decoder
    cat > decode-openbao-root-token.py <<'PY'
#!/usr/bin/env python3
"""Decode an OpenBao root token from attempt.json and update.json offline."""

import base64
import json
import os
from pathlib import Path

os.umask(0o077)

try:
    attempt = json.loads(Path("attempt.json").read_text())
    result = json.loads(Path("update.json").read_text())
    if result["complete"] is not True:
        raise ValueError("Submit the required recovery shares first.")
    if result["nonce"] != attempt["nonce"]:
        raise ValueError("The responses belong to different recovery attempts.")

    encoded = result["encoded_token"]
    raw = base64.b64decode(encoded + "=" * (-len(encoded) % 4), validate=True)
    otp = attempt["otp"].encode("ascii")
    if not raw or len(raw) != len(otp):
        raise ValueError("The encoded token and OTP lengths differ or are empty.")
    token = bytes(a ^ b for a, b in zip(raw, otp)).decode("ascii")

    # Exclusive creation protects a header saved by an earlier recovery attempt.
    with Path("root-header").open("x") as header:
        header.write("X-Vault-Token: " + token + "\n")
except (OSError, ValueError, KeyError, TypeError) as error:
    raise SystemExit(f"Cannot decode the root token: {error}")

print("Created root-header. Keep it until the temporary root token is revoked.")
PY
python3 decode-openbao-root-token.py

  9. Set the new nmc-admin password. Back it up outside the cluster before continuing.

    IFS= read -r -s -p 'New nmc-admin password: ' new_password
printf '\n'
IFS= read -r -s -p 'Confirm new password: ' confirm_password
printf '\n'
if [ -n "$new_password" ] && [ "$new_password" = "$confirm_password" ]; then
  printf '%s' "$new_password" | jq -Rs '{password: .}' > password.json
  curl --fail --silent --show-error --header @root-header \
    --header 'Content-Type: application/json' --data-binary @password.json \
    "$BAO_ADDR/v1/auth/userpass/users/nmc-admin/password"
else
  printf 'The password is empty or the entries do not match. Repeat this step.\n' >&2
fi
unset new_password confirm_password
rm -f password.json

  10. Log in with the new password using the temporary client pod from Log in as the nmc-admin operator. Exit the pod shell and delete the client pod as described there. Back in the recovery terminal, revoke the temporary root token:

    curl --fail --silent --show-error --header @root-header \
  --request POST "$BAO_ADDR/v1/auth/token/revoke-self"

  11. Restore the listener to disable_unauthed_generate_root_endpoints = true or remove the temporary setting. Repeat the pod replacement procedure from step 3. Restart the port-forward and confirm that the legacy endpoint now returns HTTP 405.

    curl --silent --show-error --output /dev/null --write-out '%{http_code}\n' \
  "$BAO_ADDR/v1/sys/generate-root/attempt"

  12. After any issued root token is revoked and the legacy endpoint returns HTTP 405, stop the port-forward, remove the private recovery files, and restore your original umask. Restore the original auto-sync settings on k8s-admin-openbao, then root.

    cd -
rm -r "$recovery_dir"
umask "$recovery_umask"
unset BAO_ADDR recovery_dir recovery_umask nonce

If you stop before token generation completes, cancel the attempt with DELETE /v1/sys/generate-root/attempt, then finish cleanup with steps 11 and 12. If a root token was issued and has not been revoked, complete step 8 if needed. Run the revocation command in step 10, then finish with steps 11 and 12.

Verify#

Confirm OpenBao is up:

  • Confirm the server is unsealed and ready: query /v1/sys/health?standbyok=true on the OpenBao service, which is the path the readiness and liveness probes use.

  • Confirm self-init completed: the pki, pki_int, and nmc mounts exist and the CA chain is present at pki_int/cert/ca_chain.

  • Confirm the configure Job succeeded at PostSync. Downstream Applications gate on it.

  • Confirm nmc-admin can log in via userpass.