Certificates#

Overview#

NVIDIA Mission Control issues and distributes TLS material with two Argo CD Applications, cert-manager and trust-manager, in the cert-manager namespace. cert-manager issues each component’s leaf certificate. trust-manager distributes a trust bundle that combines the cluster root with the public CAs, so workloads trust those leaves and public endpoints.

How it works#

cert-manager and trust-manager run their upstream Helm charts. cert-manager provisions a cluster-wide issuer, nmc-openbao-ca, that signs leaf certificates from the OpenBao intermediate CA (refer to OpenBao). Each component owns its leaf Certificate and references this issuer, so it never manages CA material itself.

%%{init: {'theme': 'base', 'themeVariables': {'lineColor': '#888888', 'edgeLabelBackground': '#e8e8e8'}}}%% flowchart TB subgraph ob["OpenBao"] Root["Root CA"] --> Int["Intermediate CA"] end subgraph cm["cert-manager namespace"] Issuer["ClusterIssuer<br/>nmc-openbao-ca"] Leaf["Component leaf<br/>Certificates"] Anchor["nmc-ca-anchor<br/>Certificate"] ASecret[("nmc-ca-anchor-tls<br/>Secret = root")] TM["trust-manager"] end Bundle["nmc-ca-bundle<br/>in every namespace"] Int -.->|"signs leaves"| Issuer Issuer --> Leaf Int -.->|"anchors root"| Anchor Anchor --> ASecret ASecret --> TM TM -->|"distributes root + public CAs"| Bundle style ob fill:#ede7f6,stroke:#5e35b1,stroke-width:1px style cm fill:#eef5ff,stroke:#1565c0,stroke-width:2px style Root fill:#ede7f6,stroke:#5e35b1,color:#000 style Int fill:#ede7f6,stroke:#5e35b1,color:#000 style Issuer fill:#ef7b4d,stroke:#c5410a,color:#fff style Leaf fill:#fffde7,stroke:#f9a825,color:#000 style Anchor fill:#e8f5e9,stroke:#2e7d32,color:#000 style ASecret fill:#fff3e0,stroke:#e65100,color:#000 style TM fill:#e3f2fd,stroke:#1565c0,color:#000 style Bundle fill:#e8f5e9,stroke:#2e7d32,color:#000 linkStyle default stroke-width:1.5px

cert-manager also issues a CA-anchor Certificate (nmc-ca-anchor) that writes the root into the nmc-ca-anchor-tls Secret. trust-manager combines that root with the public CAs from its default CA package and distributes the result to every namespace as the nmc-ca-bundle trust bundle. Servers present their leaf and intermediate, so trusting the root validates NMC leaves. A workload can mount the nmc-ca-bundle ConfigMap and point SSL_CERT_FILE at the mounted file to trust NMC leaves and public endpoints together.

The two Applications sync in order: cert-manager and its issuer first, then trust-manager once the root is available.

To bring your own gateway certificate, refer to NMC Gateway.

For the issuer’s OpenBao endpoint, auth role, and PKI paths, refer to the cert-manager Application values in the component template tree (apps/cert-manager/).

Configuration#

Add these settings under the certManager and trustManager keys in values.yaml. Each key already carries the pinned version and syncWave from Installing GitOps-Managed Components. The following fields are the ones you set. A field shown with a value uses that value as its default when you omit it.

cert-manager#

certManager:
  enabled: true                  # optional, deploy cert-manager
  replicaCount: 1                # optional, controller replicas
  webhookReplicaCount: 1         # optional, webhook replicas
  cainjectorReplicaCount: 1      # optional, cainjector replicas
  enableGatewayAPI: false        # optional, Gateway API support in cert-manager
  prometheusEnabled: true        # optional, expose Prometheus metrics
  serviceMonitorEnabled: true    # optional, create a ServiceMonitor

  • enabled – deploy the cert-manager Application. Default true.

  • replicaCount / webhookReplicaCount / cainjectorReplicaCount – controller, webhook, and cainjector replicas. Default 1.

  • enableGatewayAPI – enable Gateway API support in cert-manager. Default false.

  • prometheusEnabled / serviceMonitorEnabled – expose metrics and create a ServiceMonitor. Default true.

trust-manager#

trustManager:
  enabled: true                  # optional, deploy trust-manager
  webhookHelmCertEnabled: true   # optional, Helm-generated webhook cert (false delegates to cert-manager)
  extraCaEnabled: false          # optional, add a CA you place in OpenBao to the trust bundle
  metricsEnabled: true           # optional, expose metrics
  serviceMonitorEnabled: true    # optional, create a ServiceMonitor

  • enabled – deploy the trust-manager Application. Default true.

  • webhookHelmCertEnabled – use a Helm-generated webhook certificate when true, delegate to cert-manager when false. Default true.

  • extraCaEnabled – add a CA you place in OpenBao to the nmc-ca-bundle trust bundle. Default false. Use it when the gateway serves a certificate that chains to a private CA. Refer to Add a private CA to the trust bundle.

  • metricsEnabled / serviceMonitorEnabled – expose metrics and create a ServiceMonitor. Default true.

Add a private CA to the trust bundle#

When the gateway serves a certificate that chains to a private certificate authority (CA), in-cluster clients must trust that CA. Add it to the nmc-ca-bundle trust bundle through trust-manager. A publicly trusted CA needs nothing, because the bundle already carries the public CAs.

You place the CA in OpenBao at nmc/cert-manager/extra-ca. External Secrets projects it into the nmc-extra-ca Secret, and trust-manager adds it to the bundle. Enable this with trustManager.extraCaEnabled: true, then provide the CA in the flow that matches your cluster.

At install, add k8s_bootstrap_secrets_extra_ca_crt to secrets.yaml (refer to the Secrets section). The seed step loads the CA into OpenBao before trust-manager publishes the bundle.

On a running cluster, write it to OpenBao as the nmc-admin operator (refer to Log in as the nmc-admin operator), then publish:

bao kv put nmc/cert-manager/extra-ca ca.crt=@/tmp/ca.pem

Establish this trust before the gateway serves the certificate, so clients trust it from the first request. Confirm the bundle carries the CA:

kubectl get bundle nmc-ca-bundle

Secrets#

cert-manager consumes no bootstrap secrets. The trust anchor is produced in-cluster: the nmc-ca-anchor Certificate writes the root CA to the ca.crt key of the nmc-ca-anchor-tls Secret in the cert-manager namespace, and the nmc-ca-bundle Bundle sources that key together with the public default CAs.

trust-manager consumes one bootstrap secret, and only when you add a private CA to the trust bundle. Add it to secrets.yaml:

# Only when adding a private CA to the trust bundle (trustManager.extraCaEnabled: true):
k8s_bootstrap_secrets_extra_ca_crt: "{{ lookup('file', '/path/to/ca.pem') }}"   # private CA, PEM

  • k8s_bootstrap_secrets_extra_ca_crt – the private CA to add to the nmc-ca-bundle trust bundle, in PEM. Give an absolute path to the file lookup, or paste the PEM inline as a block scalar. Used when trustManager.extraCaEnabled is true.

For enabling and the running-cluster flow, refer to Add a private CA to the trust bundle.

Operations#

The nmc-ca-anchor Certificate renews automatically on its configured duration and renewBefore, refreshing the root anchor that trust-manager publishes. Each consuming Application renews its own leaf Certificate.

Rotating the intermediate CA happens entirely in OpenBao under the nmc-admin operator (refer to OpenBao for the procedure).

Rotating the root CA is a break-glass procedure. Regenerate the root, re-issue the intermediate, then re-distribute the root. trust-manager republishes it in-cluster once the anchor Secret updates. IT trust distribution pushes it to external clients.

Break-glass and backup#

The root and intermediate private keys never leave OpenBao, and the anchor Secret holds only the public root, so certificate recovery follows OpenBao’s own backup procedure rather than a separate key escrow. Refer to OpenBao for OpenBao snapshot and seal-key backup.

Root CA rotation is the break-glass path described in the preceding Operations section.

Verify#

Confirm the OpenBao-backed issuer and the anchor Certificate are ready:

kubectl get clusterissuer nmc-openbao-ca
kubectl get certificate nmc-ca-anchor -n cert-manager

Both report Ready=True when healthy. The anchor Secret exists with a populated root:

kubectl get secret nmc-ca-anchor-tls -n cert-manager -o jsonpath='{.data.ca\.crt}'

Confirm trust-manager has published the bundle and synced it to every namespace:

kubectl get bundle nmc-ca-bundle

The Bundle reports Synced=True when healthy.