Kubernetes Security Hardening#

This section of the install guide details a series of Kubernetes security hardening recommendations for NVIDIA Mission Control’s user and admin space Kubernetes clusters. Apply these recommendations after setting up Kubernetes via the BCM provided wizard.

Follow this hardening guide after you install Kubernetes and the other NVIDIA Mission Control components, including autonomous hardware recovery and autonomous job recovery. Do not select Kyverno in the cm-kubernetes-setup wizard. This guide describes how to install Kyverno with Helm on each cluster.

User and Admin Cluster Background#

In order to promote the principle of least privilege, NVIDIA Mission Control segments services so that only services which must have access to the out of band (OOB) network have access to it. User space includes those services that do not require access to the out of band (OOB) network. Admin space includes those services that do require access to the OOB network.

To implement this separation of user and admin spaces, NVIDIA Mission Control segments the control plane into two distinct Kubernetes clusters: the user and admin space Kubernetes clusters. The user space cluster runs services that do not need access to the out of band network. The admin space cluster runs services that need access to the out of band (OOB) network.

The user space cluster is used for the deployment of Run:ai. If Run:ai is not in use, then the user space cluster is not deployed. The user space cluster includes three control plane nodes along with the compute trays.

The admin space cluster is used for the deployment of NetQ, Grafana, Prometheus, Loki, NMC autonomous job recovery, and NMC autonomous hardware recovery. It is always present in every DGX GB200 and later NMC deployment. It includes three control plane nodes only.

Securing Control Nodes#

Only administrators should have ssh access to the control nodes that underlie the user and admin space Kubernetes clusters.

To achieve this, we will leverage BCM to restrict access to the user and admin space control node categories. To restrict ssh access to only administrators for the user space control nodes, run the following command:

[a03-p1-head-01->category[k8s-system-user]]% set usernodelogin never
[a03-p1-head-01->category*[k8s-system-user*]]% commit
[a03-p1-head-01->category[k8s-system-user]]%

Note in the above example, replace k8s-system-user with the category name used for your user space Kubernetes control node category.

Now we will also introduce this restriction for the admin space control nodes:

[a03-p1-head-01->category[k8s-system-admin]]% set usernodelogin never
[a03-p1-head-01->category*[k8s-system-admin*]]% commit
[a03-p1-head-01->category[k8s-system-admin]]%

Note in the above example, replace k8s-system-admin with the category name used for your admin space Kubernetes control node category.

BCM has now enabled and configured ssh PAM to deny connections. To add back ssh access, you must explicitly add users and groups to the whitelist of PAM.

Restricting kubectl Access#

Certificate based authentication#

Access to Kubernetes is limited by default. By default, only the root user can access kubectl.

Administrators can grant kubectl access to additional users by modifying the file permissions associated with ~/.kube/config. By default, this file is owned by root, and read and write is restricted solely to root.

BCM also provides tooling to add users to Kubernetes clusters. This creates new certificates and populates the users home directory for ~/.kube/config

An example:

cm-kubernetes-setup --cluster k8s-admin --add-user johndoe --role cluster-admin

Configuring BCM head nodes to restrict network access#

Kubernetes API is available via the nginx instance on the BCM head nodes used for proxying traffic back to the Kubernetes cluster(s). By default this is configured to listen on 10443/tcp passing traffic to 6443/tcp on the Kubernetes admin space cluster, k8s-admin in our example.

A snippet of the nginx.conf file:

stream {
        upstream kube-k8s-admin {
            server a03-p1-netq-x86-01:6443;
            server b04-p1-netq-x86-03:6443;
            server b03-p1-netq-x86-02:6443;
        }
        server {
            listen 10443;
            proxy_pass kube-k8s-admin;
            proxy_next_upstream on;
            proxy_connect_timeout 300ms;
        }

You can test this by using curl from any node in the user space. In this example, we’ll test using a rack of DGX GB200 compute nodes:

pdsh -g rack=B05 'curl -m 1 -k -s https://master:10443 >/dev/null 2>&1; [ $? -eq 28 ] && echo "TIMEOUT" || echo "WORKING"' | dshbak -c
----------------
b05-p1-dgx-05-c[01-18]
----------------
WORKING

To restrict access, we’ll configure shorewall on the BCM head nodes.

Add rules to accept traffic from admin space nodes#

for admin_ip in $(cmsh -c "device list -t headnode -f ip" | tail -n +1; cmsh -c "device list -c k8s-system-admin -f ip" | tail -n +1); do \
    for headnode in $(cmsh -c "device list -t headnode -f hostname" | tail -n +1); do \
        echo "Configuring shorewall on $headnode to accept $admin_ip/32 on port 10443/tcp" && \
        cmsh -c "device; use $headnode; roles; use firewall; openports; add accept nat 10443 tcp fw $admin_ip/32; commit"; \
    done; \
done

Note: substitute the k8s-system-admin category name in the example above with the actual category name used on your cluster if it differs.

Add rule to drop all other traffic from non-matching nodes#

for headnode in $(cmsh -c "device list -t headnode -f hostname" | tail -n +1); do \
    echo "Configuring shorewall on $headnode to deny remaining traffic" && \
    cmsh -c "device; use $headnode; roles; use firewall; openports; add drop nat 10443 tcp fw 0.0.0.0/0; commit"; \
done

Configuration via BCM (cmsh) will automatically restart the shorewall daemon on the BCM head nodes

Validate that we now drop traffic#

pdsh -g rack=B05 'curl -m 1 -k -s https://master:10443 >/dev/null 2>&1; [ $? -eq 28 ] && echo "TIMEOUT" || echo "WORKING"' | dshbak -c
----------------
b05-p1-dgx-05-c[01-18]
----------------
TIMEOUT

Configuring kubernetes nodes to restrict network access#

Network and Kyverno policies are deployed using Helm charts from the NVIDIA NGC NMC Collections and used to restrict network access to the Kubernetes clusters.

and the nmc-kyverno-policies chart from the nmc-kyverno-policies chart on NGC.

helm fetch https://helm.ngc.nvidia.com/nvidia/nv-mission-control/charts/nmc-kyverno-policies-2.4.9.tgz --username='$oauthtoken' --password=<YOUR API KEY>

For NGC access, refer to the NGC User Guide.

Deploying network policies with Helm (nmc-network-policies)#

NVIDIA Mission Control’s Kubernetes installation wizard provisions Calico out of the box for network security (Calico Helm chart v3.32.0, includes Tigera Operator). Calico is an open source network security solution for Kubernetes. Using Calico, administrators can manage network access between services within the SuperPod.

Deploy the nmc-network-policies Helm chart to apply additional Calico network policies to the admin (k8s-admin) and user (k8s-user) Kubernetes clusters. Configure a values file to specify the cluster type and the BCM head node CIDRs (bcmHeadNodeCidrs).

The following approach is recommended for switching between clusters via the command line from the BCM head node:

List available Kubernetes modules#

module avail kubernetes
------------------------------------------------------------------------------------------- /cm/local/modulefiles -------------------------------------------------------------------------------------------
kubernetes/k8s-admin/1.32.7-1.1  kubernetes/k8s-user/1.32.7-1.1

Display which modules are presently loaded#

module list
Currently Loaded Modulefiles:
 1) shared   2) cluster-tools/11.0   3) cm-image/11.0   4) cmd   5) cmsh   6) cm-setup/11.0   7) slurm/slurm/24.11   8) kubernetes/k8s-admin/1.32.7-1.1

Set access to a cluster#

module load kubernetes/k8s-user

Switch between Kubernetes clusters#

module swap kubernetes/k8s-admin kubernetes/k8s-user

Apply network policies to the k8s-admin cluster#

For the k8s-admin cluster:

Log in to the BCM Head Node and load the k8s-admin module so kubectl targets the admin cluster.
Download the nmc-network-policies chart from the nmc-network-policies chart on NGC to the BCM Head Node.

helm fetch https://helm.ngc.nvidia.com/nvidia/nv-mission-control/charts/nmc-network-policies-2.4.9.tgz --username='$oauthtoken' --password=<YOUR API KEY>

Create a values file (for example, values-admin.yaml) that sets the cluster type to admin and the BCM head node CIDRs:

# Specify admin cluster
clusterType:
  user: false
  admin: true
# CIDRs of the BCM head nodes
bcmHeadNodeCidrs:
  - "203.0.113.0/24"   # Replace with your BCM head node CIDR

Install or upgrade the chart:

helm upgrade --install nmc-network-policies ./nmc-network-policies-2.4.9.tgz -f values-admin.yaml --wait

Apply network policies to the k8s-user cluster#

For the k8s-user cluster where Run:ai is deployed:

Log in to the BCM Head Node and load the k8s-user module so kubectl targets the user cluster.

Download the nmc-network-policies chart from the nmc-network-policies chart on NGC to an accessible location on the BCM Head Node. For example:

helm fetch https://helm.ngc.nvidia.com/nvidia/nv-mission-control/charts/nmc-network-policies-2.4.9.tgz --username='$oauthtoken' --password=<YOUR API KEY>

Create a values file (for example, values-user.yaml) that sets the cluster type to user and the BCM head node CIDRs:

# Specify user cluster
clusterType:
  admin: false
  user: true
# CIDRs of the BCM head nodes
bcmHeadNodeCidrs:
  - "203.0.113.0/24"   # Replace with your BCM head node CIDR

Install or upgrade the chart:

helm upgrade --install nmc-network-policies ./nmc-network-policies-2.4.9.tgz -f values-user.yaml --wait

Configuring Border Top Of Rack (BTOR) switching#

To secure access to the kubeapi-server server TCP port, we’ll configure a ACL to restrict access.

Perform the following configuration on both BTOR switches.

Create IP ACLs for each address in the control plane to permit communication with kubeapi-server. IPs in the example would be replaced with the IPs on the current system (BCM head node IPs, VIP, and Admin K8s cluster) for the admin space Kubernetes cluster:

nv set acl acl-kubeapi type ipv4
nv set acl acl-kubeapi rule 10 match ip tcp dest-port 6443
nv set acl acl-kubeapi rule 10 match ip source-ip 7.241.28.8/31
nv set acl acl-kubeapi rule 10 action permit
nv set acl acl-kubeapi rule 20 match ip tcp dest-port 6443
nv set acl acl-kubeapi rule 20 match ip source-ip 7.241.28.10/32
nv set acl acl-kubeapi rule 20 action permit
nv set acl acl-kubeapi rule 30 match ip tcp dest-port 6443
nv set acl acl-kubeapi rule 30 match ip source-ip 7.241.28.12/31
nv set acl acl-kubeapi rule 30 action permit
nv set acl acl-kubeapi rule 40 match ip tcp dest-port 6443
nv set acl acl-kubeapi rule 40 match ip source-ip 7.241.28.14/32
nv set acl acl-kubeapi rule 40 action permit

Create catch-all deny ACLs:

nv set acl acl-kubeapi rule 50 match ip tcp dest-port 6443
nv set acl acl-kubeapi rule 50 action deny

Apply to the physical interfaces, NOT the bond interfaces that the hosts are connected. Each of the hosts is connected twice to each BTOR. You must use inband physical ports, NOT out-of-band physical ports:

nv set interface swp3s1 acl acl-kubeapi outbound
nv set interface swp3s2 acl acl-kubeapi outbound
nv set interface swp2s2 acl acl-kubeapi outbound
nv set interface swp6s3 acl acl-kubeapi outbound
nv set interface swp5s0 acl acl-kubeapi outbound

Save and apply the configuration:
```
nv config apply
nv config save
```

Configuring Kyverno#

The Kubernetes API can also be hardened to restrict capabilities to only those services which need them. To do this, NVIDIA Mission Control leverages Kyverno, a widely employed declarative engine (a Kubernetes admission controller) to apply governance policies.

Installing Kyverno#

For both the k8s-user and k8s-admin Kubernetes clusters, the same approach and set of commands is used to install the Kyverno engine.

Add the Kyverno Helm repository and create the namespace:

helm repo add kyverno https://kyverno.github.io/kyverno/
kubectl create ns kyverno
kubectl label namespace kyverno zarf.dev/agent=ignore

Note

The label command ensures that Zarf won’t interfere with the Kyverno namespace.

Copy the values-kyverno.yaml file to an accessible location on the BCM Head Node. Configure the NGC CLI with ngc config set, including a default org, before you run the download.
```
ngc registry resource download-version "nvidia/nv-mission-control/nmc-k8s-security:2.4.9"
```
This file configures resource limits and tolerations for Kyverno controllers to run on control plane nodes. Use it as a template to configure the resource limits and tolerations for your environment. For instance, you might need to increase the memory limits if you have a lot of policies.
```
admissionController:
   container:
      resources:
         limits:
            memory: 3840Mi # increase this if you have a lot of policies
         requests:
              cpu: 1000m
              memory: 1280Mi # increase this if you have a lot of policies
```

Install Kyverno (Helm chart 3.8.2, installs Kyverno v1.18.2):

helm upgrade --install kyverno kyverno/kyverno --namespace kyverno --version 3.8.2 --values values-kyverno.yaml --wait --timeout 10m

Verify the installation:
```
kubectl get pods -n kyverno
```
All Kyverno pods should be in Running state before proceeding.

Applying Security Policies#

The user and admin space Kubernetes clusters require different Kyverno policies because they run different software.

Apply policies to the k8s-admin cluster#

For the admin space Kubernetes cluster (k8s-admin):

Log in to the BCM Head Node and ensure the k8s-admin module is loaded:
```
module load kubernetes/k8s-admin
```

Download the nmc-kyverno-policies Helm chart from the NGC NMC Collections (refer to the note above).

helm fetch https://helm.ngc.nvidia.com/nvidia/nv-mission-control/charts/nmc-kyverno-policies-2.4.9.tgz --username='$oauthtoken' --password=<YOUR API KEY>

Create a values file (for example, values-kyverno-admin.yaml) that sets the cluster type to admin:
```
clusterType:
  admin: true
  user: false
```

Install or upgrade the chart:

helm upgrade --install nmc-kyverno-policies ./nmc-kyverno-policies-2.4.9.tgz -f values-kyverno-admin.yaml -n kyverno --wait

Verify the policies are applied:
```
kubectl get clusterpolicy
```
You should see the applied ClusterPolicy resources listed.

Apply policies to the k8s-user cluster#

For the user space Kubernetes cluster (k8s-user) where Run:ai is deployed:

Log in to the BCM Head Node and ensure the k8s-user module is loaded:
```
module load kubernetes/k8s-user
```
Download the nmc-kyverno-policies chart from the nmc-kyverno-policies chart on NGC to the BCM Head Node.

helm fetch https://helm.ngc.nvidia.com/nvidia/nv-mission-control/charts/nmc-kyverno-policies-2.4.9.tgz --username='$oauthtoken' --password=<YOUR API KEY>

Create a values file (for example, values-kyverno-user.yaml) that sets the cluster type to user:
```
clusterType:
  user: true
```

Install or upgrade the chart:

helm upgrade --install nmc-kyverno-policies ./nmc-kyverno-policies-2.4.9.tgz -f values-kyverno-user.yaml -n kyverno --wait

Verify the policies are applied:
```
kubectl get clusterpolicy
```
You should see the applied ClusterPolicy resources listed.

Validating Policy Enforcement#

After applying policies, you can verify they are enforcing security restrictions:

# Check for policy violations
kubectl get policyreport -A

# View details of a specific policy
kubectl describe clusterpolicy disallow-privileged-containers

Policy enforcement defaults to Enforce. Matching policy violations can block new admission requests, subject to policy exclusions and existing-resource handling. Background scans report violations on existing resources without stopping them. A policy report alone does not confirm admission blocking.

GitOps Policy Enforcement Override#

For a deployment approved by NVIDIA Mission Control Product Management, set spec.components.argocdAppsK8sAdmin.parameters.kyvernoPolicies.validationFailureAction to Audit in the cluster spec. For the user cluster, use spec.components.argocdAppsK8sUser.parameters.kyvernoPolicies.validationFailureAction. Configure each cluster separately. For example:

spec:
  components:
    argocdAppsK8sAdmin:
      parameters:
        kyvernoPolicies:
          validationFailureAction: Audit

Audit allows requests that violate matching upstream kyverno-policies admission rules while recording violations. This applies across that chart’s policies on the selected cluster, so non-compliant workloads can be admitted. The NVIDIA Mission Control Kyverno overlay and Calico network policies, including default-deny, remain in effect. The overlay’s existing optional policy flags retain their defaults.

To restore admission blocking, remove the override or set it to Enforce. Apply the updated cluster spec to the affected cluster:

deployctl apply -f <cluster-spec>.yaml argocdAppsK8sAdmin

For the user cluster, use argocdAppsK8sUser instead. After Argo CD syncs, inspect the effective action with kubectl describe clusterpolicy. If allowExistingViolations: true applies, an update to a resource that already violated the policy can remain non-compliant. An update after that resource becomes compliant is subject to enforcement.

Run:ai Project Namespaces#

The k8s-user policies do not restrict host paths or host namespaces for Run:ai workloads. The user-cluster policy values exempt the runai namespace, and any namespace that carries the runai/queue label, from the disallow-host-path and disallow-host-namespaces policies. Run:ai applies that label to its project namespaces, so a workload submitted through Run:ai can mount a host path even when policy validation succeeds.

The exemption exists because Run:ai GPU fractions mount a host path into each fractional workload pod by default. Run:ai can use the device plugin for that mount instead when you set clusterConfig.global.devicePluginBindings: true in the Run:ai cluster configuration. NVIDIA has not validated NVIDIA Mission Control with that setting and the exemption removed, so do not remove the exemption from the policy values.

Restrict host paths for Run:ai workloads through Run:ai workload policy instead. Set the following values in the policy:

runAsNonRoot: true
runAsUid with canEdit: false
allowPrivilegeEscalation: false
storage.hostPath.instances.canAdd: false

Running Workloads Outside Run:ai (Raw Kubernetes)#

Submit workloads through Run:ai projects and queues whenever you can. Those workloads land in project namespaces labeled with runai/queue, which is the supported placement model on k8s-user.

If you deploy with kubectl or Helm directly and omit the namespace, Kubernetes places the workload in default. On NVIDIA Mission Control clusters, default is not exempt from the Kyverno Pod Security policies, so privileged containers, hostPath, hostNetwork, host ports, and extra capabilities are rejected there.

Use the following pattern instead:

Create a dedicated namespace.
Deploy with -n or --namespace.
Keep the chart non-privileged whenever you can.

kubectl create namespace my-app
kubectl apply -f chart.yaml -n my-app
# or
helm upgrade --install my-app ./chart -n my-app --create-namespace

If the chart must be privileged, do not deploy it to default. Use a dedicated namespace and add a narrow policyExclude entry in the nmc-kyverno-policies values for only the policies that the chart requires. Treat that entry as a documented security exception: scope it with RBAC, and avoid broad wildcards unless you intend them.

Add the entry under kyverno-policies-user in the values file that you used to install the chart on k8s-user (for example, values-kyverno-user.yaml). A policyExclude key at the top level of the file has no effect.

Your entry replaces the chart’s default exclusion list for that policy. It does not add to it. Copy the chart’s full default list for the policy into your entry, and then append your namespace. To display the defaults, run:

helm show values ./nmc-kyverno-policies-2.4.9.tgz

For example, to exempt my-privileged-app from disallow-privileged-containers with chart version 2.4.9:

kyverno-policies-user:
  policyExclude:
    disallow-privileged-containers:
      any:
        - resources:
            kinds:
              - Pod
            namespaces:
              # Chart defaults for this policy; keep all of them
              - cm
              - gpu-operator
              - local-path-storage
              - calico-system
              - metallb-system
              - lws-system
              - knative-serving
              - nvidia-dra-driver-gpu
              - mpi-operator
              - network-operator
              - sriov-network-operator
              # Your namespace
              - my-privileged-app

If you omit the defaults, the chart’s system namespaces lose their exemption. For example, the NVIDIA GPU driver pods in gpu-operator are then rejected. For disallow-host-path and disallow-host-namespaces, the defaults also include the runai namespace and a namespaceSelector for namespaces labeled runai/queue. If you omit them, Run:ai workloads that use hostPath volumes or host namespaces are rejected.

Apply nmc-kyverno-policies again, and then verify the result with kubectl describe clusterpolicy.

Configuring PermissionManager#

When adding new users to Kubernetes, it is important to restrict user capabilities to only those capabilities that are necessary for the user.

Run:ai features an advanced authorization policy mechanism to control access and rights for the user space Kubernetes cluster. If users do not need kubectl access, Run:ai’s authorization mechanism is the best way to restrict functionality for the user space cluster.

If users need access to the user space Kubernetes cluster, PermissionManager is included with the Kubernetes deployment. PermissionManager is an open source tool for Kubernetes role based access control (RBAC) management. To configure PermissionManager, refer to their documentation.