Kubernetes Security Policies (Air-Gapped)#

Install Kyverno and the NMC security policies on each Kubernetes cluster that you deploy. The k8s-admin and k8s-user clusters use the same Helm charts, but each cluster needs its own Kubernetes context and policy values. Complete the procedure for k8s-admin after installing the NMC components on it, and repeat it for k8s-user after installing Run:ai.

Before installing the policies, import and verify the shared bundle as described in Importing Artifacts into the Air-Gapped Environment. The registry must contain the images required by the charts; installing the same chart on a second cluster does not require a second image import.

Run the following commands on the BCM head node from the directory that holds bundle. Confirm that the three Helm charts are present:

test -f ./bundle/helm/kyverno-3.8.2.tgz
test -f ./bundle/helm/nmc-kyverno-policies-2.4.9.tgz
test -f ./bundle/helm/nmc-network-policies-2.4.9.tgz

If a chart is missing, check the bundle contents and the artifact download in Downloading NVIDIA Mission Control Artifacts with nmc-airgap and Helper Scripts before proceeding. Prepare values-kyverno.yaml for the Kyverno engine using the resource limits and tolerations in Kubernetes Security Hardening. You can use that file on both clusters.

Install Policies on the Admin Cluster#

Load the k8s-admin module on the BCM head node. If the k8s-user module is already loaded, swap the modules instead. Confirm that kubectl targets k8s-admin before running Helm:
```
module load kubernetes/k8s-admin
kubectl config current-context
kubectl get nodes
```
Save the following Kyverno policy values as values-kyverno-admin.yaml:
```
clusterType:
  admin: true
  user: false
```
Save the following network policy values as values-admin.yaml. Replace the example CIDR with the CIDR of your BCM head nodes:
```
clusterType:
  admin: true
  user: false
bcmHeadNodeCidrs:
  - "203.0.113.0/24"
```

Create and label the Kyverno namespace so Zarf does not interfere with it. Then install Kyverno and the two policy charts from the bundle:

kubectl create ns kyverno
kubectl label namespace kyverno zarf.dev/agent=ignore
helm upgrade --install kyverno ./bundle/helm/kyverno-3.8.2.tgz \
  --namespace kyverno --values values-kyverno.yaml \
  --wait --timeout 10m
helm upgrade --install nmc-kyverno-policies ./bundle/helm/nmc-kyverno-policies-2.4.9.tgz \
  --namespace kyverno --values values-kyverno-admin.yaml --wait
helm upgrade --install nmc-network-policies ./bundle/helm/nmc-network-policies-2.4.9.tgz \
  --values values-admin.yaml --wait

Verify that the Kyverno pods are running and the policies are installed on k8s-admin:

kubectl get pods -n kyverno
kubectl get clusterpolicy
helm status nmc-kyverno-policies -n kyverno
helm status nmc-network-policies

Install Policies on the User Cluster#

The k8s-user cluster hosts Run:ai. Use its context and user-specific values; the admin policy values select the wrong policy set for this cluster.

Load the k8s-user module on the BCM head node. If k8s-admin is already loaded, use module swap kubernetes/k8s-admin kubernetes/k8s-user instead. Confirm that kubectl targets k8s-user before running Helm:
```
module load kubernetes/k8s-user
kubectl config current-context
kubectl get nodes
```
Save the following Kyverno policy values as values-kyverno-user.yaml:
```
clusterType:
  admin: false
  user: true
```
Save the following network policy values as values-user.yaml. Replace the example CIDR with the CIDR of your BCM head nodes:
```
clusterType:
  admin: false
  user: true
bcmHeadNodeCidrs:
  - "203.0.113.0/24"
```

Install Kyverno on k8s-user and then install its policy charts. These Helm releases are separate from the releases on k8s-admin:

helm upgrade --install kyverno ./bundle/helm/kyverno-3.8.2.tgz \
  --namespace kyverno --create-namespace --values values-kyverno.yaml \
  --wait --timeout 10m
helm upgrade --install nmc-kyverno-policies ./bundle/helm/nmc-kyverno-policies-2.4.9.tgz \
  --namespace kyverno --values values-kyverno-user.yaml --wait
helm upgrade --install nmc-network-policies ./bundle/helm/nmc-network-policies-2.4.9.tgz \
  --values values-user.yaml --wait

Verify the installation on k8s-user:

kubectl get pods -n kyverno
kubectl get clusterpolicy
helm status nmc-kyverno-policies -n kyverno
helm status nmc-network-policies

Warning

These policies do not restrict host paths for Run:ai workloads. The user-cluster policy values exempt the runai namespace, and any namespace carrying the runai/queue label, from the disallow-host-path and disallow-host-namespaces policies. Run:ai applies that label to its project namespaces, so a workload submitted through Run:ai can still mount a host path even when the preceding verification reports success.

The exemption exists because Run:ai GPU fractions mount a host path into each fractional workload pod by default. Run:ai can use the device plugin for that mount instead when you set clusterConfig.global.devicePluginBindings: true in the Run:ai cluster configuration. NVIDIA has not validated NVIDIA Mission Control with that setting and the exemption removed, so do not remove the exemption from the policy values.

Restrict host paths through Run:ai workload policy instead: set runAsNonRoot: true and runAsUid with canEdit: false, allowPrivilegeEscalation: false, and storage.hostPath.instances.canAdd: false.

For policy enforcement checks and additional Kubernetes security configuration, refer to Kubernetes Security Hardening.