Prepare Your Kubernetes Cluster

Do these steps in your cluster before you connect it to DGX Cloud Lepton. First, read BYOK Requirements.

You install and maintain these prerequisites. If a compatible component is already installed, update its configuration. Do not install a second copy. The commands use the recommended versions. To use a newer release, change the chart version and follow the installation instructions of that release.

1. Check Your Cluster

Use a kubeconfig with cluster-admin access to your cluster. Before you install components, make sure that the context is correct:

Make sure that:

• All GPU nodes that you plan to add are Ready.
• Cluster DNS, the CNI, LoadBalancer provisioning, and the outbound connections operate correctly.
• Your cluster has the necessary platform capacity.

2. Prepare GPU Nodes and DaemonSets

Move your other applications off the GPU nodes that you plan to add. Keep the infrastructure Pods for networking, storage, and GPU access.

Remove all taints other than nvidia.com/gpu from each GPU node, and uncordon the node. Lepton does not complete the addition of a node that has other taints. If another controller added a taint, find its source before you remove it. Replace GPU_NODE_NAME with the name of the Node, and TAINT_KEY with the key of the taint:

Apply the nvidia.com/gpu:NoSchedule taint to each GPU node. This taint keeps Pods without a matching toleration off the node:

Add this toleration to the Pod template of all DaemonSets, and also of your own DaemonSets. If possible, set it through the Helm values of the component:

A toleration without key and effect tolerates all taints. Remove the tolerations that set tolerationSeconds without a key, because Kubernetes evicts the Pod after that time. Do not add this toleration to other workload types, such as Deployments.

3. Install cert-manager

For a new installation, the chart also installs the CRDs. If you uninstall the Helm release, the CRDs stay in the cluster.

Make sure that the controller, webhook, and cainjector Deployments are available. If cert-manager is already installed, keep its current installation method. Follow the cert-manager upgrade procedure before you change the CRDs or the chart version. For version guidance, see Required Components.

4. Install Prometheus Operator and node-exporter

Lepton needs Prometheus Operator, its monitoring CRDs, and node-exporter on all nodes that you add. Lepton installs its own Prometheus agent and kube-state-metrics. This example installs only the necessary parts of kube-prometheus-stack:

Make sure that the Operator Deployment is available and that the CRDs exist. Make sure that a node-exporter Pod runs on all nodes that you plan to add.

Lepton collects the node-exporter metrics through its own PodMonitor. The example disables the node-exporter ServiceMonitor to prevent duplicate metrics.

If you already have Prometheus Operator or node-exporter, do not install a second copy. Instead, make sure that:

• The Operator watches the kube-prometheus-stack namespace. Lepton installs its Prometheus agent in this namespace.
• The node-exporter Pods have the label app.kubernetes.io/name=prometheus-node-exporter, expose port 9100, and tolerate all taints.

Do not apply this reduced configuration to a full monitoring stack. First, examine the components that it removes.

5. Install GPU Operator

GPU Operator supplies the device plugin that publishes nvidia.com/gpu. Lepton GPU workloads need this extended resource, also when DRA is installed.

Install GPU Operator in the gpu-operator namespace. Keep the container toolkit, device plugin, GPU feature discovery, DCGM exporter, and validator enabled. Lepton checks these Pods before a GPU node becomes Healthy.

If GPU Operator installs and manages the NVIDIA driver, use this command:

The two toleration settings are necessary. The first applies to the GPU Operator node components. The second applies to the Node Feature Discovery workers. Make sure that these Pods run on all GPU nodes that you plan to add.

The devicePlugin.enabled=true setting keeps the device plugin enabled. GPU Operator v26.7 and later also has a GPUCluster mode, which allocates GPUs through DRA. Do not enable this mode with gpuCluster.deployCR=true. It does not deploy the device plugin, so the nodes do not publish nvidia.com/gpu.

If you manage the NVIDIA driver on the hosts, add --set driver.enabled=false. Keep the driver and the Fabric Manager compatible with your hardware. For the containerd configuration and socket paths of your distribution, see the NVIDIA installation guide.

6. Install the DRA Driver for GB200 and GB300 Nodes

GB200 and GB300 nodes need the NVIDIA DRA driver for ComputeDomains. Before these nodes become Healthy, the DRA kubelet plugin must be Ready, and the nodes must have the nvidia.com/gpu.clique label. For other GPUs, skip this step.

The DRA driver manages only ComputeDomains. GPU allocation continues to use the device plugin and nvidia.com/gpu.

Before you install the driver, meet the ComputeDomain prerequisites. These include compatible drivers, Container Toolkit with CDI, the NVLink fabric, and the IMEX configuration.

Use the release name and namespace in this command. Lepton finds the kubelet plugin by its DaemonSet name and namespace. If GPU Operator manages the driver, use this command:

Set nvidiaDriverRoot for your driver installation:

Make sure that the controller is available, and that the kubelet plugin runs on each GB200 and GB300 node. Make sure that the CRDs and DeviceClasses exist, and that each node has the nvidia.com/gpu.clique label. Ready Pods do not prove cross-node communication. Validate it separately for your hardware and workloads.

7. Check the GPU Nodes

Examine the GPU capacity, labels, taints, and Pods on each GPU node:

Make sure that each GPU node meets these conditions:

• The node is Ready and not cordoned. Capacity and Allocatable show the expected nvidia.com/gpu count.
• The nvidia.com/gpu.product label identifies the GPU.
• The node has no taints other than nvidia.com/gpu.
• The GPU Operator Pods and a Node Feature Discovery worker Pod are Ready on the node.
• A node-exporter Pod is Ready on the node.
• Each DaemonSet that must run on the node has a Ready Pod there.
• On GB200 and GB300 nodes, the DRA kubelet plugin Pod is Ready, and the node has the nvidia.com/gpu.clique label.
• Your other applications are off the node.

Optional GPU Smoke Test

Before you add the node, save this Job as byok-gpu-check.yaml. Replace GPU_NODE_HOSTNAME with the kubernetes.io/hostname label of the node. The Job requests one GPU. It runs nvidia-smi and initializes the CUDA driver in a Python image without CUDA libraries:

Run this block. It creates a namespace, runs the check, shows the output, and removes the test resources if the check passes. If the namespace already exists, the block stops. Do not use or delete a namespace that belongs to another task.

The check passes if the Job completes, nvidia-smi shows the allocated GPU, and the output shows cuInit(0) = 0. If the check fails, examine the Pod, its logs, and its events. Then run the last two commands of the block to remove the test resources. Do the check on other GPU nodes as necessary. This check covers scheduling, device access, and CUDA initialization for one GPU. It is not a GPU stress test or an MNNVL validation.

When all checks pass, connect your cluster to Lepton.