BYOK Requirements
Review the infrastructure, components, and access that you need to connect your Kubernetes cluster to DGX Cloud Lepton.
Prepare these items before you connect your cluster. Required items apply to all BYOK clusters. Recommended items are deployment guidance. Conditional items apply only to the features that you use. Registration does not check all of these items.
Responsibilities and Access
| Area | You provide and maintain | Lepton manages |
|---|---|---|
| Infrastructure | Machines, capacity, repairs, replacement, and data protection. | Workloads on the nodes that you add. Lepton does not provision, replace, or delete your machines. |
| Kubernetes | Control plane, OS, kubelet, container runtime, networking, storage, and their compatibility. | Platform components and workloads, through the authorized connection to your Kubernetes API. |
| Prerequisites | Installation, configuration, and upgrades of the required components. | Platform components that Lepton installs after you connect the cluster. |
| Cluster agent | Authorization, installation, upgrades, and removal. | Agent releases and the Lepton services that keep the connection. |
| Node operations | Node preparation, infrastructure maintenance, and maintenance windows. | Scheduling state, workload drain, health monitoring, and the recovery policies that you enable. |
Review this access with your cluster administrator before you install the
cluster agent. The agent gets ongoing Kubernetes cluster-admin access to all
of your cluster, with Nodes, workloads, Secrets, and RBAC in all namespaces.
The outbound tunnel does not reduce this access.
Your admission and security policies must allow the CRDs, webhooks, RBAC, and node components of the platform. Some of these components use privileged containers and host mounts. Configure exceptions for these components. Do not disable admission policies for all of the cluster. The namespace of the cluster agent must allow the Pod Security baseline level. The restricted level blocks the agent.
Lepton does not apply NetworkPolicies on BYOK clusters. Thus, Lepton does not isolate network traffic between workspaces, or between workspaces and your other workloads.
Lepton uses namespaces that start with ws- for workspaces. Do not use this prefix for your own namespaces. Drain and Release can evict the Pods in these namespaces.
If you enable Auto Reboot for a node group, Lepton can reboot an unhealthy node that has no workloads. Workspace administrators can also use Force Reboot on a node. A reboot stops all applications on the machine, not only Lepton workloads.
If you remove the cluster agent, the other platform components and their permissions stay in your cluster. See Disconnect the Cluster.
Kubernetes, Networking, and Storage
| Requirement | Level | What to prepare |
|---|---|---|
| Kubernetes | Required | Version 1.36 or later. Keep the nodes Ready, and keep the kubelet versions in the Kubernetes version skew policy. |
| Cluster capabilities | Required | Standard Kubernetes APIs, admission webhooks, and projected ServiceAccount tokens. Keep the clocks of the nodes synchronized. |
| Installation access | Required | The workspace administrator role in Lepton and cluster-admin access to your cluster. On the installation host: bash, curl, kubectl, and Helm 3.14 or later, or Helm 4. |
| Cluster networking | Required | A CNI that supports Kubernetes NetworkPolicy, cluster DNS, and Pod access to the Kubernetes API and to the necessary services. |
| IP family | Required | IPv4. For IPv6-only or dual-stack clusters, speak with Lepton before you connect the cluster. |
| Outbound connectivity | Required | DNS and HTTPS (TCP 443) to the enrollment and tunnel endpoints. Access to image registries, installation artifacts, health reporting, and observability destinations. See Outbound Connections. |
| LoadBalancer Services | Required | A controller or provider integration that gives type: LoadBalancer Services addresses that their clients can reach. Lepton does not install this integration. |
| Persistent storage | Conditional | For Slurm or image acceleration, provide a CSI driver with dynamic provisioning and a default StorageClass for ReadWriteOnce volumes. Image acceleration is off by default on BYOK. When it is on, it always uses the default class. Slurm uses the default class unless you set storage classes in the Slurm configuration. |
A StorageClass does not supply local disk space for container images and logs. Size the local disk of each node for these files.
Outbound Connections
- The tunnel uses long-lived gRPC/HTTP2 connections over TCP 443.
- After registration, the cluster agent continues to use the enrollment endpoint to renew its certificates. Keep this endpoint reachable.
- Do not put a TLS-intercepting proxy in the connections to the enrollment and tunnel endpoints.
- Some observability destinations use ports other than 443.
- Lepton gives you the list of destinations for your environment.
Platform Capacity
The cluster agent and the platform components need capacity on CPU nodes outside the GPU nodes that you add to node groups. Lepton requests about 21 vCPU and 92 GiB RAM. During upgrades, this can increase to about 31 vCPU and 125 GiB RAM. You can use any node shapes that meet these conditions:
- At least two nodes. The cluster agent runs two replicas on different nodes.
- One node with 8 vCPU and 50 GiB RAM available for the largest platform Pod.
- To drain a node without interruption, a second node with the same available capacity.
Add capacity for your own components on these nodes. For example, three nodes with 16 vCPU and 128 GiB RAM each meet all conditions. For evaluation, two nodes with 16 vCPU and 64 GiB RAM each are enough, but you cannot drain a node.
GPU Nodes
Use the BYOC hardware recommendations as a reference for CPU, memory, and local storage. The BYOC provisioning and disk management procedures do not apply to BYOK. You configure the disks and the host software.
| Requirement | Level | What to prepare |
|---|---|---|
| GPU software | Required | Compatible NVIDIA drivers, container toolkit, and container runtime configuration. GPU workloads must have access to the devices and the driver libraries. You do not need a CUDA toolkit or NVCC on the host. |
| GPU Operator | Required | GPU Operator in the gpu-operator namespace, with the container toolkit, device plugin, GPU feature discovery, DCGM exporter, and validator enabled. Lepton checks these Pods before a node becomes Healthy. |
| Other taints | Required | No taints other than nvidia.com/gpu, and the node is not cordoned. Lepton does not complete the addition of a node that has other taints. |
| DaemonSets | Recommended | An operator: Exists toleration on all DaemonSets. This includes your own DaemonSets. Do not set key, effect, or tolerationSeconds on this toleration. |
| GPU node taint | Recommended | The nvidia.com/gpu:NoSchedule taint, applied before you connect the cluster. It keeps Pods without a matching toleration off the GPU nodes. If the taint is not there, Lepton adds it. |
| Other workloads | Recommended | Move your other applications off the GPU nodes before you add them. Keep only the necessary infrastructure Pods. |
| GB200 and GB300 nodes | Conditional | The NVIDIA DRA driver for ComputeDomains, and the nvidia.com/gpu.clique node label from GPU feature discovery. Lepton checks both before these nodes become Healthy. |
| Specialized GPU networking | Conditional | The Fabric Manager, RDMA drivers and fabric, and MNNVL prerequisites for your hardware. Lepton does not configure the physical network. |
When you add a node to a node group, Lepton adds a NoExecute taint to the node. Kubernetes evicts the Pods that do not tolerate this taint. Lepton also adds temporary taints while it prepares the node.
Required Components
Install and maintain these prerequisites before you connect your cluster. Use the recommended versions or newer compatible releases. A newer release must keep the capabilities and configuration in this guide. Select releases that are compatible with your OS, hardware, and Kubernetes distribution.
| Component | Recommended version | Required capability |
|---|---|---|
| cert-manager | v1.21.2 | Certificate controller and CRDs for platform certificates and webhooks. |
| kube-prometheus-stack | 75.13.0 | Prometheus Operator, monitoring CRDs such as PrometheusAgent and PodMonitor, and node-exporter on all nodes that you add. Lepton installs its own Prometheus agent and kube-state-metrics. |
| NVIDIA GPU Operator | v26.3.3 | Device plugin, container toolkit, GPU feature discovery, DCGM exporter, and validator for GPU nodes, in the gpu-operator namespace. |
| NVIDIA DRA driver | 0.5.0 | Only for GB200 and GB300 nodes: ComputeDomains for multi-node NVLink (MNNVL). |
cert-manager 1.21 is the first release that supports Kubernetes 1.36. If you already have these components at the recommended versions or later, you can keep them. Examine their configuration first.
Lepton GPU workloads request the nvidia.com/gpu extended resource. Keep the
NVIDIA device plugin enabled, and do not use the GPUCluster mode of GPU
Operator. Use DRA only for ComputeDomains, because ResourceClaim GPU
allocation does not replace nvidia.com/gpu.
Next, prepare your cluster.