Manage Your BYOK Cluster

After you connect your cluster to DGX Cloud Lepton, you continue to operate the cluster and its machines. Use the Dashboard to manage Lepton workloads and the nodes in node groups. Use your own tools to maintain Kubernetes, networking, storage, and the prerequisites.

Monitor and Maintain Nodes

Monitor node health and workload status in the Dashboard. Also monitor the Kubernetes Node conditions and your infrastructure. Investigate infrastructure faults with your usual operations process.

Lepton can also do node-level operations. If Auto Reboot is enabled for a node group, Lepton can reboot an unhealthy node. Workspace administrators can use Force Reboot. Include these operations in your maintenance plans.

For planned maintenance, do these steps for each node. For details of each operation, see Node Operations.

1. In the Nodes tab of the node group, select Pause Schedule. Lepton stops scheduling new workloads on the node.
2. Tell the workload owners to checkpoint, stop, or move their work. If necessary, make sure that replacement capacity is available.
3. Select Drain. If the workloads can move, select Graceful Drain. If not, select Drain.
4. Wait until the operation completes. Make sure that the Lepton workloads stopped or moved.
5. Examine your own workloads and infrastructure Pods on the node.
6. Do the maintenance.
7. Make sure that the node is Ready, the DaemonSets run, nvidia.com/gpu resources show, and Lepton shows the node as Healthy.
8. Select Resume Schedule.

Upgrade Components

Prerequisites

You upgrade Kubernetes, the drivers, the container runtime, cert-manager, Prometheus Operator, node-exporter, GPU Operator, and the DRA driver. Use the recommended versions or newer releases that keep the required capabilities and configuration.

During upgrades, keep this configuration:

• The operator: Exists toleration on all DaemonSets.
• GPU Operator in the gpu-operator namespace and in ClusterPolicy mode, with its components enabled.
• The nvidia.com/gpu resource from the device plugin.

After an upgrade, do the GPU node checks again. Then run a typical GPU workload. If you use ComputeDomains, validate them separately.

Cluster Agent

To upgrade the cluster agent, run the installer with --upgrade. You do not need a registration token. The installer upgrades to the latest published release of the cluster agent.

Make sure that you use the correct cluster, and that the release exists:

Run the installer. If you installed with a custom namespace or release name, also add --namespace=NAMESPACE and --release-name=RELEASE_NAME:

Wait for the rollout, and make sure that all containers are Ready:

Make sure that the Dashboard still shows the cluster as READY.

The upgrade keeps the chart values that you set before. If the release does not exist, the upgrade fails. The install command without --upgrade does not change a release that exists.

You do not need Renew for an upgrade. Renew replaces the token of a registration that did not complete. If the identity of the registered agent is lost, contact Lepton support before you register again.

Add or Release Capacity

To add capacity, prepare more GPU nodes. Then add them to a node group. Lepton does not provision machines for a BYOK node group.

To release a node:

1. Examine the workloads on the node. Protect or move data as necessary.
2. In the menu of the node, select Release.
3. Select Graceful Release or Release.
4. Optionally, enter a reason.
5. Submit the request.
6. Wait until the release and cleanup complete. Then use the node for other work.

Release keeps the Kubernetes Node and the machine. Lepton removes its labels, annotations, and taints from the Node. Your labels, the provider labels, and the nvidia.com/gpu taint stay. Release also uncordons the node. If you must keep workloads off the node, cordon it again. Release does not remove the platform components or the prerequisites.

Disconnect the Cluster

Keep the cluster agent connected until Lepton completes the node release and the platform cleanup.

1. Stop new workloads. Handle the current workloads, persistent data, and service endpoints that use the cluster.
2. Release all nodes from their node groups. Wait until the cleanup completes.
3. Delete the node groups of the cluster.
4. Delete the domains that route traffic to the cluster.
5. Work with Lepton support to find and remove the remaining platform components, resources, and permissions. Protect your own components and data.
6. On the Kubernetes Clusters page, delete the cluster. Lepton blocks this step until you delete the node groups and domains.

After you delete the cluster, uninstall the cluster agent. If you use a custom release name or namespace, use your values:

Then examine the namespace of the cluster agent. Secrets with private key material, a ConfigMap, and Leases stay after the uninstall. When the namespace contains nothing that you need, delete it.

If you delete the lepton-cluster-agent ClusterRoleBinding, you revoke only the cluster-admin access of the cluster agent. Other components keep their permissions. For an urgent revocation, use your incident process and contact Lepton support. If you disconnect early, Lepton cannot do the remote cleanup. Record what remains, and complete the cleanup yourself. Disconnection does not return the cluster to its exact state before the connection.