Adding and Removing Nodes from Run:ai or Slurm#

Overview#

Use these instructions to add or remove compute nodes from existing Run:ai or Slurm installations in your NVIDIA Mission Control environment.

Prerequisites#

To move a compute node between workload managers, change its BCM category and reboot it. Configure the target category before moving the node.

Basic requirements:

  • The cluster nodes are managed by BCM

  • Categories are configured for compute nodes that use Run:ai or Slurm

Refer to Node and Category Management and NVIDIA Run:ai Installation for more information on setting up node categories.

Adding or removing a compute node from a Run:ai cluster#

This assumes you already have Run:ai deployed and running.

These steps apply when adding a new compute node, moving a compute node from Slurm to Run:ai, or removing a node from an existing Run:ai cluster.

Add a compute node#

  1. Add the compute node to the proper Run:ai GPU worker category (e.g. “dgx-gb200-k8s”).

  2. If you are moving the node from Slurm, remove slurmd.service through the Run:ai category’s software image, package selection, or provisioning configuration. Ensure that the unit is absent after reboot. Stopping, disabling, or masking the service does not remove it.

  3. Reboot the compute node (wait for it to come back online).

After adding a node, it should appear in the Run:ai control plane as an available worker node. Slurm automatically removes migrated nodes from the relevant partitions.

Autonomous Hardware Recovery (AHR) uses slurmd.service and /var/lib/runai to tell Slurm nodes from Run:ai nodes. When you move a node between workload managers, remove whichever of these the previous workload manager left on the node: the unit, the directory, or both.

After migrating a node from Slurm, reload systemd’s unit definitions and verify that slurmd.service is absent:

sudo systemctl daemon-reload
systemctl list-unit-files | awk '{print $1}' | grep -x 'slurmd.service'

The verification command must produce no output. If it prints slurmd.service, remove the remaining unit from the software image or provisioning configuration and repeat the verification.

Remove a compute node#

  1. Update the compute node to any non-Run:ai category.

  2. Reboot the compute node (wait for it to come back online).

After removing a node, it should no longer appear in the Run:ai control plane automatically.

Note

On BCM 11.33 and earlier, BCM does not automatically delete the compute node from Kubernetes. Instead, it stops running kubelet and transitions to the NotReady state, and BCM assigns it the bcm-kubelet-role-unassigned node role, indicating that BCM no longer manages the node. Because the Run:ai scheduler still accounts for these NotReady nodes, you must either exclude them from Run:ai or delete them from Kubernetes manually with kubectl delete node <node>.

In BCM 11.34 and later, this deletion is the default behavior, so the kubectl delete node command is no longer required for worker nodes.

Adding or removing a compute node from a Slurm cluster#

This assumes you already have Slurm deployed and running.

These steps apply when adding a new compute node, moving a compute node from Run:ai to Slurm, or removing a node from an existing Slurm cluster.

Add a compute node#

  1. Add the compute node to the proper Slurm category (e.g. “dgx-gb200”).

  2. Reboot the compute node (wait for it to come back online).

  3. If you moved the node from Run:ai, remove /var/lib/runai from the node if it remains.

After adding a node, it should appear in Slurm as an available node. If you moved this compute node from an existing Run:ai cluster, Run:ai automatically removes it from its worker pool.

AHR uses slurmd.service and /var/lib/runai to tell Slurm nodes from Run:ai nodes. When you move a node between workload managers, remove whichever of these the previous workload manager left on the node: the unit, the directory, or both.

Remove a compute node#

  1. Update the compute node to any non-Slurm category.

  2. Reboot the compute node (wait for it to come back online).

After removing a node, it should no longer appear in Slurm.

Verify that your compute node appears in the Slurm cluster by running sinfo or scontrol show node <node hostname>.