DocumentationSelf-Managed DeploymentSetupHelm

Multinode Networking

View as Markdown

Multi-node GPU training requires high-bandwidth, low-latency east-west networking between nodes. Without it, NCCL falls back to standard Ethernet and training times increase significantly. The Helm chart installs Kyverno policies that automatically inject the correct networking resources for four cloud providers.

Prerequisites

  • Kyverno installed in the cluster
  • Volcano scheduler installed (multi-node jobs use Volcano)
  • Nodes with high-performance networking hardware (EFA, InfiniBand, etc.)
  • The cloud provider’s device plugin or network operator deployed so that networking resources are visible to Kubernetes

The NeMo Platform does not provision or manage the underlying cloud networking infrastructure. Each cloud provider section below lists the cluster-level prerequisites your administrator must configure. Refer to your cloud provider’s documentation for setup instructions.

How It Works

When a job requests more than one node, the jobs controller annotates the pod with nmp.nvidia.com/enable-multi-node-networking: "true". Kyverno watches for this annotation and mutates the pod spec to inject the cloud-specific device requests, environment variables, and volume mounts that NCCL needs for high-performance inter-node communication.

Configuration

Enable exactly one cloud provider in the multinodeNetworking section of your values.yaml. Enabling more than one causes a Helm install error.

Device-count parameters (efaDevicesPerGPU, rdmaDevicesPerGPU) must match the hardware ratio of your instance type. A mismatch causes jobs to either fail scheduling or silently run without high-speed networking.

AWS (EFA)

Injects Elastic Fabric Adapter device requests (vpc.amazonaws.com/efa), mounts the EFA OFI library, and configures shared memory.

1multinodeNetworking:
2 aws:
3 enabled: true
4 efaDevicesPerGPU: 4 # must match your instance type
ParameterDescription
efaDevicesPerGPUEFA interfaces to request per GPU. The Kyverno policy multiplies this value by the number of GPUs requested by each container to produce the total vpc.amazonaws.com/efa resource request.

To determine the correct value, divide the number of EFA interfaces on your instance type by the number of GPUs. For example, p5.48xlarge has 32 EFA interfaces and 8 GPUs: 32 / 8 = 4. Consult the AWS EFA instance types documentation for your instance type’s EFA count.

Cluster prerequisites:

  • EFA-enabled instance types (e.g., p5.48xlarge, p5e.48xlarge)
  • EFA device plugin deployed
  • Security group permitting all traffic between nodes in the placement group
  • See AWS EFA documentation

Azure (InfiniBand / RDMA)

Injects RDMA device requests, IPC_LOCK capability, NCCL topology, and UCX environment variables.

1multinodeNetworking:
2 azure:
3 enabled: true
4 rdmaDevicesPerGPU: 1
5 rdmaDeviceName: "hca_shared_devices_a"
ParameterDescription
rdmaDevicesPerGPURDMA devices per GPU. Must match your VM size.
rdmaDeviceNameResource name exposed by the RDMA device plugin (e.g., hca_shared_devices_a).

Cluster prerequisites:

  • InfiniBand-capable VM sizes (e.g., Standard_ND96asr_v4, Standard_ND96amsr_A100_v4)
  • RDMA device plugin or Network Operator deployed
  • See Azure InfiniBand documentation

GCP (TCP-XO)

Injects a GPUDirect-TCPXO sidecar daemon, multi-NIC annotations, and NCCL Fastrak environment variables.

1multinodeNetworking:
2 gcp:
3 enabled: true

Cluster prerequisites:

  • A3+ GPU node pools with multi-NIC networking enabled
  • GKE multi-network support configured with the required gpu-nic networks
  • See the GKE GPUDirect-family documentation, specifically the TCPXO sections

OCI (SR-IOV / RDMA)

Injects Mellanox NIC device requests (nvidia.com/mlnxnics), SR-IOV network annotations via Multus, and InfiniBand-tuned NCCL environment variables.

1multinodeNetworking:
2 oci:
3 enabled: true
4 rdmaDevicesPerGPU: 8
ParameterDescription
rdmaDevicesPerGPUMellanox RDMA NICs per GPU. Must match your bare-metal shape.

Cluster prerequisites: