> This page is for version Latest (default).
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt
> - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt
> - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Multinode Networking

Multi-node GPU training requires high-bandwidth, low-latency east-west networking between nodes. Without it, NCCL falls back to standard Ethernet and training times increase significantly. The Helm chart installs [Kyverno](https://kyverno.io/) policies that automatically inject the correct networking resources for four cloud providers.

## Prerequisites

* [Kyverno](https://kyverno.io/docs/installation/) installed in the cluster
* [Volcano](https://volcano.sh/en/docs/) scheduler installed (multi-node jobs use Volcano)
* Nodes with high-performance networking hardware (EFA, InfiniBand, etc.)
* The cloud provider's device plugin or network operator deployed so that networking resources are visible to Kubernetes

> **Note**
>
> NeMo Helix does not provision or manage the underlying cloud networking infrastructure. Each cloud provider section below lists the cluster-level prerequisites your administrator must configure. Refer to your cloud provider's documentation for setup instructions.

## How It Works

When a job requests more than one node, the jobs controller annotates the pod with `nhx.nvidia.com/enable-multi-node-networking: "true"`. Kyverno watches for this annotation and mutates the pod spec to inject the cloud-specific device requests, environment variables, and volume mounts that NCCL needs for high-performance inter-node communication.

## Configuration

Enable **exactly one** cloud provider in the `multinodeNetworking` section of your `values.yaml`. Enabling more than one causes a Helm install error.

> **Note**
>
> Device-count parameters (`efaDevicesPerGPU`, `rdmaDevicesPerGPU`) must match the hardware ratio of your instance type. A mismatch causes jobs to either fail scheduling or silently run without high-speed networking.

### AWS (EFA)

Injects [Elastic Fabric Adapter](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html) device requests (`vpc.amazonaws.com/efa`), mounts the EFA OFI library, and configures shared memory.

```yaml
multinodeNetworking:
 aws:
 enabled: true
 efaDevicesPerGPU: 4 # must match your instance type
```

| Parameter          | Description                                                                                                                                                                                  |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `efaDevicesPerGPU` | EFA interfaces to request per GPU. The Kyverno policy multiplies this value by the number of GPUs requested by each container to produce the total `vpc.amazonaws.com/efa` resource request. |

To determine the correct value, divide the number of EFA interfaces on your instance type by the number of GPUs. For example, `p5.48xlarge` has 32 EFA interfaces and 8 GPUs: `32 / 8 = 4`. Consult the [AWS EFA instance types documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html) for your instance type's EFA count.

**Cluster prerequisites:**

* EFA-enabled instance types (e.g., `p5.48xlarge`, `p5e.48xlarge`)
* [EFA device plugin](https://github.com/aws/eks-charts/tree/master/stable/aws-efa-k8s-device-plugin) deployed
* Security group permitting all traffic between nodes in the placement group
* See [AWS EFA documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html)

### Azure (InfiniBand / RDMA)

Injects RDMA device requests, `IPC_LOCK` capability, NCCL topology, and UCX environment variables.

```yaml
multinodeNetworking:
 azure:
 enabled: true
 rdmaDevicesPerGPU: 1
 rdmaDeviceName: "hca_shared_devices_a"
```

| Parameter           | Description                                                                     |
| ------------------- | ------------------------------------------------------------------------------- |
| `rdmaDevicesPerGPU` | RDMA devices per GPU. Must match your VM size.                                  |
| `rdmaDeviceName`    | Resource name exposed by the RDMA device plugin (e.g., `hca_shared_devices_a`). |

**Cluster prerequisites:**

* InfiniBand-capable VM sizes (e.g., `Standard_ND96asr_v4`, `Standard_ND96amsr_A100_v4`)
* RDMA device plugin or Network Operator deployed
* See [Azure InfiniBand documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nd-family)

### GCP (TCP-XO)

Injects a GPUDirect-TCPXO sidecar daemon, multi-NIC annotations, and NCCL Fastrak environment variables.

```yaml
multinodeNetworking:
 gcp:
 enabled: true
```

**Cluster prerequisites:**

* A3+ GPU node pools with multi-NIC networking enabled
* GKE multi-network support configured with the required `gpu-nic` networks
* See the [GKE GPUDirect-family documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/gpu-bandwidth-gpudirect-tcpx), specifically the TCPXO sections

### OCI (SR-IOV / RDMA)

Injects Mellanox NIC device requests (`nvidia.com/mlnxnics`), SR-IOV network annotations via Multus, and InfiniBand-tuned NCCL environment variables.

```yaml
multinodeNetworking:
 oci:
 enabled: true
 rdmaDevicesPerGPU: 8
```

| Parameter           | Description                                                   |
| ------------------- | ------------------------------------------------------------- |
| `rdmaDevicesPerGPU` | Mellanox RDMA NICs per GPU. Must match your bare-metal shape. |

**Cluster prerequisites:**

* RDMA-capable bare-metal shapes (e.g., `BM.GPU.A100-v2.8`, `BM.GPU.H100.8`)
* Network Operator and SR-IOV device plugin deployed
* See [OCI cluster networking documentation](https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/managingclusternetworks.htm)