How-to Guides

Run NCCL Benchmarks

View as Markdown
1spec:
2  categories:
3    - domain: communication
4      variant: nccl-all-reduce
5    - domain: communication
6      variant: nccl-all-gather
7    - domain: communication
8      variant: nccl-alltoall

The controller measures bus bandwidth for each collective and compares against per-architecture thresholds.

Via WorkloadRun (ad hoc)

1apiVersion: cre.nvidia.com/v1alpha1
2kind: WorkloadRun
3metadata:
4  name: nccl-all-reduce
5spec:
6  image: nvcr.io/nvidia/pytorch:26.01-py3
7  framework:
8    mpi:
9      binary: /usr/local/bin/all_reduce_perf_mpi
10      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
11      mpirunPath: /usr/local/mpi/bin/mpirun
12  numNodes: 4
13  bandwidthMeasurement:
14    logProfileRef: nccl-bandwidth
15    testType: all_reduce

Interpreting results

The report shows measured bus bandwidth (GB/s) versus the expected threshold for the detected GPU architecture. Results below threshold indicate a network problem — degraded links, misconfigured EFA/RoCE, or a faulty NIC.

Threshold table by architecture coming soon.