Benchmarking And Measurement#

Performance measurements on Vera-Rubin NVL72 should be reproducible at the tray, partial-rack, and full-rack levels. Keep topology, software, power, and cooling metadata with every result.

Minimum Run Record#

Capture the following with each benchmark:

  • Platform image, kernel, driver, CUDA, NCCL, UCX/HPC-X, firmware, and fabric manager versions.

  • CPU topology, SMT mode, memory capacity, and host page size from lscpu, numactl -H, and getconf PAGE_SIZE.

  • GPU inventory and topology from nvidia-smi -L and nvidia-smi topo -m.

  • Network inventory from ibdev2netdev, rdma link, and ucx_info -d.

  • Storage inventory for the 4 Gen6 E1.S data SSDs and 1 Gen5 E1.S boot SSD when data-cache or checkpoint performance is part of the result.

  • NCCL topology dump for baseline collective tests.

  • Power profile, CPU power limits, GPU power limits, and throttling state.

  • Liquid-cooling state when available.

  • Exact launch command, rank count, rank placement, CPU binding, and GPU binding.

CPU Measurements#

Use the CPU settings in Understanding Your Vera-Rubin NVL72 System and the platform-qualified OS guidance. Validate both CPU-only and CPU-assisted GPU workloads:

  • STREAM or site-approved memory bandwidth tests for SOCAMM memory behavior.

  • Pointer-chasing latency tests for cache and memory hierarchy sensitivity.

  • CPU preprocessing throughput with the same page size and AutoNUMA state used in production.

  • PCIe and storage paths when data cache SSDs are part of the workload.

Compile CPU benchmarks with Vera-aware compiler flags when available. Avoid using previous-generation CPU masks or core counts.

NCCL Tests#

Use nccl-tests for collective and point-to-point communication. Multi-process and multi-node measurements require a build with MPI=1 and an MPI-compatible launcher. For one rank per GPU, launch one process per allocated GPU with -g 1; a standalone invocation with that setting tests only one GPU. Verify the total rank count, per-rank GPU visibility, CPU placement, and selected transport before comparing results. Follow the public nccl-tests build and usage instructions for the installed version’s tests and message-size options.

Compare one tray, partial rack, and full rack with the same launch policy. Keep NCCL_DEBUG=INFO and NCCL_TOPO_DUMP_FILE outputs for baseline runs.

NVSHMEM#

NVSHMEM benchmarks measure one-sided GPU memory operations, synchronization, and collectives. Use them to isolate stream-ordered and device-initiated paths, especially when applications use communication from GPU kernels.

Begin with the release-qualified library and site configuration, and change only one tuning control at a time. Keep communication buffers at least 16-byte aligned, and prefer block-scoped APIs when a GPU thread block can cooperate on the operation. Block-scoped forms can improve performance portability across NVLink and network transports.

Map processing elements with the discovered Vera-Rubin topology. Bind each PE to its intended GPU, nearby CPU resources, and CX9 path, then validate on-rack NVLink traffic separately from off-rack network traffic. Record symmetric-heap placement, transport selection, message size, synchronization pattern, and the software and firmware stack with each result.

Use NVSHMEM_INFO and NVSHMEM_DEBUG to confirm initialization and transport selection, and use NVSHMEM_NVTX when correlating communication with an Nsight Systems trace. Treat transport or hardware-acceleration toggles as diagnostic experiments rather than permanent defaults. See the NVSHMEM Performance Best Practice Guide for current alignment, API, topology, profiling, and environment guidance.

Network Tests#

Run network tests at both per-NIC and per-GPU levels:

  • Per-NIC fabric validation to detect link, MTU, ECN/PFC, or error-counter issues.

  • UCX tests for host memory and CUDA memory paths.

  • NCCL tests for GPU collective behavior.

  • Application-level tests for production launch behavior.

Do not declare a rack healthy from only one benchmark type. CX9 rail mapping, BF4 north-south policy, NVLink health, CPU placement, and power profile can each dominate different tests.

Profiling#

Use Nsight Systems to understand CPU/GPU overlap, launch stalls, and communication/computation overlap. For an Open MPI application, run this command inside each SLURM task with the launch template’s resource and GPU binding. Keep the launcher outside the profiler:

nsys profile \
  --trace=cuda,nvtx,osrt,mpi --mpi-impl=openmpi \
  --sample=process-tree --cpuctxsw=process-tree \
  -o 'run_%h_rank%q{SLURM_PROCID}_%p' ./app

The output name includes the hostname, SLURM rank, and target process ID so ranks do not overwrite one another’s reports. Use the MPI implementation appropriate to the application, and confirm profiling permissions with the installed tool. See the public Nsight Systems CLI documentation.

For multi-node jobs, capture a small representative slice first. Full-rack profiling can generate very large traces and should be scoped to a short interval or a subset of ranks.