Overview#

The NVIDIA Multi-Node NVLink (MNNVL) system is a rack-scale solution combining NVIDIA ARM-based Grace CPUs, NVIDIA Blackwell GPUs and NVIDIA NVSwitches connected through a passive backplane assembly.

NVIDIA MNNVL systems enables higher GPU power and performance, improved serviceability and larger NVLink domains.

The MNNVL rack contains:

  • 18x compute trays, each with 2 Grace CPUs and 4 Blackwell GPUs

  • 9x NVLink switch trays providing 57.6 terabits per second (Tbps) of full duplex bandwidth over fifth generation NVLink connections.

Illustration of an NVL72 rack.

Figure 1 Illustration of an NVL72 rack.#

Software Overview#

A MNNVL system consists of software and firmware components across both the compute nodes and NVLink switches.

CUDA Software Platform#

NVIDIA CUDA® enables applications to use GPUs for accelerated computing and provides more than 150 libraries to create high-performance applications across programming languages and use cases. CUDA provides the tools to optimize and debug GPU accelerated application.

CUDA-enabled applications automatically leverage the NVLink network to provide distributed processing across every GPU in the MNNVL system.

Refer to the NVIDIA CUDA documentation for more information about CUDA.

Internode Memory Exchange Service#

NVIDIA Internode Memory Exchange (IMEX) is a secure service facilitating the mapping of GPU memory over NVLink between the GPUs in an NVLink domain. The service starts on all nodes in the NVLink domain during system startup or during the launch of the CUDA job.

IMEX manages the lifecycle of application shared memory and registers for memory import events with the GPU driver. IMEX doesn’t rely on CUDA but communicates across compute nodes using TCP and gRPC.

Tip

IMEX provides inter-GPU memory mapping information over TCP/IP on the data or management Ethernet network. Inter-GPU memory read and writes are over the NVLink backplane.

IMEX calls the collection of nodes sharing memory an IMEX domain.

Tip

Single node systems or processes don’t require IMEX.

Compute nodes communicate using an IMEX channel, a logical /dev character device. CUDA applications with permissions to access to the IMEX channel can securely share memory across nodes.

Refer to IMEX Service for NVLink Networks for more information.

DOCA Software Framework#

The DOCA framework unlocks the potential of the NVIDIA® BlueField® networking platform by enabling the rapid creation of applications and services that offload, accelerate, and isolate datacenter workloads. Developers can create software-defined, cloud-native, DPU- and SuperNIC-accelerated services with zero-trust protection, which addresses the performance and security demands of modern data centers.

DOCA-Host includes the host drivers and tools for your BlueField and ConnectX® devices in the rack.

Refer to the NVIDIA DOCA Software Framework for more information.

NVIDIA Communication Libraries#

NVIDIA’s communication libraries like NVIDIA Collective Communications Library (NCCL), NVIDIA NVSHMEM, and Unified Communication X (UCX) provide a framework for efficient and scalable GPU-to-GPU communication. These communication libraries support multiple interconnect technologies including PCIe, NVLink, InfiniBand, RoCE and more.

NCCL, provides GPU-to-GPU communication primitives that are topology-aware, removing the need to optimize applications for specific systems. NCCL implements collective communication and point-to-point send and receive primitives for intra- and inter-node data transfers. Refer to the NCCL documentation for more information.

NVSHMEM provides a partitioned global address space (PGAS) for data that spans the memory of multiple GPUs. NVSHMEM provides device APIs enabling low latency communication between GPUs and enables for fine grained or fused overlap of compute and communications from the CUDA kernel. Refer to the NVIDIA NVSHMEM documentation for more information.

UCX is a framework that provides a common set of CPU-initiated tag matching, active messaging, and remote memory access APIs over a variety of network protocols. Refer to NVIDIA UCX documentation for more information.

Data Center GPU Manager#

NVIDIA Data Center GPU Manager (DCGM) is a suite of tools that manage and monitor NVIDIA datacenter GPUs and NVSwitches in MNNVL systems. DCGM includes active health monitoring, comprehensive diagnostics, system alerts, and governance policies. DGCM can be a standalone monitoring platform or integrate into existing tools from NVIDIA partners. Refer to the NVIDIA DCGM documentation for more information.

NVIDIA Switch OS#

The NVLink switch tray comes with the NVIDIA NVSwitch™ operating system (NVOS) enabling the management and configuration of NVIDIA’s switch system platforms. NVOS provides a suite of management options, incorporating an industry-standard command line interface (CLI) and OpenAPI (Swagger) interface, enabling system configuration and management.

NVOS includes the Subnet Manager (SM), Fabric Manager (FM), NMX-Controller (NMX-C), NMX-Telemetry (NMX-T) and the NVSwitch firmware.

NMX-Controller#

NMX-Controller (NMX-C) is a cluster application for fabric SDN services.

NMX-C manages SDN services configuration and operation through a gRPC interface with external manager or the NVOS CLI or REST APIs.

Refer to the NMX-C documentation for more information.

Tip

The NVOS software image includes the NMX-C application. There’s no standalone software to install.

Fabric Manager#

NVIDIA Fabric Manager (FM) configures the NVSwitch memory fabric to form a single memory fabric among the participating GPUs. FM coordinates with the GPU driver to initialize the GPUs and configure NVLink port mapping and routing between compute nodes and NVLink switches. FM provides fabric performance and error monitoring.

Tip

The NVOS software image includes the FM application. There’s no standalone software to install.

NMX-Telemetry#

NMX-Telemetry (NMX-T) collects, aggregates, and transmits telemetry data from NVLink switches.

Refer to the NMX-T documentation for more information.