Hardware#
DGX Vera Rubin rack scale systems are rack scale solutions for graphics processing units (GPUs) connected by NVLink through the NVLink passive copper cable cartridge backplane. The complete DGX Vera Rubin rack system comprises compute trays with two compute boards, NVLink switch trays, an NVLink passive copper cable backplane, power shelves, a bus bar, and liquid cooling manifolds.
Rack Configuration#
An NVL72 rack is an NVIDIA rack with a 72-GPU NVL domain. Each NVL72 rack contains:
18 compute trays, each 1 RU high, with 2 Vera CPUs and 4 Rubin GPUs.
9 NVLink 6 switch trays, each 1 RU high.
2 top-of-rack (TOR) switches for management.
Power shelves for supplying power to all trays and switches.
DGX Vera Rubin NVL72 Rack Configuration#
The rear side of the system provides access to the cable management system, the inlets and outlets to the liquid cooling manifolds and the manifolds themselves, the cable cartridges, and the power bus bar.
DGX Vera Rubin NVL72 Rack Rear View#
Compute Trays#
The compute tray in a DGX Vera Rubin rack system is an enclosure that holds the compute boards and peripheral accessory boards and runs an OS image. The compute trays are cooled by liquid that runs up and down the rack through the inner manifold, then through the cold plates that are attached to the CPUs, GPUs, and various networking and storage components in the tray. The compute trays in a rack are interconnected via NVLink through the connectors at the back of the tray, which enables communication with the other compute trays via the NVSwitch trays.
DGX Vera Rubin Compute Tray#
The following table provides information about the DGX Vera Rubin rack system compute tray configuration.
Component |
Function |
Configuration |
|---|---|---|
Compute |
Compute node |
2 Vera™ CPUs and 4 Rubin™ GPUs |
Networking |
Cluster network |
2 NVIDIA ConnectX-9 I/O cards for InfiniBand cluster interconnect and multi-rack clustering |
Networking |
Storage/management network |
1 NVIDIA BlueField-4 DPU for Ethernet storage and system management networking |
Networking |
Out-of-band management network |
Host BMC with RJ45 |
Storage |
Data cache |
4 E1.S NVMe cache drives per compute tray |
Storage |
Boot drive |
1 E1.S NVMe boot drive |
DGX Vera Rubin Compute Tray Block Diagram#
Out-of-band (OOB) management of the compute tray hardware resources is provided by a combination of Board Management Controller (BMC) and Host Management Controller (HMC) microcontrollers. External access is available through standard BMC and console interfaces.
The following figure shows a top view of the DGX Vera Rubin rack system compute tray and identifies the main components.
DGX Vera Rubin Compute Tray Top View#
The following figure shows a front view of the DGX Vera Rubin rack system compute tray and identifies the main components.
DGX Vera Rubin Compute Tray Front View#
NVLink Switch Trays#
Each Vera Rubin NVL72 rack has 9 NVLink 6 switch trays, providing NVLink connectivity to all GPUs in the single 72-GPU L1 domain.
Each switch tray has the following components:
4 NVSwitch ASICs, each with 72 NVLink 6 ports
BMC for attestation and recovery
CPU ERoT for security
DGX Vera Rubin NVLink Switch Tray#
Power Shelves#
The DGX Vera Rubin NVL72 rack system uses a liquid-cooled bus bar structure to distribute power in the rack. AC power whips energize the power shelves from a remote power panel, and the power shelves convert AC power into a 54 V DC output that is distributed through the bus bar to the rack components.
Each rack has 4 power shelves. Each shelf is 3 RU high and rated at 110 kW, with 6 air-cooled 18.3 kW PSUs. Two 100 A whips supply each shelf at 415–480 V AC.
DGX Vera Rubin Power Shelf#
Top-of-Rack Ethernet Switches#
Top-of-rack (TOR) switches are used to connect all BMCs in the compute trays (including BlueField-4 BMCs) and the switch trays to the management network. The following ports are connected to these switches:
Out-of-band management BMC from the compute trays
BlueField-4 BMCs from the compute trays
DGX Vera Rubin Top-of-Rack Ethernet Switch#
Console Access#
Each NVLink switch provides an RJ45 interface for serial access. The default serial baud rate is 115200 baud.
Leak Detection#
Leak detection in the DGX Vera Rubin NVL72 rack system’s liquid cooling system prevents damage to equipment, ensures system reliability and uptime, protects data integrity, and maintains safety. Early detection helps avoid costly repairs, downtime, and regulatory compliance issues, optimizes the cooling efficiency, and prevents secondary damage to infrastructure.