System Hardware & Components#
This section describes the hardware elements of the Enterprise RA.
The NVIDIA GB300 NVL72 (Figure 1) delivers breakthrough AI performance and can deliver up to a 50x overall increase in AI factory output performance compared to NVIDIA Hopper-based platforms. The NVIDIA Blackwell Ultra GPU architecture provides the latest technologies that bring months of computational effort down to days and hours on some of the largest AI/ML workloads.
Figure 1 NVIDIA GB300 NVL72#
To accommodate the massive compute power within a limited space, the GB300 NVL72 rack is liquid cooled, based on the MGX architecture. Some key highlights of the GB300 NVL72 compute rack include:
9 NVSwitch trays for full non-blocking P2P connectivity of all 72 Blackwell Ultra GPUs, with a total aggregated bandwidth of 130 TB/s
2 SN2201 OOB switches for integrated management access of rack components. The GB300 NVL72 in-rack switches use DC power since they are connected to the DC busbar; all other SN2201 switches use AC power
Integrated tray-level and rack-level liquid leakage detection
8 power shelves of 33 kW, with each shelf having six 5.5 kW PSUs
Full rack requiring up to 142 kW
NVIDIA GB300 NVL Compute Tray#
The compute building block for the NVIDIA GB300 NVL72 is the GB300 NVL compute tray/node (Figure 2), with 4 Blackwell Ultra GPUs and 2 Grace CPUs per tray. A logical diagram is shown in Figure 3.
Figure 2 NVIDIA GB300 NVL Compute Tray#
The compute tray is a fully integrated NVLink scale-out architecture, so that each GPU in the entire rack is directly connected to every other GPU via NVLink. For compute network connectivity, each compute tray includes 2 Mezzanine Network Boards with 2 ConnectX-8 silicon chips in each, for a total of 4 ConnectX-8 Host Channel Adapters (HCA). Each tray also includes 1 M.2 NVMe device for Operating System (OS) storage and 4 E1.S NVMe storage devices typically used as a fast, local cache.
Figure 3 Sample logical design for the compute tray#
To ensure balanced compute performance with the GPUs, each CPU is paired with a NVIDIA ConnectX-8 Mezzanine Board. Additionally, to support advanced storage acceleration, each tray includes a NVIDIA Bluefield-3 DPU.
Table 1: Compute Tray Components
Component |
Quantity |
|---|---|
NVIDIA Grace Processor, with a total of 72 ARM Neoverse V2 cores, connected via NVLink C2C, and 1TB aggregated LPDDR5 CPU main memory |
2 |
NVIDIA B300 GPU, with 1,152 GB aggregated HBM3 memory |
4 |
NVIDIA ConnectX-8 Mezzanine Boards, with 2 ConnectX-8 network adapters each |
2 |
Dual-port QSFP112 NVIDIA BlueField-3 B3240 DPU |
1 |
Control Plane/Management Nodes#
The customer should have guidance from the OEM regarding an IaaS-managed control plane nodes. The High Availability (HA) features of the control plane nodes should be enabled to ensure reliability for the enterprise customer. In this Enterprise RA, we assume a control plane composed of 12 management nodes, providing sufficient capacity and HA for large enterprise deployments. Table 2 outlines the specifications of the x86 Management Nodes while Table 3 outlines the specifications of the ARM Management Nodes.
Table 2: Management Node Components (x86 control node)
Component |
Configuration |
|---|---|
CPUs |
Intel based x86 2x Socket, 32c |
System Memory |
512 GB (8x DDR5 64GB) |
Boot Drive |
OS 2x960 GB M.2 or SATA/SAS SSD/ SW RAID1 |
Data Cache Drive |
2x 7.68 TB NVMe SSD |
Storage/in-band Network Adapter |
4x ConnectX-7 200G single-port |
Platform Security |
TPM 2.0, SHA-256 and support for secure boot |
Baseboard Management Controller (BMC) |
Host BMC with discrete connection, supports Redfish 1.4 or above, Serial-over-LAN via SSH, IPv6 support, support discovery of BMC IP in-band via Redfish |
Management Network |
2 x 1 Gb/s In-band port |
Host UEFI |
UEFI 2.6 compliant BIOS implementation for HTTP(s) network boot |
Table 3: Management Node Components (ARM control node)
Component |
Configuration |
|---|---|
CPUs |
2x NVIDIA Grace with 72 cores |
System Memory |
480 GB (LPDDR5X on-module) |
Boot Drive |
OS 2x960 GB M.2 or SATA/SAS SSD/ SW RAID1 |
Data Cache Drive |
2x 7.68 TB NVMe SSD |
Storage/in-band Network Adapter |
4x ConnectX-7 200G single-port |
Platform Security |
TPM 2.0, SHA-256 and support for secure boot |
Baseboard Management Controller (BMC) |
Host BMC with discrete connection, supports Redfish 1.4 or above, Serial-over-LAN via SSH, IPv6 support, support discovery of BMC IP in-band via Redfish |
Management Network |
2 x 1 Gb/s In-band port |
Host UEFI |
UEFI 2.6 compliant BIOS implementation for HTTP(s) network boot |
NVIDIA NVLink#
The NVIDIA Blackwell architecture introduces fifth-generation NVLink, delivering up to 1800 GB/s per GPU – doubling the bandwidth of the previous generation. A new NVSwitch chip and NVLink switch have also been introduced to enable even larger NVLink domain sizes. Together, they enable high bandwidth communication between Blackwell GPUs and unlock significant benefits for real-time, cost-effective large model inferencing.
Since the GB300 NVL72 integrates 36 NVIDIA Grace CPUs with 72 NVIDIA Blackwell Ultra GPUs in a rack-scale design, the fifth-generation NVLink enables all 72 GPUs to function as a single multi-GPU unit of compute enabling 30x faster real-time trillion-parameter inference compared to prior generations.
NVIDIA NVLink Switch Tray#
Each GB300 NVL72 rack includes 9 NVLink Fifth-Generation switch trays, with 2 NVSwitch ASICs per tray. Each GPU has 18 NVLink Fifth-Generation links, one per in-rack NVSwitch via the copper backplane. This way a single rack may form a fully connected L1 Domain with 72 GPUs. The NVLink Switch is a managed switch running the NVOS networking operating system.
Figure 4 NVLink Switch#
NVIDIA GB300 NVL72 Networking#
The NVIDIA GB300 NVL72 platform networking configuration enables tremendous Al performance and scale. It leverages the NVIDIA expertise in Al cloud data centers and optimizes network traffic flow:
East/West (Compute Network) traffic: This refers to traffic between the compute nodes/trays within the GB300 NVL72 cluster, typically for multi-node Al training, HPC collective operations, and other workloads.
North/South (Customer and Storage Network) traffic: This involves traffic between the GB300 NVL72 and any external resources including cloud management and orchestration systems, remote data storage nodes, and other parts of the data center or the Internet.
The GB300 NVL72 comes pre-integrated with a comprehensive networking stack design to optimize East/West and North/South traffic. Each compute tray is equipped with ConnectX-8 network interface cards (NICs) delivering 800 Gb/s and BlueField-3 data processing units, delivering up to 400 Gb/s. This configuration ensures high-bandwidth, low-latency communication across the 18 compute trays and during data ingress and egress from the system. This architecture simplifies deployment and ensures consistent performance across the system.
Compute (Node East/West) Ethernet Networking#
Each tray within the GB300 NVL72 rack has four dual-port ConnectX-8 SuperNICs for GPU to GPU (East/West) traffic. Each ConnectX-8 SuperNIC offers up to 800 Gb/s, low-latency network connectivity between GPUs in the AI cluster. ConnectX-8 SuperNIC offers up to 800 Gb/s, low-latency network connectivity between GPUs in the Al cluster, featuring RDMA and RoCE acceleration, with NVIDIA® GPUDirect® and GPUDirect Storage technologies. For data centers that deploy Ethernet, ConnectX-8 offers a range of advancements including RoCE optimizations and multi-tenancy, which are key to in Al cloud environments. Additionally, ConnectX-8 offers advanced hardware offloads for overlay networks such as VXLAN and NVGRE, which help reduce CPU load and improve network efficiency.
Beyond raw speed, ConnectX-8 enhances network performance with features like advanced congestion control, telemetry-based routing, and quality of service (QoS) capabilities to meet diverse AI application and training requirements. Security is built in with hardware acceleration for cryptographic protocols like IPSec and MACSec, taking these tasks over from the CPU to maintain performance.
Multi-node deployments with an NVIDIA B300 platform should adhere to the following total compute network bandwidth per GPU recommendations:
The ConnectX-8 SuperNIC is integrated onto the baseboard, maintaining a 1:1 GPU-to-NIC ratio to ensure optimal bandwidth and direct connectivity for each GPU.
Total Minimum Compute Network Bandwidth
400 GB/s (8x 400 Gb/s Ethernet NICs)
Total Recommended Compute Network Bandwidth
800 GB/s (16x 400 Gb/s Ethernet NICs using breakout)
Converged (Node North/South) Ethernet Networking#
This section describes the BlueField-3 role for the North/South (North/South) Ethernet network in the GB300 NVL72. Each tray within the GB300 NVL72 rack has one dual-port NVIDIA BlueField-3 B3240 DPUs for North/South traffic. The NVIDIA BlueField-3 B3240 is a PCIe based data processing unit (DPU) that can handle an aggregate bandwidth of approximately 480 Gb/s. The BlueField-3 DPU is optimized for the North/South network such as storage, complementing the ConnectX-8 SuperNIC which is optimized for the East/West, Al compute fabric.
The BlueField-3 B3240 DPU offers several essential capabilities and benefits within the NVIDIA GB300 NVL72 platform:
Out of Band Provisioning: BlueField-3 serves as an optimized compute platform for the data center control-plane, enabling automated provisioning and elasticity. This empowers GB300 NVL72 to scale resources dynamically based on fluctuating demand, ensuring efficient allocation of computing resources for transient Al workloads.
Storage Acceleration: BlueField-3 provides advanced storage acceleration features that optimize data storage access. Its innovative BlueField SNAP technology enables remote storage devices to function as local, improving Al performance and streamlining cloud operations.
Secure Infrastructure: BlueField-3 operates in a highly secure zero-trust mode and functions independently from the host, significantly enhancing the security of the GB300 NVL72 platform. In addition, BlueField-3 enables a wide range of accelerated security services, including next-generation firewall and micro-segmentation, bolstering the overall security posture of the infrastructure.
BlueField-3 B3240 DPU’s integration in the NVIDIA GB300 NVL72 platform delivers crucial capabilities that enhance resource management, performance, security, and scalability, making the NVIDIA GB300 NVL72 platform an ideal choice for Al workloads in datacenter environments.
The BlueField-3 DPU supports three modes of operation, with the Embedded Function (ECPF) or DPU mode used in this Enterprise RA. In this mode, the NIC resources and functionality are owned and controlled by the embedded Arm subsystem. All network communication to the host flows through a virtual switch control plane hosted on the DPU Arm cores, and only then proceeds to the host.
To securely control and manage the NVIDIA GB300 NVL72 platform through the BlueField DPU, NVIDIA has integrated an onboard Baseboard Management Controller (BMC) into its BlueField products. The onboard BlueField BMC enables the provisioning and management of the BlueField and NVIDIA GB300 NVL72 using standard tools, including Redfish APIs. It includes an external root-of-trust to ensure that the BMC firmware is secured. The main interface of the BlueField BMC is a 1Gb/s out-of-band management port that connects to the data center and management network.