Networking Physical Topologies#

The RA configurations use three physical network fabrics:

  • Compute (Node East/West) Network

  • Converged (Node North/South) Network

  • Out-of-Band Management Network

Each network is discussed in this section.

Compute (Node East/West) Network#

The compute fabric (East/West) is built using switches with NVIDIA Spectrum technology in a full non-blocking fat tree topology to provide the highest level of performance for the application running over the HGX H100, H200 or B200 8-GPU cluster. The compute fabric is an RDMA-based fabric. It is designed to provide the shortest hops through the network for the application, in a leaf-spine manner, where the GPUs are connected using a rail-optimized network topology through their respective BlueField-3 B3140H SuperNICs. This design allows the most efficient communication for multi-GPU applications within and across the nodes.

The leaf and spine architecture allows for a scalable and reliable network that can fit varied sizes of clusters using the same architecture.

The compute fabric is designed to maximize bandwidth and minimize network latency required to connect GPUs within a server and within a rail.

The compute network is not necessarily required for inference workloads. Most common pre-trained models do not exceed the size of a single H100, H200 or B200 SXM GPU. Models beyond 13B parameters may require model parallelism and require more than one H100, H200 or B200 SXM GPU. This parallelism will still reside within the same node as each server node can hold up to 8x H100, H200 or B200 SXM GPUs.

For larger network design points due to the high number of end-points required the Compute Fabric utilizes a super-spine, spine and leaf networking architecture. This is principally to support the required design points but also to allow extensibility in the architecture.

Compute Fabric Excluded for Pure Inference Deployments#

For the use case of pure inference, a compute network may not be necessary. Each H100, H200 or B200 SXM GPU can support a maximum model size of approximately 13B parameters. Also, tensor parallelism yields less performant results per GPU than running the model on the same GPUs in data parallel mode.

Since there are no performance gains from running an inference model on multiple GPUs, either on a single node or across multiple nodes, a compute network is not necessary.

The drawback to implementing an infrastructure without a compute network is that the infrastructure cannot be used for any hybrid workflows including small model training. The infrastructure can be deployed without a compute network and can be retrofitted later.

Converged (Node North/South) Network#

A converged network for both storage and in-band management is used in the RA. This provides Enterprises with flexible storage allocation and easy network management. The converged network has the following attributes:

  • Provide high bandwidth to shared storage and connects the customer through the converged network.

  • It is independent of the compute fabric to maximize both storage and application performance.

  • Each compute and management node are connected with two 200 GbE ports to two separate switches to provide redundancy and high storage throughput that can reach up to 40 GB/s per node.

The fabric is built on Ethernet technology with RDMA over Converged Ethernet (RoCE) support and utilizes NVIDIA BlueField-3 B3220 DPUs in each compute node to deliver existing and emerging cloud and storage services.

  • It is flexible and can scale to meet specific capacity and bandwidth requirements.

  • Tenant-controlled management nodes provide tenant the flexibility to deploy OS and their job scheduler of choice.

  • Hybrid storage fabric design with support for tenant isolation provides access to both shared and dedicated storage per tenant.

  • Used for node provisioning, data movement, Internet access, and other services that must be accessible by the users.

Out-of-Band Management (Node) Network#

The OOB management network connects all the base management controller (BMC) ports, as well as other devices that should be physically isolated from system users to allow the infrastructure management. This includes the 1 GbE switch management ports and the BlueField-3 SuperNIC and DPU management ports.