Networking#
This section covers the DGX system network ports and an overview of the networks used by DGX BasePOD.
DGX H200/H100 System Network Ports#
Figure 1 shows the physical layout of the back of the DGX H200/H100 system.
Figure 2 shows how the DGX H200/H100 network ports are used in this deployment guide.
The following ports are selected for DGX BasePOD networking:
Eight ports in four OSFP connections are used for the InfiniBandcompute fabric (marked in green).
Two ports of the dual-port ConnectX-7 cards are configured as abonded Ethernet interface for in-band management and storage ethernet networks. These are the left ports (port 2) from slot 1 and slot 2 (marked in blue).
Optional: Two ports of the dual-port ConnectX-7 cards when InfinIBand storage is used instead of Ethernet. These are the right ports (port 1) from slot 1 and slot 2 (marked in red).
BMC network access is provided through the out-of-band network (marked in purple).
The networking ports and their mapping are described in detail in the Network Ports section of the NVIDIA DGX H200/H100 System User Guide.
DGX BasePOD Network Overview#
There are four networks in a DGX BasePOD configuration:
managementnet (internalnet)—Network used exclusively within the cluster, for storage and in-band management.
externalnet—Network connecting the DGX BasePOD to an external network, such as a corporate or campus network.
oobmanagementnet (ipminet)—Network for out of band management, connecting BMCs.
computenet (ibnet)—InfiniBand network connecting all DGX systems’ ConnectX-7 Compute Fabric HCAs.
These are shown in Figure 3.
Figure 3. Network design and topology diagram.
Managementnet and External networks#
managementnet (internalnet) and externalnet are configured on the SN4600C switches, which are the backbone of the DGX BasePOD Ethernet networking. Each DGX system connects to the SN4600C switches with a bonded interface that consists of two physical interfaces; slot 1 port 2 (storage 1-2) and slot 2 port 2 (storage 2-2) as described in the Network Ports section of the NVIDIA DGX H200/H100 System User Guide.
The K8s (Workload Manager nodes in the topology diagram) nodes and the NFS storage device have a similar bonded interface configuration connected to SN4600C switches. Two SN4600C switches with Multi-chassis Link Aggregation (MLAG) provides the redundancy for DGX systems, K8s nodes, and other devices with bonded interfaces.In this deployment guide, BGP (Border Gateway Protocol) is used for network connectivity between the managementnet (internalnet) and externalnet networks.
Table 2. BGP protocols
Protocol |
Description |
|---|---|
eBGP |
Used as required for routing between switches and customer network |
iBGP |
Configured between the two SN4600C switches using the MLAG peerlink.4094 interface |
oobmanagementnet (ipminet)#
On the oobmanagementnet(ipminet) SN2201 switches, all the switch ports connected to the end hosts are configured as access ports. Each BCM headnode has its BMC interface connected to the IPMI switch. Uplinks are connected to SN4600C switches.
externalnet uplink#
All connected subnets are redistributed into BGP. IPMI switches can also be uplinked to a separate management network if required, rather than the SN4600C switches; still IPMI subnet route must be advertised to the in-band network so that BCM can control hosts using the IPMI network.
computenet (ibnet)#
For the computenet (ibnet), 4 ports of the DGX OSFP ports are connected to QM9700-1 InfiniBand switch, and 4 ports are connected to QM9700-2 InfiniBand switch. To manage the InfiniBand fabric, at least one subnet manager is required to be enabled on the QM9700 switches.
The networking ports and their mapping are described in the Network Ports section of the NVIDIA DGX H200/H100 System User Guide.