Node Configurations#

This appendix provides information on the compute node configurations.

Table 9: HGX H100, H200 or B200 8-GPU system components.

Component

Quantity

Specification

GPUs

8 on baseboard

Eight H100, H200 or B200 SXM GPUs on a H100, H200 or B200 baseboard
• H100 variant has up to 640 GB of GPU memory
• H200 variant has up to 1128 GB of GPU memory
• B200 variant has up to 1440 GB of GPU memory

DPU

1

NVIDIA BlueField-3 B3220 DPU with 2x200G ports and 1 Gb RJ45 management port

SuperNIC

8

NVIDIA BlueField-3 B3140H SuperNIC with a 400G port and 1 Gb RJ45 management port

CPUs

2

Minimum of 32 physical CPU cores per socket
Recommendation of 56 physical CPU cores per socket

System Memory

1

Minimum of 1.5TB system memory

Storage

3

Inference Servers: Minimum 1 TB NVMe drive per CPU socket
Training / DL Servers: Minimum 2 TB NVMe drive per CPU socket
HPC Servers: Minimum 1 TB NVMe drive per CPU socket
1×1 TB NVMe boot drive

BMC/iLO

1

1 GbE RJ45 management port

NVIDIA-Certified systems can be found here.

Switches and Cables#

Table 10 and Table 11 provides an overview of the switches and cables used in the cluster. The network adapters of the HGX H100, H200 or B200 8-GPU servers, the storage servers, and the control plane servers are included for better understanding of the connectivity requirements.

Table 10: Switching with required inter-switch links, transceivers, and cables.

Component

32 Server Nodes

64 Server Nodes

128 Server Nodes

HGX H100/H200 SXM GPUs (eight per node)

256

512

1024

BlueField-3 B3140H SuperNIC, Compute fabric (eight per node)

256

512

1024

BlueField-3 B3220 DPU, In-band management, Customer & Storage fabric and Support Servers

32

64

128

SN5600 Ethernet switch, compute core fabric

6

12

24

SN5600 Ethernet switch, converged core fabric

2

2

10

SN2201 leaf switches for OOB management fabric.

4

8

16

OSFP, 2x400G transceiver used for E/W and N/S inter-switch links (ISL)

274

544

1408

Cable used for E/W and N/S inter-switch links (ISL)

274

544

1408

QSFP 100G transceivers used for OOB leaf switches

8

16

32

OSFP, 2x400G transceiver used for OOB leaf switches

2

2

4

Cable used for OOB leaf switches

4

4

8

Table 11: End-point connections with required transceivers, and cables.

Component

32 Server Nodes

64 Server Nodes

128 Server Nodes

HGX H100/H200 SXM GPUs (eight per node)

256

512

1024

BlueField-3 B3140H SuperNIC, Compute fabric (eight per node)

256

512

1024

BlueField-3 B3220 DPU, In-band management and Storage fabric

32

64

128

OSFP, 2x400G transceiver used for Switch to DPU & SuperNIC connections

144

288

576

QSFP, 1x400G used for SuperNIC and DPU to switch

320

640

1280

Cable for Switch to B3140H

256

512

1024

Cable for Switch to B3220 DPU

32

64

128

OSFP, 2x400G transceiver used for switch to storage

4

8

16

QSFP 100G transceiver used for upstream storage

32

64

128

Cable for storage network

8

16

32

OSFP, 2x400G transceiver used for switch to customer network

8

16

32

QSFP 100G transceiver used for customer network

64

128

256

Cable for customer network

16

32

64

OSFP, 2x400G transceiver used for switch to control-plane B3220 DPUs

4

4

4

QSFP, 400G transceiver used for control plane B3220 DPUs

16

16

16

Cable for control-plane

8

8

8

CAT6 RJ45 cable for 1G OOB fabric

192

384

768

For exact BOM and part numbers, please work with your OEM/ODM partners and our NVIDIA support team depending on your specific rack and data center conditions.

On this page