Major Components#
Major components for the DGX SuperPOD configuration are listed in Table 7. These are representative of the configuration and must be finalized based on actual design.
Count |
Component |
Recommended Model |
|---|---|---|
DGX Compute and Racks |
||
576 |
DGX B300 |
DGXB300-AC |
144 |
19” Standard AC, AIE for DGX Nodes |
Legrand NVIDPD13 or equivalent |
48 |
19” Standard AC, AIE for Switches and Management |
Legrand NVIDPD13 or equivalent |
432 |
DGX rPDU for Standard Rack |
Raritan (PX3-5091R-P1Q2R1A5) or Equiv |
96 |
Mgmt Vertical PDU |
Raritan (PX3-5747V-V2) or equivalent |
7 |
AC Powered x86 Control Nodes for Mission Control |
Dell validated control plane nodes |
Compute Fabric |
||
64 |
IB XDR (Q3400-RA) - Compute Leafs - AC |
920-9B36F-00RX-8S0 |
36 |
IB XDR (Q3400-RA) - Compute Spines - AC |
920-9B36F-00RX-8S0 |
4608 |
Transceivers on DGX |
980-9IATO-00XM00 |
2034 |
Twin port OSFP Transceivers on Leaf <-> DGX |
980-9IAH1-00XM00 |
4608 |
Twin port OSFP Transceivers on Leaf <-> Spine |
980-9IAH1-00XM00 |
4608 |
DGX-Leaf Cable |
980-9I570-00N030 |
4608 |
Leaf-Spine Cable |
980-9I570-00N030 |
Ethernet Storage Fabric |
||
16 |
SP4 (SN5610) - Storage Leafs DGX Side - AC |
920-9N42F-00RI-3C1 |
4 |
SP4 (SN5610) - Storage Leafs Storage Side - AC |
920-9N42F-00RI-3C1 |
12 |
SP4 (SN5610) - Storage Spines - AC |
920-9N42F-00RI-3C1 |
1152 |
400G Transceiver (for BF3240) |
980-9I693-F4NS00 |
8 |
400G Transceiver (for Control Plane Nodes) |
980-9I693-F4NS00 |
576 |
Twin port OSFP Transceivers on Leaf <-> Spine |
980-9I510-F4NS00 |
1152 |
DGX-Leaf Cable |
980-9I570-00N030 |
1120 |
Leaf-Spine Cable |
980-9I570-00N030 |
InfiniBand Storage Fabric |
||
16 |
NDR Switches, Storage Leaf DGX Side |
920-9B210-00FN-0M0 |
8 |
NDR Switches, Storage Leaf Storage Side |
920-9B210-00FN-0M0 |
8 |
NDR Switches, storage spines |
920-9B210-00FN-0M0 |
512 |
400G Transceiver (for BF3240) |
980-9I693-00NS00 |
8 |
400G Transceiver (for Control Plane Nodes and UFM) |
980-9I51S-00NS00 |
324 |
Twin port OSFP Transceivers on Leaf <-> DGX/storage/Mgmt |
980-9I510-00NS00 |
512 |
Twin port OSFP Transceivers on Leaf <-> Spine |
980-9I510-00NS00 |
2 |
UFM Appliance 3.5 |
920-9B020-00RA-0D0 |
2 |
UFM <-> Inband Transceiver |
980-9I693-F4NS00 |
648 |
Leaf Cable |
980-9I570-00N030 |
512 |
Leaf-Spine Cable |
980-9I570-00N030 |
Inband Fabric |
||
6 |
SP4 (SN5610) - Inband Leafs |
920-9N42F-00RI-3C1 |
2 |
SP4 (SN5610) - Inband Spines |
920-9N42F-00RI-3C1 |
1152 |
400G Transceiver (for BF3240) |
980-9I693-F4NS00 |
24 |
400G Transceiver (for Control Plane Nodes) |
980-9I51S-F4NS00 |
294 |
Twin port OSFP Transceivers on Leaf <-> DGX, Management |
980-9I510-F4NS00 |
48 |
Twin port OSFP Transceivers on Leaf <-> Spine |
980-9I510-F4NS00 |
589 |
DGX, Mgmt - Leaf Cable |
980-9I557-00N030 |
48 |
Leaf-Spine Cable |
980-9I570-00N030 |
10 |
800G Transceivers (between leaf -> NFS & Uplink) |
980-9I30H-F4NM00 |
20 |
DR4 -> 4x DR1 Fabric Splitter Leaf->NFS, SN2201, Uplink |
NV08M31P2X1BHM020 |
4 |
100G DR1 (Uplink, NFS -> SP4) |
Customer provided |
32 |
DR1 Storage |
980-9I042-00C000 |
Out of Band Management |
||
69 |
SN2201 |
920-9N110-00R1-0C0 |
3290 |
Total RJ45 Count |
Cat6 |
138 |
100G DR1 Transceiver (SN2201 @OOB ) |
980-9I042-00C000 |
36 |
800G optics (SN5610 Inband Spines -> SN2201) |
980-9I30H-F4NM00 |
For the reference design, we assumed the following storage and management requirement:
7 Control Plane Nodes with 4 OSFP (CX7/CX8) connectors each
32x 100GbE ports for Home / NFS Storage
256x 400GbE/NDR ports for High Performance Storage
258x 1 GbE OOB ports for High Performance Storage and Home Storage
2x 1GbE Inband ports for High Performance Storage