Networking Planning and Design#

A NVIDIA team will have a network requirements gathering meeting with a SuperPOD customer directly to discuss network requirements and obtain the IP information. The tables below shows the information that is needed in the planning stages:

Logical Network Design#

Logical Network Design for GB200

Figure 1 Logical Network Design Architecture#

Short Summary:

  • BGP EVPN provides control plane signaling and tunnel discovery support in a VxLAN overlay network.

  • Loopback interfaces are commonly used for VTEPs (VXLAN Tunnel Endpoints) to establish BGP peering. Each VTEP is assigned a loopback IP address, which serves as a stable endpoint for routing EVPN control messages.

  • This approach enhances scalability and reliability, allowing for dynamic MAC address and IP address learning while maintaining tenant isolation. The VXLAN encapsulation enables seamless communication across the underlying network infrastructure, making it ideal for multi-tenant data centers and cloud environments.

Ethernet Networking (North-to-South)#

Reference: IP Subnet Range/CIDR Requirements per BCM Network

Network

IP Range Requirement

In Band Mgmt

  • internalnet: Control Plane Data

  • dgxnet[1-n]: DGX

Out of Band (OOB)

  • ipminet[1-n]: IPMI Covers GB200 (2x per GB200 rack) SN2201s, …. Control plane 2201

Loopback

/24 - to cover address from each switches

Client P2P IP Addressing (TOR to CE) (Border TOR to EDGE Network)

/31 - a minimum of 8 subnets

Non-Routable IP Range

ETH Storage Network

100.127.0.0/16

IB Compute Network

100.126.0.0/16

Vendor Storage Prefix

100.127.124.0/24

Subnet Breakout per POD#

POD | OOB Total

OOB Root

DATA Total

DATA Root

ROOT

1

2×/24 + 2×/23

/21

2×/23 + 2×/25

/21

/20

2

2×/24 + 4×/23

/20

4×/23 + 4×/25

/20

/19

3

1×/24 + 6×/23

/20

6×/23 + 6×/25

/20

/19

4

1×/24 + 8×/23

/19

8×/23 + 8×/25

/19

/18

5

1×/24 + 10×/23

/19

10×/23 + 10×/25

/19

/18

6

1×/24 + 12×/23

/19

12×/23 + 12×/25

/19

/17

7

1×/24 + 14×/23

/19

14×/23 + 14×/25

/18

/17

8

1×/24 + 16×/23

/18

16×/23 + 16×/25

/18

/17

BGP ASN 4-Byte Range#

ROLEs

4-bytes ASN

OOB

4294902361 - 4294902460

TOR/STOR

4294902461 - 4294902660

SPINEs

4294902661 - 4294902750

SuperSPINEs

4294902751 - 4294902760

Note

TOR to CE Transceiver Compatibility, see Hardware Compatibility List (HCL)

We recommend that you review and refer to SuperPOD Network Requirements Document

Ethernet Planning#

Confirm Accurate Hardware Models and Quantities#

  • Ethernet switches x amount of SN5600

  • Transceivers (optical) and cables (DAC / AOC)

  • Fibers

Confirm You Have the Correct Cumulus Linux Software Version#

You need to ensure that your are using the correct version of the Linux distribution called Cumulus Linux.

Obtain Customer Network Requirements#

  • Fill out the “NVDA DGX SPOD GB200 Network Requirements” Excel file - Assuming in SIB

    Content/Consideration List

    • Customer-Edge to Cluster connectivity

      • Transceiver type, compatibility and HW order status

      • Electrical signaling/encoding (NRZ vs PAM4)

      • Speed/Bandwidth

      • Uplink Quantity (transceiver, cable, fiber)

      • IP Addressing - P2P Uplinks

      • Routing protocol (default: BGP)

    • Routable IP Address Allocation

      • Discussed with and provided by the customer

    • Non-Routable IP Address Allocation

      • Default NVDA address ranges should be used

        • IB Compute = 100.126.0.0/16

        • Ethernet Underlay = 10.254.0.0/16

        • Ethernet Overlay Storage = 10.127.0.0/16

    • VLAN VNI Per TOR Pair

DGX Compute VLAN#

POD#

DGX Compute VLANs

Compute VNI

POD#1

101,102

101,102

POD#2

201,202

201,202

POD#3

301,302

301,302

POD#4

401,402

401,402

POD#5

501,502

501,502

POD#6

601,602

601,602

POD#7

701,702

701,702

POD#8

801,802

801,802

POD#9

901,902

901,902

POD#10

1001,1002

1001,1002

POD#11

1001,1002

1001,1002

POD#12

1201,1202

1201,1202

POD#13

1301,1302

1301,1302

POD#14

1401,1402

1401,1402

POD#15

1501,1502

1501,1502

POD#16

1601,1602

1601,1602

OOB VLAN / VNI#

POD#

OOB VLAN

Compute VNI

POD#1

111,112

111,112

POD#2

221,222

221,222

POD#3

331,332

331,332

POD#4

441,442

441,442

POD#5

551,552

551,552

POD#6

661,662

661,662

POD#7

771,772

771,772

POD#8

881,882

881,882

POD#9

991,992

991,992

POD#10

1011,1012

1011,1012

POD#11

1111,1112

1111,1112

POD#12

1221,1222

1221,1222

POD#13

1331,1332

1331,1332

POD#14

1441,1442

1441,1442

POD#15

1551,1552

1551,1552

POD#16

1661,1662

1661,1662

  • VRF - VNI

    • OOB - 10010

    • Inband - 200100

    • Storage - 400100

  • VxLAN - VLAN ID

    • OOB - 1001

    • Inband - 2001

  • Cluster Route Advertisement and External Reachability

  • BMS Connectivity - Data Center Environment Monitoring

  • 3rd Party Appliance Connectivity Requirements (Storage, etc.)

    • Physical Connectivity (Type, Speed, etc.)

    • Logical Connectivity (L2, Bond, L3, etc.)

    • IP Addressing

  • NVIS Deployment Preparation Information Requirements

    • MAC Address Collection of Devices (for DHCP) - “Factory File”

      • Host: BMC + N/S Provisioning Interfaces

        • Mgmt Servers (control plane)

        • GB200 Servers

        • UFM Servers

        • NMX servers

      • Switch: Mgmt interface (mgmt. or eth0)

        • Ethernet

        • Infiniband

        • NVSwitch

      • PDUs: Mgmt interface (mgmt. or eth0)

      • “Factory File” Available from Manufacturing Partner

        • Includes component level MAC/Interface/SN/PN Information

          • GB200 compute tray

          • NVSwitch

          • MGX Rack Power Shelf

          • SN2201

          Alternative: NVIS builds sheet with required information

      Point-to-Point Cabling Connectivity Plan

      • Created and finalized in the SIB

Edge Network Handoff#

Edge Network Handoff

Figure 2 Edge Network Handoff#

Edge Network is the network that connects the SuperPOD to the customer’s network.

  • In-Band management for all ControlPlane mgmt and handoff routes.

  • VTEP on Leaf Switches: When VTEP functionality is terminated at the leaf layer, it means that leaf switches handle the encapsulation and decapsulation of VXLAN traffic. This reduces latency for traffic destined for or originating from the end hosts connected to the leaf switches.

  • eBGP between Border TOR and Customer Edge: Establish an external Border Gateway Protocol (eBGP) session between the border leaf switches (which connect the EVPN fabric to external networks) and the CE devices.