Networking Planning and Design#

A NVIDIA team will have a network requirements gathering meeting with a SuperPOD customer directly to discuss network requirements and obtain the IP information. The tables below shows the information that is needed in the planning stages:

Logical Network Design#

Logical Network Design for GB200

Figure 1 Logical Network Design Architecture#

Short Summary:

The following items provide a short summary of the logical network design:

  • BGP EVPN provides control plane signaling and tunnel discovery support in a VxLAN overlay network.

  • Loopback interfaces are commonly used for VTEPs (VXLAN Tunnel Endpoints) to establish BGP peering. Each VTEP is assigned a loopback IP address, which serves as a stable endpoint for routing EVPN control messages.

  • This approach enhances scalability and reliability, allowing for dynamic MAC address and IP address learning while maintaining tenant isolation. The VXLAN encapsulation enables seamless communication across the underlying network infrastructure, making it ideal for multi-tenant data centers and cloud environments.

Ethernet Networking (North-to-South)#

Reference: IP Subnet Range/CIDR Requirements per BCM Network

Table 4 IP Subnet Range/CIDR Requirements per BCM Network#

Network

IP Range Requirement

In Band Mgmt

  • internalnet: Control Plane Data

  • dgxnet[1-n]: DGX

Out of Band (OOB)

  • ipminet[1-n]: IPMI Covers GB200 (2x per GB200 rack), SN2201s, … Control plane 2201

Loopback

/24 - to cover address from each switch

Client P2P IP Addressing (TOR to CE / Border TOR to EDGE Network)

/31 - a minimum of 8 subnets

Non-Routable IP Range

ETH Storage Network

100.127.0.0/16

IB Compute Network

100.126.0.0/16

Vendor Storage Prefix

100.127.124.0/24

Subnet Breakout per POD#

The following table shows the subnet breakout per POD:

Table 5 Subnet Breakout per POD#

POD

OOB Total

OOB Root

DATA Total

DATA Root

ROOT

1

2x/24 + 2x/23

/21

2x/23 + 2x/25

/21

/20

2

2x/24 + 4x/23

/20

4x/23 + 4x/25

/20

/19

3

1x/24 + 6x/23

/20

6x/23 + 6x/25

/20

/19

4

1x/24 + 8x/23

/19

8x/23 + 8x/25

/19

/18

5

1x/24 + 10x/23

/19

10x/23 + 10x/25

/19

/18

6

1x/24 + 12x/23

/19

12x/23 + 12x/25

/19

/17

7

1x/24 + 14x/23

/19

14x/23 + 14x/25

/18

/17

8

1x/24 + 16x/23

/18

16x/23 + 16x/25

/18

/17

BGP ASN 4-Byte Range#

The following table shows the BGP ASN 4-byte range:

Table 6 BGP ASN 4-Byte Range#

ROLES

4-bytes ASN

OOB

429490**2361** - 429490**2460**

TOR/STOR

429490**2461** - 429490**2660**

SPINEs

429490**2661** - 429490**2750**

SuperSPINEs

429490**2751** - 429490**2760**

Note

TOR to CE Transceiver Compatibility, see Hardware Compatibility List (HCL)

We recommend that you review and refer to SuperPOD Network Requirements Document for more information.

Ethernet Planning#

Confirm Accurate Hardware Models and Quantities#

Confirm the you have the accurate hardware models and correct amount. You can reference this list of items:

  • Ethernet switches x amount of SN5600

  • Transceivers (optical) and cables (DAC / AOC)

  • Fibers

Confirm You Have the Correct Cumulus Linux Software Version#

You need to ensure that your are using the correct version of the Linux distribution called Cumulus Linux.

Obtain Customer Network Requirements#

You need to obtain the customer network requirements from the customer.

  • Fill out the “NVDA DGX SPOD GB200 Network Requirements” Excel file - Assuming in SIB

Content/Consideration List

  • Customer-Edge to Cluster connectivity

    • Transceiver type, compatibility and HW order status

    • Electrical signaling/encoding (NRZ vs PAM4)

    • Speed/Bandwidth

    • Uplink Quantity (transceiver, cable, fiber)

    • IP Addressing - P2P Uplinks

    • Routing protocol (default: BGP)

  • Routable IP Address Allocation

    • Discussed with and provided by the customer

  • Non-Routable IP Address Allocation

    The default NVDA address ranges should be used:

    • IB Compute = 100.126.0.0/16

    • Ethernet Underlay = 10.254.0.0/16

    • Ethernet Overlay Storage = 10.127.0.0/16

  • VLAN VNI Per TOR Pair

DGX Compute VLAN#

The following table shows the DGX Compute VLAN and VNI per POD:

Table 7 DGX Compute VLAN and VNI per POD#

POD#

DGX Compute VLANs

Compute VNI

POD#1

101,102

101,102

POD#2

201,202

201,202

POD#3

301,302

301,302

POD#4

401,402

401,402

POD#5

501,502

501,502

POD#6

601,602

601,602

POD#7

701,702

701,702

POD#8

801,802

801,802

POD#9

901,902

901,902

POD#10

1001,1002

1001,1002

POD#11

1001,1002

1001,1002

POD#12

1201,1202

1201,1202

POD#13

1301,1302

1301,1302

POD#14

1401,1402

1401,1402

POD#15

1501,1502

1501,1502

POD#16

1601,1602

1601,1602

OOB VLAN / VNI#

The following table shows the OOB VLAN and VNI per POD:

POD#

OOB VLAN

Compute VNI

POD#1

111,112

111,112

POD#2

221,222

221,222

POD#3

331,332

331,332

POD#4

441,442

441,442

POD#5

551,552

551,552

POD#6

661,662

661,662

POD#7

771,772

771,772

POD#8

881,882

881,882

POD#9

991,992

991,992

POD#10

1011,1012

1011,1012

POD#11

1111,1112

1111,1112

POD#12

1221,1222

1221,1222

POD#13

1331,1332

1331,1332

POD#14

1441,1442

1441,1442

POD#15

1551,1552

1551,1552

POD#16

1661,1662

1661,1662

The following shows the settings for VRF - VNI, VxLAN - VLAN ID, cluster route advertisement and external reachability, NVIS Deployment Preparation Information Requirements, and Point-to-Point Cabling Connectivity Plan.

  • VRF - VNI

    • OOB - 10010

    • Inband - 200100

    • Storage - 400100

  • VxLAN - VLAN ID

    • OOB - 1001

    • Inband - 2001

  • Cluster Route Advertisement and External Reachability

    • BMS Connectivity - Data Center Environment Monitoring

    • 3rd Party Appliance Connectivity Requirements (Storage, etc.)

      • Physical Connectivity (Type, Speed, etc.)

      • Logical Connectivity (L2, Bond, L3, etc.)

      • IP Addressing

  • NVIS Deployment Preparation Information Requirements

    • MAC Address Collection of Devices (for DHCP) - “Factory File”

      • Host: BMC + N/S Provisioning Interfaces

        • Mgmt Servers (control plane)

        • GB200 Servers

        • UFM Servers

        • NMX servers

      • Switch: Mgmt interface (mgmt. or eth0)

        • Ethernet

        • Infiniband

        • NVSwitch

      • PDUs: Mgmt interface (mgmt. or eth0)

      • “Factory File” Available from Manufacturing Partner

        • Includes component level MAC/Interface/SN/PN Information

          • GB200 compute tray

          • NVSwitch

          • MGX Rack Power Shelf

          • SN2201

          Alternative: NVIS builds sheet with required information

      • Point-to-Point Cabling Connectivity Plan

        • Created and finalized in the SIB

Edge Network Handoff#

This section describes the Edge Network Handoff.

Edge Network Handoff

Figure 2 Edge Network Handoff#

Edge Network is the network that connects the SuperPOD to the customer’s network.

The edge network is the network that connects the SuperPOD to the customer’s network and includes the following:

  • In-Band management for all ControlPlane mgmt and handoff routes.

  • VTEP on Leaf Switches: When VTEP functionality is terminated at the leaf layer, it means that leaf switches handle the encapsulation and decapsulation of VXLAN traffic. This reduces latency for traffic destined for or originating from the end hosts connected to the leaf switches.

  • eBGP between Border TOR and Customer Edge: Establish an external Border Gateway Protocol (eBGP) session between the border leaf switches (which connect the EVPN fabric to external networks) and the CE devices. This allows the SuperPOD to advertise routes to the customer’s network and receive routes from the customer’s network.