Automated Rack Import Process#

The rack consists of eighteen GB200 compute trays, nine NVLink Switch trays, and eight power shelves. The configuration of these components is largely taken care of by the rack import process that is a part of the bcm-netautogen tool and the bcm-post-install automation.

  1. Get rack Inventory file from factory once the rack passes L11 testing.

  2. Use the bcm-netautogen tool with its inputs from the rack inventory file, the Point to Point (P2P) file, and a siteinformation.yaml to create .json files for each GB200 compute tray, NVLink switch, and powershelf. IPs will be assigned based on the available networking subnets as defined by the customer..

  3. Run the bcm-post-install automation to import the .json files into BCM.

For more details about the inputs to the bcm-netautogen tool and the bcm-post-install automation, please see the NVIDIA Mission Control DGX SuperPOD Ethernet North-South Network Configuration Guide guidance.

Rack Inventory File#

After each rack passes L11 testing at the factory, a rack inventory file will be generated and sent to the customer or the relevant deployment contact. After import, all the device information will be in BCM, including rack data for the rack management feature such as RU location on the rack.

Rack Entry .json Creation#

From the rack inventory file, the bcm-netautogen script will generate a .json file for each rack component. bcm-netautogen script generates the appropriate IPs per interface. Pulls the appropriate information such as NIC MAC addresses and assigns IPs accordingly. The following is an example of all the .json files that are generated by bcm-netautogen for import:

Example: Rack .json files for a DGX GB200 rack

a05-p1-dgx-01-c01.json
a05-p1-dgx-01-c02.json
a05-p1-dgx-01-c03.json
a05-p1-dgx-01-c04.json
a05-p1-dgx-01-c05.json
a05-p1-dgx-01-c06.json
a05-p1-dgx-01-c07.json
a05-p1-dgx-01-c08.json
a05-p1-dgx-01-c09.json
a05-p1-dgx-01-c10.json
a05-p1-dgx-01-c11.json
a05-p1-dgx-01-c12.json
a05-p1-dgx-01-c13.json
a05-p1-dgx-01-c14.json
a05-p1-dgx-01-c15.json
a05-p1-dgx-01-c16.json
a05-p1-dgx-01-c17.json
a05-p1-dgx-01-c18.json
a05-p1-nvsw-01.json
a05-p1-nvsw-02.json
a05-p1-nvsw-03.json
a05-p1-nvsw-04.json
a05-p1-nvsw-05.json
a05-p1-nvsw-06.json
a05-p1-nvsw-07.json
a05-p1-nvsw-08.json
a05-p1-nvsw-09.json
a05-p1-pwr-01.json
a05-p1-pwr-02.json
a05-p1-pwr-03.json
a05-p1-pwr-04.json
a05-p1-pwr-05.json
a05-p1-pwr-06.json
a05-p1-pwr-07.json
a05-p1-pwr-08.json

Compute tray .json#

The following example provides details for a compute tray .json file that can be modified to import node information into BCM11.

Example: Compute tray .json
{
"baseType": "Device",
"biosSetup": null,
"bmcSettings": {
  "baseType": "BMCSettings",
  "firmwareManageMode": "GB200"
},
"bootLoader": "CATEGORY",
"bootLoaderProtocol": "CATEGORY",
"category": "dgx-gb200",
"childType": "PhysicalNode",
"cmdaemonUrl": "https://7.241.21.141:8081",
"creationTime": 1744054247,
"extra_values": {
  "Leak Detection": "GB200"
},
"fips": "CATEGORY",
"hostname": "a08-p1-dgx-04-c01",
"interfaces": [
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkBondInterface",
    "interfaces": [
      "enP22p3s0f0np0",
      "enP6p3s0f0np0"
    ],
    "ip": "7.241.21.141",
    "mode": 4,
    "name": "bond0",
    "network": "dgxnet2",
    "onNetworkPriority": 70,
    "options": "miimon=100",
    "startIf": "ALWAYS"
  },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "name": "ibp3s0",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "name": "ibP2p3s0",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "name": "ibP16p3s0",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "name": "ibP18p3s0",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "7.241.7.10",
    "name": "eth3",
    "network": "ipminet2",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS",
    "mac": "E0:9D:73:E8:A0:69"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "7.241.7.11",
    "name": "eth4",
    "network": "ipminet2",
    "onNetworkPriority": 10,
    "startIf": "ALWAYS",
    "mac": "E0:9D:73:E8:9F:F7"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkBmcInterface",
    "ip": "7.241.7.12",
    "name": "rf0",
    "network": "ipminet2",
    "onNetworkPriority": 10,
    "startIf": "ALWAYS",
    "mac": "3C:6D:66:15:B7:FB"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "7.241.20.141",
    "name": "enP5p9s0",
    "network": "internalnet2",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS",
    "mac": "D0:F4:05:5B:E4:28"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "name": "enP6p3s0f0np0",
    "onNetworkPriority": 65,
    "startIf": "ALWAYS",
    "mac": "E0:9D:73:E8:A0:44"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "name": "enP22p3s0f0np0",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS",
    "mac": "E0:9D:73:E8:9F:D2"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "100.127.1.109",
    "name": "enP6p3s0f1np1",
    "network": "storagenet",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS",
    "mac": "E0:9D:73:E8:A0:45"
    },
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "100.127.129.109",
    "name": "enP22p3s0f1np1",
    "network": "storagenet",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS",
    "mac": "E0:9D:73:E8:9F:D3"
    }
],
"mac": "E0:9D:73:E8:9F:D2",
"managementNetwork": "dgxnet2",
"partition": "base",
"provisioningInterface": "bond0",
"provisioningTransport": "RSYNCDAEMON",
"rackPosition": {
  "baseType": "RackPosition",
  "position": 11,
  "rack": "A08"
},
"tag": "1830625000348"
}

Power shelf .json#

Example: Power shelf .json
{
"accessSettings": {
  "baseType": "AccessSettings",
  "username": "admin",
  "password": "0penBmc"
},
"baseType": "Device",
"childType": "PowerShelf",
"creationTime": 1744054247,
"hostname": "a08-p1-pwr-01",
"interfaces": [
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "7.241.6.141",
    "mac": "00:18:23:0C:40:7D",
    "name": "eth0",
    "network": "ipminet1",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS"
  }
],
"mac": "00:18:23:0C:40:7D",
"managementNetwork": "ipminet1",
"partition": "base",
"pmcSettings": {
  "baseType": "PMCSettings",
  "userName": "root",
  "password": "0penBmc"
},
"rackPosition": {
  "baseType": "RackPosition",
  "rack": "A08",
  "position": 6
}
}