Manual Rack Import Process#

If manual configuration of the node entries is needed, the .json examples in this section can be modified and then imported.

  1. Get rack Inventory file from factory once the rack passes L11 testing. If the rack inventory file is unavailable/unobtainable, get the MAC address info from the Bluefield cards in the compute trays and the eth0/COMe0 or eth1/COMe1 ports on the NVLink Switch trays.

  2. Manually create a .json for each GB200/GB300 compute tray, NVLink switch, and powershelf.

  3. Import the .json files into BCM.

    Note

    For rack hostnames, follow the convention <RACK_LOCATION>-<RU>-<POD_NUMBER>-<TRAY_TYPE>-<NODE_NUMBER>. Tray Type can be DGX, NVSW, or PWR.

Rack Entry .json Creation#

From the rack inventory file, the bcm-netautogen script will generate a .json file for each rack component:

  1. The bcm-netautogen script generates the appropriate IPs per interface. Pulls the appropriate information such as NIC MAC addresses and assigns IPs accordingly.

  1. If bcm-netautogen is unavailable, and manual creation of the .json file(s) is needed, a reference for each node type has been provided in this section.

    Below is an example of all the .json files that are generated by bcm-netautogen for import:

    Example: All Rack .json files generated by bcm-netautogen :icon: code :open:
    # Example: All Rack .json files generated by bcm-netautogen
    
    a05-p1-dgx-01-c01.json
    a05-p1-dgx-01-c02.json
    a05-p1-dgx-01-c03.json
    a05-p1-dgx-01-c04.json
    a05-p1-dgx-01-c05.json
    a05-p1-dgx-01-c06.json
    a05-p1-dgx-01-c07.json
    a05-p1-dgx-01-c08.json
    a05-p1-dgx-01-c09.json
    a05-p1-dgx-01-c10.json
    a05-p1-dgx-01-c11.json
    a05-p1-dgx-01-c12.json
    a05-p1-dgx-01-c13.json
    a05-p1-dgx-01-c14.json
    a05-p1-dgx-01-c15.json
    a05-p1-dgx-01-c16.json
    a05-p1-dgx-01-c17.json
    a05-p1-dgx-01-c18.json
    a05-p1-nvsw-01.json
    a05-p1-nvsw-02.json
    a05-p1-nvsw-03.json
    a05-p1-nvsw-04.json
    a05-p1-nvsw-05.json
    a05-p1-nvsw-06.json
    a05-p1-nvsw-07.json
    a05-p1-nvsw-08.json
    a05-p1-nvsw-09.json
    a05-p1-pwr-01.json
    a05-p1-pwr-02.json
    a05-p1-pwr-03.json
    a05-p1-pwr-04.json
    a05-p1-pwr-05.json
    a05-p1-pwr-06.json
    a05-p1-pwr-07.json
    a05-p1-pwr-08.json
    

Compute tray .json#

The following example provides details for a compute tray .json file that can be modified to import node information into BCM11.

Example: Compute tray .json file
{
  "baseType": "Device",
  "biosSetup": null,
  "bmcSettings": {
    "baseType": "BMCSettings",
    "firmwareManageMode": "GB200"
  },
  "bootLoader": "CATEGORY",
  "bootLoaderProtocol": "CATEGORY",
  "category": "dgx-gb200",
  "childType": "PhysicalNode",
  "cmdaemonUrl": "https://7.241.21.141:8081",
  "creationTime": 1744054247,
  "extra_values": {
    "Leak Detection": "GB200"
  },
  "fips": "CATEGORY",
  "hostname": "a08-p1-dgx-04-c01",
  "interfaces": [
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkBondInterface",
      "interfaces": [
        "enP22p3s0f0np0",
        "enP6p3s0f0np0"
      ],
      "ip": "7.241.21.141",
      "mode": 4,
      "name": "bond0",
      "network": "dgxnet2",
      "onNetworkPriority": 70,
      "options": "miimon=100",
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibp3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibP2p3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibP16p3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibP18p3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "7.241.7.10",
      "name": "eth3",
      "network": "ipminet2",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:A0:69"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "7.241.7.11",
      "name": "eth4",
      "network": "ipminet2",
      "onNetworkPriority": 10,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:9F:F7"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkBmcInterface",
      "ip": "7.241.7.12",
      "name": "rf0",
      "network": "ipminet2",
      "onNetworkPriority": 10,
      "startIf": "ALWAYS",
      "mac": "3C:6D:66:15:B7:FB"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "7.241.20.141",
      "name": "enP5p9s0",
      "network": "internalnet2",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "D0:F4:05:5B:E4:28"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "enP6p3s0f0np0",
      "onNetworkPriority": 65,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:A0:44"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "enP22p3s0f0np0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:9F:D2"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "100.127.1.109",
      "name": "enP6p3s0f1np1",
      "network": "storagenet",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:A0:45"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "100.127.129.109",
      "name": "enP22p3s0f1np1",
      "network": "storagenet",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:9F:D3"
    }
  ],
  "mac": "E0:9D:73:E8:9F:D2",
  "managementNetwork": "dgxnet2",
  "partition": "base",
  "provisioningInterface": "bond0",
  "provisioningTransport": "RSYNCDAEMON",
  "rackPosition": {
    "baseType": "RackPosition",
    "position": 11,
    "rack": "A08"
  },
  "tag": "1830625000348"
}

How to edit the GB200/GB300 tray .json:#

  1. In the above example, remove the BMC settings section. These settings should be inherited by the category.

  2. The default category name is dgx-gb200. Make sure this aligns with whatever was manually created.

  3. Ensure that for each node .json, that the rack name and position are set correctly. For nodes 01 to 08, the correlating positions are 10 to 17, respectively. For nodes 09 to 18, the positions are 27 to 36.

  4. Ensure that the cmdaemon URL is set to the IP of the node.

  5. Replace the network names for each interface to match what was manually defined for the networks, then set their IPs accordingly.

  6. For GB200: enP6p3s0f0np0 and enP22p3s0f0np0 are M1 (management 1) and M2 (management 2) according to the P2P; these are the net names for the 200G management network connections (can be dgxnet or internalnet depending on the cluster network configuration).

    For GB300: only enP22p3s0f0np0 is assigned to M1 (management 1); there is no M2.

  7. For GB200: enP6p3s0f1np1 and enP22p3s0f1np1 are port S1 and S2 according to the P2P and are for fast storage network connections. They are not present in the reference example and can be added to set up storage.

    For GB300: only enP22p3s0f1np1 is assigned to S1; there is no S2.

  8. For GB300 remove the interface entries for enP6p3s0f0np0 and enP6p3s0f1np1. Remove the bond0 entry, and set the provisioning interface to enP22p3s0f0np0 (M1).

  9. eth 3 and eth 4 are for Bluefield BMC connections. This can be omitted as these ports are not used in production.

  10. Setting up interface enP5p9s0 is optional. This is the 1G LOM port on the reference DGX GB200 compute tray. It is not cabled by default. internalnet2 can also be omitted.

Power shelf .json#

Example: Power shelf .json
{
"accessSettings": {
  "baseType": "AccessSettings",
  "username": "admin",
  "password": "0penBmc"
},
"baseType": "Device",
"childType": "PowerShelf",
"creationTime": 1744054247,
"hostname": "a08-p1-pwr-01",
"interfaces": [
  {
    "baseType": "NetworkInterface",
    "bringupduringinstall": "NO",
    "childType": "NetworkPhysicalInterface",
    "ip": "7.241.6.141",
    "mac": "00:18:23:0C:40:7D",
    "name": "eth0",
    "network": "ipminet1",
    "onNetworkPriority": 60,
    "startIf": "ALWAYS"
  }
],
"mac": "00:18:23:0C:40:7D",
"managementNetwork": "ipminet1",
"partition": "base",
"pmcSettings": {
  "baseType": "PMCSettings",
  "userName": "root",
  "password": "0penBmc"
},
"rackPosition": {
  "baseType": "RackPosition",
  "rack": "A08",
  "position": 6
}
}