Manual Rack Import Process#

If manual configuration of the node entries is needed, the .json examples in this section can be modified and then imported.

Get rack Inventory file from factory once the rack passes L11 testing. If the rack inventory file is unavailable/unobtainable, get the MAC address info from the Bluefield cards in the compute trays and the eth0/COMe0 or eth1/COMe1 ports on the NVLink Switch trays.
Manually create a .json for each GB200 compute tray, NVLink switch, and powershelf.
Import the .json in BCM.

Note

For rack hostnames, follow the convention <Rack Location>-<RU>-<POD Number>-<Tray Type>-<node number>. Tray Type can be DGX, NVSW, or PWR.

Rack Entry .json Creation#

From the rack inventory file, the bcm-netautogen script will generate a .json file for each rack component:

bcm-netautogen script generates the appropriate IPs per interface. Pulls the appropriate information such as NIC MAC addresses and assigns IPs accordingly.

If bcm-netautogen is unavailable, and manual creation of the .json file(s) is needed, a reference for each node type has been provided in this section.

Below is an example of all the .json files that are generated by bcm-netautogen for import:

Example: Rack .json files

a05-p1-dgx-01-c01.json
a05-p1-dgx-01-c02.json
a05-p1-dgx-01-c03.json
a05-p1-dgx-01-c04.json
a05-p1-dgx-01-c05.json
a05-p1-dgx-01-c06.json
a05-p1-dgx-01-c07.json
a05-p1-dgx-01-c08.json
a05-p1-dgx-01-c09.json
a05-p1-dgx-01-c10.json
a05-p1-dgx-01-c11.json
a05-p1-dgx-01-c12.json
a05-p1-dgx-01-c13.json
a05-p1-dgx-01-c14.json
a05-p1-dgx-01-c15.json
a05-p1-dgx-01-c16.json
a05-p1-dgx-01-c17.json
a05-p1-dgx-01-c18.json
a05-p1-nvsw-01.json
a05-p1-nvsw-02.json
a05-p1-nvsw-03.json
a05-p1-nvsw-04.json
a05-p1-nvsw-05.json
a05-p1-nvsw-06.json
a05-p1-nvsw-07.json
a05-p1-nvsw-08.json
a05-p1-nvsw-09.json
a05-p1-pwr-01.json
a05-p1-pwr-02.json
a05-p1-pwr-03.json
a05-p1-pwr-04.json
a05-p1-pwr-05.json
a05-p1-pwr-06.json
a05-p1-pwr-07.json
a05-p1-pwr-08.json

Compute tray .json#

The following example provides details for a compute tray .json file that can be modified to import node information into BCM11.

Example: Compute tray .json

{
  "baseType": "Device",
  "biosSetup": null,
  "bmcSettings": {
    "baseType": "BMCSettings",
    "firmwareManageMode": "GB200"
  },
  "bootLoader": "CATEGORY",
  "bootLoaderProtocol": "CATEGORY",
  "category": "dgx-gb200",
  "childType": "PhysicalNode",
  "cmdaemonUrl": "https://7.241.21.141:8081",
  "creationTime": 1744054247,
  "extra_values": {
    "Leak Detection": "GB200"
  },
  "fips": "CATEGORY",
  "hostname": "a08-p1-dgx-04-c01",
  "interfaces": [
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkBondInterface",
      "interfaces": [
        "enP22p3s0f0np0",
        "enP6p3s0f0np0"
      ],
      "ip": "7.241.21.141",
      "mode": 4,
      "name": "bond0",
      "network": "dgxnet2",
      "onNetworkPriority": 70,
      "options": "miimon=100",
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibp3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibP2p3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibP16p3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "ibP18p3s0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "7.241.7.10",
      "name": "eth3",
      "network": "ipminet2",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:A0:69"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "7.241.7.11",
      "name": "eth4",
      "network": "ipminet2",
      "onNetworkPriority": 10,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:9F:F7"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkBmcInterface",
      "ip": "7.241.7.12",
      "name": "rf0",
      "network": "ipminet2",
      "onNetworkPriority": 10,
      "startIf": "ALWAYS",
      "mac": "3C:6D:66:15:B7:FB"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "7.241.20.141",
      "name": "enP5p9s0",
      "network": "internalnet2",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "D0:F4:05:5B:E4:28"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "enP6p3s0f0np0",
      "onNetworkPriority": 65,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:A0:44"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "name": "enP22p3s0f0np0",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:9F:D2"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "100.127.1.109",
      "name": "enP6p3s0f1np1",
      "network": "storagenet",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:A0:45"
    },
    {
      "baseType": "NetworkInterface",
      "bringupduringinstall": "NO",
      "childType": "NetworkPhysicalInterface",
      "ip": "100.127.129.109",
      "name": "enP22p3s0f1np1",
      "network": "storagenet",
      "onNetworkPriority": 60,
      "startIf": "ALWAYS",
      "mac": "E0:9D:73:E8:9F:D3"
    }
  ],
  "mac": "E0:9D:73:E8:9F:D2",
  "managementNetwork": "dgxnet2",
  "partition": "base",
  "provisioningInterface": "bond0",
  "provisioningTransport": "RSYNCDAEMON",
  "rackPosition": {
    "baseType": "RackPosition",
    "position": 11,
    "rack": "A08"
  },
  "tag": "1830625000348"
}

How to edit the GB200 tray .json:#

In the above example, remove the BMC settings section. These settings should be inherited by the category.

The default category name is dgx-gb200. Make sure this aligns with whatever was manually created.

Ensure that for each node .json, that the rack name and position are set correctly. For nodes 01 to 08, the correlating positions are 10 to 17, respectively. For nodes 09 to 18, the positions are 27 to 36.

Ensure that the cmdaemon URL is set to the IP of the node.

Replace the network names for each interface to match what was manually defined for the networks, then set their IPs accordingly.

enP6p3s0f0np0 and enP22p3s0f0np0 are M1 (management 1) and M2 (management 2) according to the P2P, these are the net names for the 200G management network connections (can be dgxnet or internalnet depending on the cluster network configuration).

enP6p3s0f1np1 and enP22p3s0f1np1 are port S1 and S2 according to the P2P and are for fast storage network connections. They are not present in the reference example and can be added to set up storage.

eth 3 and eth 4 are for Bluefield BMC connections. This can be omitted as these ports are not used in production

Setting up interface enP5p9s0 is optional. This is the 1G LOM port on the reference DGX GB200 compute tray. It is not cabled by default. internalnet2 can also be omitted

NVLink Switch .json#

How to edit the NVLink Switch tray .json:#

Ensure that the access settings are set to the correct username and password.

For the BMC settings, set the firmwareManageMode to GB200sw.

Ensure that the NVLink switch is set to the correct hostname.

The MAC is after the line for “kind”: “NVLINK” is the generally the same as eth0/COMe0. By default, this is the port that is connected while eth1/COMe1 is generally left unconnected.

Ensure that for each node .json, that the rack name and position are set correctly. For NVLink switch 01 to 09, the correlating positions are 18 to 26, respectively.

eth0 and eth1 (also known as COMe0 and COMe1) are the primary connections to the NVLink Switch network. Replace the network names for each interface to the NVLink Switch network (often the same as ipminet).

rf0 is the BMC port for the NVLink Switch. This should be set to the IP of the BMC port on the NVLink Switch.

The managementNetwork is the NVLink Switch network.

Manual NVLink Switch tray .json script#

The following script will take a .csv file as an input and create the .json entries for the NVLink Switch trays.

Reference: Input NVLinkSwitch .csv.

# data should be in the following format:

#hostname,eth0_ip,eth0_mac,eth1_ip,eth1_mac,rf0_ip,rf0_mac,rack,position,,

a18-p1-nvsw-01,172.16.6.61,E0:9D:73:A8:F3:E4,172.16.6.71,00:00:00:00:00:00,172.16.6.81,2C:5E:AB:B2:17:7E,A18,18
a18-p1-nvsw-02,172.16.6.62,E0:9D:73:C4:88:4E,172.16.6.72,00:00:00:00:00:00,172.16.6.82,2C:5E:AB:B2:16:BE,A18,19
a18-p1-nvsw-03,172.16.6.63,E0:9D:73:B2:57:F4,172.16.6.73,00:00:00:00:00:00,172.16.6.83,2C:5E:AB:B0:E4:BE,A18,20
a18-p1-nvsw-04,172.16.6.64,7C:8C:09:1F:EE:00,172.16.6.74,00:00:00:00:00:00,172.16.6.84,2C:5E:AB:C4:6C:BE,A18,21
a18-p1-nvsw-05,172.16.6.65,E0:9D:73:A9:18:20,172.16.6.75,00:00:00:00:00:00,172.16.6.85,2C:5E:AB:B2:17:3E,A18,22
a18-p1-nvsw-06,172.16.6.66,E0:9D:73:C5:7D:00,172.16.6.76,00:00:00:00:00:00,172.16.6.86,2C:5E:AB:B2:06:7E,A18,23
a18-p1-nvsw-07,172.16.6.67,E0:9D:73:C5:7E:C8,172.16.6.77,00:00:00:00:00:00,172.16.6.87,2C:5E:AB:B0:E3:FE,A18,24
a18-p1-nvsw-08,172.16.6.68,E0:9D:73:C4:58:28,172.16.6.78,00:00:00:00:00:00,172.16.6.88,2C:5E:AB:B0:E5:BE,A18,25
a18-p1-nvsw-09,172.16.6.69,E0:9D:73:B2:45:14,172.16.6.79,00:00:00:00:00:00,172.16.6.89,B0:CF:0E:FF:E4:FE,A18,26

Reference: NVLink Switch .json creation script

  #!/bin/bash

#
# switch_config_generator.sh - Generate switch configuration JSONs from CSV data
#
# This script takes a CSV file containing network configuration details and
# generates JSON files following the format shown in the provided example.
#
# Usage: ./switch_config_generator.sh <csv_file> [output_directory] [network_name]

# Function to validate IP address format with detailed error reporting
validate_ip() {
    local ip=$1

    # Check if empty
    if [ -z "$ip" ]; then
        echo "IP address cannot be empty"
        return 1
    fi

    # Check basic format
    if ! [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
        echo "IP address '$ip' does not match pattern XXX.XXX.XXX.XXX"
        return 1
    fi

    # Check each octet is <= 255
    OIFS=$IFS
    IFS='.'
    octets=($ip)
    IFS=$OIFS

    for i in {0..3}; do
        if [ ${octets[$i]} -gt 255 ]; then
            echo "Octet #$((i+1)) in IP '$ip' exceeds 255 (${octets[$i]})"
            return 1
        fi
    done

    return 0
}

# Function to validate MAC address format with detailed error reporting
validate_mac() {
    local mac=$1

    # Empty MAC is allowed for eth1 (as seen in the example)
    if [ -z "$mac" ]; then
        return 0
    fi

    # Check MAC format: XX:XX:XX:XX:XX:XX
    if ! [[ $mac =~ ^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$ ]]; then
        if [[ $mac =~ [^0-9A-Fa-f:] ]]; then
            echo "MAC address '$mac' contains invalid characters (only hex digits and colons allowed)"
        elif [[ $mac =~ ::+ ]]; then
            echo "MAC address '$mac' contains consecutive colons"
        elif [[ $mac =~ ^: || $mac =~ :$ ]]; then
            echo "MAC address '$mac' starts or ends with a colon"
        else
            echo "MAC address '$mac' does not match format XX:XX:XX:XX:XX:XX (6 hex pairs separated by colons)"
        fi
        return 1
    fi

    return 0
}

# Check if CSV file is provided
if [ $# -lt 1 ]; then
    echo "Usage: $0 <csv_file> [output_directory] [network_name]"
    echo ""
    echo "Parameters:"
    echo " csv_file - Input CSV file with network configuration data"
    echo " output_directory - Directory to store generated JSON files (default: current directory)"
    echo " network_name - Network name to use in configuration (default: ipminet)"
    echo ""
    echo "CSV format: hostname,eth0_ip,eth0_mac,eth1_ip,eth1_mac,rf0_ip,rf0_mac,rack,position"
    echo "Example: a18-p1-nvsw-01,172.16.6.61,E0:9D:73:A8:F3:E4,172.16.6.71,,172.16.6.81,2C:5E:AB:B2:17:7E,A18,18"
    exit 1
fi

CSV_FILE=$1
OUTPUT_DIR="${2:-.}" # Use current directory if not specified
NETWORK="${3:-ipminet}" # Use 'ipminet' as default network if not specified

# Check if file exists
if [ ! -f "$CSV_FILE" ]; then
    echo "Error: File $CSV_FILE not found"
    exit 1
fi

# Create output directory if it doesn't exist
mkdir -p "$OUTPUT_DIR"

echo "-------------------------------------------"

# Process each line of the CSV
tail -n +2 "$CSV_FILE" | while IFS=, read -r hostname eth0_ip eth0_mac eth1_ip eth1_mac rf0_ip rf0_mac rack position
do
    echo "Processing host: $hostname"

    # Validate hostname
    if [ -z "$hostname" ]; then
        echo "Error: Hostname is required"
        continue
    fi

    # Validate IPs with detailed error reporting
    validation_failed=false

    # Check eth0 IP
    ip_error=$(validate_ip "$eth0_ip")
    if [ $? -ne 0 ]; then
        echo " Error in eth0 IP: $ip_error"
        validation_failed=true
    fi

    # Check eth1 IP
    ip_error=$(validate_ip "$eth1_ip")
    if [ $? -ne 0 ]; then
        echo " Error in eth1 IP: $ip_error"
        validation_failed=true
    fi

    # Check rf0 IP
    ip_error=$(validate_ip "$rf0_ip")
    if [ $? -ne 0 ]; then
        echo " Error in rf0 IP: $ip_error"
        validation_failed=true
    fi

    # Validate MACs with detailed error reporting
    # Check eth0 MAC (required)
    mac_error=$(validate_mac "$eth0_mac")
    if [ $? -ne 0 ]; then
        echo " Error in eth0 MAC: $mac_error"
        validation_failed=true
    fi

    # Check eth1 MAC (can be empty)
    mac_error=$(validate_mac "$eth1_mac")
    if [ $? -ne 0 ]; then
        echo " Error in eth1 MAC: $mac_error"
        validation_failed=true
    fi

    # Check rf0 MAC (required)
    mac_error=$(validate_mac "$rf0_mac")
    if [ $? -ne 0 ]; then
        echo " Error in rf0 MAC: $mac_error"
        validation_failed=true
    fi

    # Validate position is numeric
    if ! [[ "$position" =~ ^[0-9]+$ ]]; then
        echo " Error: Position must be a number, got '$position'"
        validation_failed=true
    fi

    # Validate rack is not empty
    if [ -z "$rack" ]; then
        echo " Error: Rack name is required"
        validation_failed=true
    fi

    # Skip JSON generation if validation failed
    if [ "$validation_failed" = true ]; then
        echo " Skipping JSON generation for $hostname due to validation errors"
        echo "-------------------------------------------"
        continue
    fi

    # Create JSON filename based on hostname
    JSON_FILE="${OUTPUT_DIR}/${hostname}.json"

    # Generate JSON file
    cat > "$JSON_FILE" << EOF
{
"accessSettings": {
"baseType": "AccessSettings",
"username": "admin",
"password": "Juliet@1234"
},
"baseType": "Device",
"bmcSettings": {
"baseType": "BMCSettings",
"firmwareManageMode": "gb200sw",
"password": "JulietBmc@123",
"privilege": "ADMINISTRATOR",
"userID": 1,
"userName": "root"
},
"childType": "Switch",
"creationTime": $TIMESTAMP,
"disablePortDetection": true,
"disableSNMP": true,
"hasClientDaemon": true,
"hostname": "$hostname",
"interfaces": [
{
"baseType": "NetworkInterface",
"bringupduringinstall": "NO",
"childType": "NetworkPhysicalInterface",
"ip": "$eth0_ip",
"mac": "$eth0_mac",
"name": "eth0",
"network": "$NETWORK",
"onNetworkPriority": 65,
"startIf": "ALWAYS"
},
{
"baseType": "NetworkInterface",
"bringupduringinstall": "NO",
"childType": "NetworkPhysicalInterface",
"ip": "$eth1_ip",
"mac": "$eth1_mac",
"name": "eth1",
"network": "$NETWORK",
"onNetworkPriority": 60,
"startIf": "ALWAYS"
},
{
"baseType": "NetworkInterface",
"bringupduringinstall": "NO",
"childType": "NetworkBmcInterface",
"ip": "$rf0_ip",
"mac": "$rf0_mac",
"name": "rf0",
"network": "$NETWORK",
"onNetworkPriority": 10,
"startIf": "ALWAYS"
}
],
"kind": "NVLINK",
"mac": "$eth0_mac",
"managementNetwork": "$NETWORK",
"nvConfiguration": null,
"nvConfigurationMode": "AUTO",
"partition": "base",
"powerControl": "rf0",
"rackPosition": {
"baseType": "RackPosition",
"position": $position,
"rack": "$rack"
},
"ztpSettings": {
"baseType": "ZTPSettings",
"enableAPI": true,
"switchImage": "nvos-amd64-25.02.2141.bin",
"ztpJsonTemplate": "nvlink-nvos.json",
"ztpScriptTemplate": "nvos-ztp.sh"
}
}
EOF

        echo " Success: Generated JSON file: $JSON_FILE"
        echo "-------------------------------------------"
    done

    echo "Processing complete. JSON files generated in $OUTPUT_DIR"

Power shelf .json#

Import Compute Nodes, NVLink Switches, and Power Shelves#

Each .json file can be imported individually into BCM.

cmsh-c "main;import <.json file location./<rack number>/<.json file name>"

Note

No separate commit is necessary after imports; the devices are committed automatically as part of the import command.

Caution

Importing a .json of the same name will overwrite the current settings without warning or prompting.

After the rack import is complete, all devices can be controlled once they are provisioned at the individual node up to the rack level.

Example: GB200 compute tray node information in BCM

a03-p1-head-01->device[b05-p1-dgx-05-c01]]% show

Parameter                              Value
--------------------------------------  ------------------------------------------------------------

Leak Detection                          GB200
Hostname                                b05-p1-dgx-05-c01
IP                                      7.241.18.139
Network                                 dgxnet2
Revision
Type                                    PhysicalNode
Mac                                     EE:67:84:CB:A1:6B
Use exclusively for                     (category:dgx-gb200)
Category                                dgx-gb200
Activation                              Mon, 10 Feb 2025 14:33:26 PST
Rack                                    B05:11
Chassis                                 < not set >
Access Settings                         <submode>
Roles                                   <0 in submode>
Software image                          dgxos7-570.82-image (category:dgx-gb200)
Node installer disk                     no
Install boot record                     no
Install mode
Next install mode
Time zone                               America/Los_Angeles (partition:base)
Management network                      dgxnet2
Data node                               no
Disk setup                              <1818 bytes> (category:dgx-gb200)
Hardware RAID configuration             <0B>
Initialize script                       <0B>
Finalize script                         <0B>
BIOS setup                              < not set >
Allow networking restart                no
PXE Label
Block devices cleared on next boot
Provisioning interface                  bond0
Provisioning Transport                  RSYNCDAEMON
Exclude list manipulate script          <0B>
Exclude list full install               <0B>
Exclude list sync install               <0B>
Exclude list update                     <0B>
Exclude list grab                       <0B>
Exclude list grab new                   <0B>
Power control                           rf0
Custom power script
Custom power script argument
Power distribution units
IO scheduler
Kernel version                          6.8.0-1018-nvidia-64k (software image:dgxos7-570.82-image)
Kernel parameters                       rd.driver.blacklist=nouveau systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller (software image:dgxos7-570.82-image)
Kernel output console                   tty0 (software image:dgxos7-570.82-image)
Boot loader                             grub (dgx-gb200)
Boot loader protocol                    HTTP (dgx-gb200)
Boot loader file
Kernel modules                          51 (software image:dgxos7-570.82-image)
FIPS                                    no (dgx-gb200)
Template node                           no
From template node
Default gateway                         7.241.18.129 (network: dgxnet2)
Default gateway metric                  0
Switch ports
Interfaces                              <13 in submode>
Static routes                           <0 in submode>
GPU Settings                            <0 in submode>
BMC Settings                            <submode>
SELinux Settings                        <submode>
Filesystem mounts                       <0 in submode>
Filesystem exports                      <0 in submode>
Services                                <0 in submode>
Userdefined1
Userdefined2
User defined resources
Supports GNSS                           no
Custom ping script
Custom ping script argument
Custom remote console script
Custom remote console script argument
Partition                               base
Version config files                    no
Part number
Serial number
Notes                                   <0B>
Prometheus metric forwarders            <0 in submode>