GB200/GB300 Rack Power On and Bring Up#

The GB200/GB300 rack bring up process can be summarized as follows. While this can be done in any order, it is advised that the NVLink Switches be brought up first so that the NVLink domain is already up and configured when the GB200 compute trays are powered on. Otherwise, all the compute nodes will have to be restarted to ensure they are able to communicate on the NVLink fabric correctly.

GB200/GB300 Compute Tray Bring Up Summary

  1. Establish and confirm power control of the rack devices in the rack that is being brought up.

  2. Power on the compute nodes to provision them. (Physical power on or through BMC either by its KVM software or using ipmitool).

  3. After provisioning login to the nodes from the headnode and confirm the state of the NICs (all bonds up, all connections are up at least for the north-south networking).

  4. After successful provisioning and bring up, assess the firmware status:

    • Check if the firmware levels match those reported in the SBOM.

    • Update the components if necessary.

NVLink Switch Tray Bring Up Summary

Check if the NVLink Switch is reachable using SSH (via the admin user on the COMe0 network):

  1. Check if the NVLink Switch BMCs are reachable using the following methods:

    • BCM device power status.

    • SSH directly into the BMC.

    • ipmitool.

  2. Install cm-lite-daemon:

    • If successful, the NVLink Switches will show as UP under the device list. It will then configure NMX-C and NMX-T automatically.

    • If cm-lite-daemon does not install successfully and the installer needs to get the NVLink domain up to progress, select an NVLink Switch to serve as the master and configure NMX-C and NMX-T manually.

    • If Zero Touch Provisioning (ZTP) is enabled/configured by bcm-netautogen, cm-lite-daemon will be installed and NMX-C and NMX-T will be configured automatically.

  3. After successful connectivity has been established on the COMe0/1 network and the BMC, assess the firmware status:

    • Check if the firmware levels match those reported in the SBOM. Update the components if necessary.

    • Check the version of NVOS and update the OS using each switch individually or use ZTP to do the NVOS update.

Power Shelves Bring Up Summary

  1. Power shelves are reporting as on. If they do not, try bouncing the ports from the switch side to get them to show status as being up.

  2. Do firmware updates.

NIC Firmware Update Bring Up Summary

Verify NIC firmware versions and update them if needed.

Initial Power On and Provisioning#

Upon completion of the rack import process or manual configuration of the cluster compute and control node entries, the GB200/GB300 racks are ready to be brought up. With the rf0 (redfish 0) ports configured within BCM with a MAC to IP, all the GB200/GB300 compute trays, NVLink Switch devices, and the power shelves will get their IP when the respective ipmitool is up.

GB200/GB300 Compute Tray Power On and Provisioning#

  1. Ensure that OOB power control of the GB200/GB300 compute trays is configured.

    Check power status:

    Individual Node:

    cmsh -c "device use <DGX GB200/GB300 compute tray>; power status"

    All devices in a rack:

    cmsh -c "device; power -r <rack number> status"

    • If the output says Skipped, that likely means the power control is not set.

    Set power control settings:

    One node:

    cmsh -c "device use <rack location>-<pod number>-dgx-<rack number>-c<node number>;set powercontrol rf0; commit"

    All nodes in a rack:

    cmsh -c "foreach -n <rack number>-<pod number>-dgx-<rack number>-c[01-18] (set powercontrol rf0; commit)"

    • If it says failed, that means that it can reach the BMC/rf0, but the credentials are incorrect. Check the bmcsettings at the category level.

  2. Confirm that WebGUI BMC access to a GB200 node is present.

    • Depending on the network configuration, the head node may need to be used as a jump point to reach the GB200 compute tray webGUI.

    • Open a web browser like Firefox and set proxy settings.

    • Enter the BMC webUI through the following URL: https://<BMC_IP>.

  3. Power on one node and watch the boot and provisioning process.

    Power on with BCM:

    cmsh -c "device use <compute node under test>; power on"

    • Alternatively, power on through the BMC webGUI server power control:

    Note

    For the GB200/GB300 Compute trays to reset properly through the “power” command, a delay needs to be set in the partition settings.

    Configure power reset delay:

    cmsh -c "partition; show bmcsettings"

    [a03-p1-head-01->partition[base]->bmcsettings]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Revision
User name                        bright
Password                         ********
User ID                          4
Power reset delay                5s  <-- set this to do : power off + sleep(5) + power on

  4. During the boot up of a GB200/GB300 compute tray.

    • Watch the node installer log to look for any issues during the provisioning.

    tail -f /var/log/node-installer

    • Watch the syslog for any issues/errors

    tail -f /var/log/syslog | grep -i cmd

    • Check cmsh to confirm the nodes are in an UP state.

  5. Once one of the GB200/GB300 compute trays provisions successfully, proceed to power on and provision the rest of the nodes.