Configuring and Managing the DGX-1

Using the BMC

The DGX-1 includes a baseboard management controller (BMC) that lets you manage and monitor the DGX-1 independently of the CPU or operating system. You can access the BMC remotely through the Ethernet connection to the IPMI port.

This section describes how to access the BMC, and describes a few common tasks that you can accomplish through the BMC. It is not meant to be a comprehensive description of all the BMC capabilities.

To access the BMC remotely:

  1. Make sure you have connected the IPMI port on the DGX-1 to your LAN.
  2. Open a Java-enabled browser within your LAN and go to http://<IPMI IP Address>/. Use Firefox or Internet Explorer. Google Chrome is not officially supported by the BMC.

  3. Log in. Your initial log in credentials are based on the ones you created when you first set up the DGX-1. Enter your username for both the User ID as well as the Password. User ID: <your username> Password: <your username>.
  4. Be sure to change your password immediately to ensure the security of the BMC. See the next section for instructions on how to change your BMC password.

Creating a Unique BMC Password for Remote Access

When you set up the DGX-1 upon powering it on for the first time, you set up a username and password for the system. These credentials are also used to log in to the BMC remotely, except that the BMC password is the username.

It is strongly recommended that you create a unique password as soon as possible.

Create a unique BMC password as follows:

  1. Open a Java-enabled web browser within your LAN and go to http://<IPMI IP address>/.

    Use Firefox or Internet Explorer. Google Chrome is not officially supported by the BMC.

  2. Log in with the username that you created when you first set up the DGX-1. Enter your username for both the User ID as well as the password: User ID: <your username> Password: <your username>.
  3. From the top menu, click Configuration and then select User.
  4. Select your usename and then click Modify User.
  5. In the Modify User dialog, select Change Password, and then enter your new password in the Password and Confirm Password boxes.
    Note: The BMC software will not accept "sysadmin" for the user name.
  6. Click Modify when finished.

Viewing System Information

The BMC opens to the dashboard, which shows information about the system and system components, such as temperatures and voltages.

Submitting BMC Log Files

The BMC provides automatic logging of system activities and status. The NVIDIA Enterprise Support team uses the log files to assist in troubleshooting. Follow these instructions to obtain the log files to send to NVIDIA Enterprise Support.

  1. Log into the BMC, then click Server Health from the top menu and select Event Log.
  2. Make sure that Text is selected at Format of Download Event Logs.

  3. Click Save Event Logs to download the event logs.

Determining Total Power Consumption

You can use the BMC dashboard to determine total power consumption of the DGX-1 as follows:

  1. Log into the BMC.
  2. From the BMC dashboard, locate the Sensor Monitoring area and then scroll down the page until you see the PSU Input rows.
  3. Add the values for all the PSUs. In this example, the total power consumption would be 216+216+135+27 = 594 watts.

Accessing the DGX-1 Console

  1. Log into the BMC.
  2. From the top menu, click Remote Control and then select Console Redirection.
  3. Click Java Console to open the popup window. The window provides interactive control of the DGX-1 console.

Powering Off / Power Cycling the System Remotely

From the DGX-1 Console Window

If you have opened the Java Viewer (Remote Control->Console Redirection) to view the console window, then you can power cycle, reset, or shutdown the DGX-1 as follows:
  1. From the JViewer top menu, click Power and then select from the available options, depending on what you want to do.

  2. Click Yes and then OK at the Power Control dialog, then wait for the system to perform the intended action.

From the BMC UI

  1. Log into the BMC.
  2. From the top menu, click Remote Control and then select Server Power Control.

  3. Select from the available options according to what you want the system to do, then click Perform Action.

Configuring a Static IP Address for the BMC

This section explains how to set a static IP address for the BMC. You will need to do this if your network does not support DHCP.

Use one of the methods described in the following sections:

Configuring a BMC Static IP Address Using ipmitool

This section describes how to set a static IP address for the BMC from the Ubuntu command line.

Note: If you cannot access the DGX-1 remotely, then connect a display (1024x768 or lower resolution) and keyboard directly to the DGX-1.
To view the current settings, enter the following command.
$ sudo ipmitool lan print 1
Set in Progress     : Set Complete
Auth Type Support    : MD5
Auth Type Enable     : Callback : MD5
                     : User  : MD5
                     : Operator : MD5
                     : Admin : MD5
                     : OEM   : MD5
IP Address Source    : DHCP Address
IP Address           : 10.31.241.190
Subnet Mask          : 255.255.255.0
MAC Address          : 54:ab:3a:72:08:a9
SNMP Community String   : Quanta
IP Header            : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control      : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 0.0 seconds
Default Gateway IP   : 10.31.241.1
Default Gateway MAC  : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC   : 00:00:00:00:00:00
802.1q VLAN ID       : Disabled
802.1q VLAN Priority : 0RMCP+ Cipher Suites  : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : XaaaaaaaaaaaXXX
                     :  X=Cipher Suite Unused
                     :  c=CALLBACK
                     :  u=USER
                     :  o=OPERATOR
                     :  a=ADMIN
                     :  O=OEM
To set a static IP address for the BMC, do the following.
  1. Set the IP address source to static.
    $ sudo ipmitool lan set 1 ipsrc static
  2. Set the appropriate address information.
    • To set the IP address (“Station IP address” in the BIOS settings), enter the following and replace the italicized text with your information.
      $  sudo ipmitool lan set 1 ipaddr  10.31.241.190
    • To set the subnet mask, enter the following and replace the italicized text with your information.
      $  sudo ipmitool lan set 1 netmask  255.255.255.0
    • To set the default gateway IP (“Router IP address” in the BIOS settings), enter the following and replace the italicized text with your information.
      $  sudo ipmitool lan set 1 defgw ipaddr  10.31.241.1

Configuring a BMC Static IP Address Using the System BIOS

This section describes how to set a static IP address for the BMC when you cannot access the DGX-1 remotely. This process involves setting the BMC IP address during system boot.

  1. Connect a keyboard and display (1024x768 or lower resolution) to the DGX-1, then turn on the DGX-1.
  2. When you see the NVIDIA logo, press Del to enter the BIOS Utility Setup Screen.

  3. At the BIOS Setup Utility screen, navigate to the Server Mgmt tab on the top menu, then scroll to BMC network configuration and press Enter.

  4. Scroll to Configuration Address Source and press Enter , then at the Configuration Address source pop-up, select Static on next reset and then press Enter.

  5. Set the addresses for the Station IP address, Subnet mask, and Router IP address as needed by performing the following for each:
    1. Scroll to the specific item and press Enter.
    2. Enter the appropriate information at the pop-up, then press Enter.

  6. When finished making all your changes, press F10 to Save & Reset, then select Yes at the confirmation pop-up and press Enter.

    You can now access the BMC over the network.

Configuring a BMC Static IP Address Using the BMC Dashboard

  1. Log into the BMC, then click Configuration from the top menu and select Network Settings.
  2. In the IPv4 Configuration section of the Network Settings page, clear the Use DHCP check box, and then enter the appropriate values for the IPv4 Address , Subnet Mask , and Default Gateway fields.

  3. Click Save when done.

Configuring Static IP Addresses for the Network Ports

During the initial boot setup process for the DGX-1, you had an opportunity to configure static IP addresses for the network ports. If you did not set this up at that time, you can configure the static IP addresses from the Ubuntu command line according to the following instructions.

Note: If you cannot access the DGX-1 remotely, then connect a display (1024x768 or lower resolution) and keyboard directly to the DGX-1.
  1. Determine the port designation that you want to configure, based on the physical ethernet port that you have connected to your network.

    Use the following port designations according to the DGX-1 Base OS software version installed on the DGX-1:

    Ethernet Port Position Port Designation: Base OS Software 2.x and earlier Port Designation: Base OS Software 3.x and later
    Right Side em1 enp1s0f0
    Left Side em2 enp1s0f1
  2. Edit the interfaces file.
    $ sudo vi /etc/network/interfaces
    ## Configure a static IP
    auto em1
    iface em1 inet static 
      address 192.168.1.14
      gateway 192.168.1.1
      netmask 255.255.255.0
      network 192.168.1.0
      broadcast 192.168.1.255
    Consult your network adiminstrator for the appropriate addresses for your network, and use the port designations that you determined in step 1.
  3. When finished with your edits, press ESC to switch to command mode, then save the file to the disk and exit the editor.
    :wq
  4. Restart the network services to put the changes into effect.
    $ sudo /etc/init.d/networking restart

Obtaining MAC Addresses

These instructions explain how to determine the MAC addresses for the IPMI port (BMC) as well as both ethernet ports of the DGX-1.

The ports are, from left to right, IPMI (BMC), em2 (or enp1s0f1), em1 (or enp1s0f0).

  1. Connect a display (1024x768 or lower resolution) and keyboard to the DGX-1.
  2. Turn the DGX-1 on or reboot.
  3. At the NVIDIA logo boot screen, press [F2] or [Del] to enter the BIOS setup screen.

  4. Select the Advanced tab from the top menu, then scroll down to view the two Quanta Dual Port 10G BASE-T Mezzanine items.

    The first item shows the MAC address for ethernet port em1, and the second item shows the MAC address for em2.
  5. Navigate to and select Server Mgmt from the top menu, then scroll down to and select BMC network configuration.

  6. Scroll down to view the Station MAC address.

    This shows the MAC address for the BMC.