ConnectX-5 Card Replacement

ConnectX-5 Card Replacement Overview

This is a high-level overview of the procedure to replace one or more Mellanox ConnectX-5 cards on the DGX-2 System.
  1. Use the nvsm show commands to identify the failed ConnectX-5 card.
  2. Get a replacement ConnectX-5 card from NVIDIA Enterprise Support.
  3. Shut down the system.
  4. Label all I/O tray cables and unplug them.
  5. Remove the I/O tray and open the lid.
  6. Locate the failed ConnectX-5 card, then remove the screw that attaches the card and remove the card.
  7. Insert the new card into the slot and secure with the screw.
  8. Close the lid on the I/O tray, then insert the tray into the system.
  9. Plug in all cables using the labels as a reference.
  10. Power on the system.
  11. Verify that the ConnectX-5 card is healthy using nvsm.

Replacing the ConnectX-5 Card

Before attempting to replace any of the ConnectX-5 cards, be sure to have performed the following:
  • Determined the location ID of the faulty ConnectX-5 card needing replacement.

    Run nvsm show health to identify the bad card. Note the PCIe bus ID and slot number.

  • Obtained the replacement ConnectX-5 card have saved the packaging for use when returning the faulty component.

CAUTION: Static Sensitive Devices: - Be sure to observe best practices for electrostatic discharge (ESD) protection. This includes making sure personnel and equipment are connected to a common ground, such as by wearing a wrist strap connected to the chassis ground, and placing components on static-free work surfaces.

  1. Power down the system..
  2. Label all the network ports (0-7) connected to the I/O tray for easy identification when reconnecting.

  3. Remove the cables.
  4. Remove the I/O tray.
    1. Loosen the two green I/O tray screws with a Philips #2 screwdriver and pull the levers outward to release the tray.

    2. Pull the I/O tray out of the system and place it on a solid, flat work surface.
      CAUTION:
      Exercise care when removing the tray as it is long and heavy, and do not handle the module from the rear connectors.
  5. Remove the I/O tray lid.
    1. Loosen the black screws and then push the lid towards you to release the lid.

    2. Lift the lid upward.

  6. Do the following for each card that needs to be replaced.
    1. To assist in locating the card to remove, refer to the service label that maps the PCIe bus ID to the slot number.
    2. Remove the screw that secures the card, then pull the card out of the slot.

    3. Insert the replacement card and secure with the screw removed from the previous step
  7. Install the I/O tray.
    1. Replace the I/O tray lid by placing it over the module using the guiding pins and grooves.

    2. Slide the lid back so that the black screws enage with the tray, then tighten the black screws to secure the lid.

    3. Push the I/O tray back into the system.

    4. Close the levers toward the center, making sure the connectors engage with the midplane, then tighten the thumbscrews by hand or with a Phillips #2 screwdriver.

  8. Connect all cables back into the ConnectX-5 card ports.
  9. Power on the system and log in.
  10. Confirm that the system is healthy.
    $ sudo nvsm show health
    There should be no new alerts listed.

Verifying the ConnectX-5 Cards

This section describes the steps needed to verify that the ConnectX-5 cards has been replaced correctly.

  1. With the DGX-2 turned on, verify that the card was installed correctly and is recognized by the system. 
    $ lspci | grep -i mellanox
    The output should show all installed Mellanox cards, including the dual port (and optional dual port) cards.

    Example

     35:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 3a:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 58:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 5d:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 86:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 86:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 b8:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 bd:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 e1:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
 e6:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
    The dual port cards are identified by bus ID 86. Look for all other cards. If eight cards (those other than bus ID 86) are not reported, then the card was not installed properly and should be reseated. If a card other than the officially supported Mellanox family of adapters appears, contact NVIDIA Enterprise Support.
  2. Verify the firmware version.
    $ cat /sys/class/infiniband/mlx5*/fw_ver
    Example output: 
     12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
    The latest InfiniBand firmware version supported for each DGX OS Server release is as follows:
    • Release 4.x: Firmware version 12.23.1020
  3. If you need to update the firmware, follow these steps:
    1. Initiate the firmware update.
      $ sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl
      The script will check the firmware version of each card and update where needed. If the firmware is updated for any card, you will need to reboot the system for the changes to take effect.
    2. Reboot the system if instructed.
    3. After rebooting the system, verify that all the Mellanox InfiniBand cards are using the current firmware.
      $ cat /sys/class/infiniband/mlx5*/fw_ver
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
 12.23.1020
  4. Verify the physical port state for the InfiniBand cards. 
    $ ibstat
    In the output text, verify that the Physical State for each card with a cable connection is LinkUp and that the port for the card is configured with a GUID. The following example output shows one card in a non-connected state, and the remaining cards in a connected state. Relevant text is highlighted in bold. 
    CA 'mlx5_0'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x248a0703000de288
 System image GUID: 0x248a0703000de288
 Port 1:
       State: Down
       Physical state: Polling
       Rate: 10
       Base lid: 65535
       LMC: 0
       SM lid: 0
       Capability mask: 0x2651e848
       Port GUID: 0x248a0703000de288
       Link layer: InfiniBand
CA 'mlx5_1'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x248a0703000de26c
 System image GUID: 0x248a0703000de26c
 Port 1:
        State: Initializing
        Physical state: LinkUp
        Rate: 100
        Base lid: 65535
        LMC: 0
        SM lid: 0
        Capability mask: 0x2651e848
        Port GUID: 0x248a0703000de26c
        Link layer: InfiniBand
CA 'mlx5_2'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x248a0703001effde
 System image GUID: 0x248a0703001effde
 Port 1:
        State: Initializing
        Physical state: LinkUp
        Rate: 100
        Base lid: 65535
        LMC: 0
        SM lid: 0
        Capability mask: 0x2651e848
        Port GUID: 0x248a0703001effde
        Link layer: InfiniBand
CA 'mlx5_3'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x7cfe900300118f22
 System image GUID: 0x7cfe900300118f22
 Port 1:
         State: Initializing
         Physical state: LinkUp
         Rate: 100
         Base lid: 65535
         LMC: 0
         SM lid: 0
         Capability mask: 0x2651e848
         Port GUID: 0x7cfe900300118f22
         Link layer: InfiniBand
CA 'mlx5_4'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x7cfe900300118f26
 System image GUID: 0x7cfe900300118f26
 Port 1:
         State: Initializing
         Physical state: LinkUp
         Rate: 100
         Base lid: 65535
         LMC: 0
         SM lid: 0
         Capability mask: 0x2651e848
         Port GUID: 0x7cfe900300118f26
         Link layer: InfiniBand
CA 'mlx5_5'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x7cfe900300118f25
 System image GUID: 0x7cfe900300118f25
 Port 1:
         State: Initializing
         Physical state: LinkUp
         Rate: 100
         Base lid: 65535
         LMC: 0
         SM lid: 0
         Capability mask: 0x2651e848
         Port GUID: 0x7cfe900300118f25
         Link layer: InfiniBand
CA 'mlx5_6'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x7cfe900300118f24
 System image GUID: 0x7cfe900300118f24
 Port 1:
         State: Initializing
         Physical state: LinkUp
         Rate: 100
         Base lid: 65535
         LMC: 0
         SM lid: 0
         Capability mask: 0x2651e848
         Port GUID: 0x7cfe900300118f24
         Link layer: InfiniBand
CA 'mlx5_7'
 CA type: MT4119
 Number of ports: 1
 Firmware version: 12.23.1020
 Hardware version: 0
 Node GUID: 0x7cfe900300118f23
 System image GUID: 0x7cfe900300118f23
 Port 1:
         State: Initializing
         Physical state: LinkUp
         Rate: 100
         Base lid: 65535
         LMC: 0
         SM lid: 0
         Capability mask: 0x2651e848
         Port GUID: 0x7cfe900300118f23
         Link layer: InfiniBand
See the Switching Between InfiniBand and Ethernet section of the NVIDIA DGX-2 User Guide for instructions on switching the port to InfiniBand or Ethernet, if required.