BlueField-3 I/O Card Replacement#

This topic describes how to replace the NVIDIA® BlueField®-3 card in the NVIDIA DGX™ B200 system.

BlueField-3 I/O Card Replacement Overview#

  1. Identify the failed BlueField-3 I/O card.

  2. Get a replacement BlueField-3 I/O card from NVIDIA Enterprise Support.

  3. Ensure that the system is shut down.

  4. If cables do not reach, label all cables and unplug them from the motherboard tray.

  5. Slide the motherboard out until it locks in place.

  6. Open the rear compartment.

  7. Pull the card directly above the failed Bluefield-3 card to make room for the procedure.

  8. Pull out the BlueField-3 I/O card.

  9. Remove the IPEX and power cables from the old card.

  10. Install the IPEX and power cables to the new card.

  11. Install the new BlueField-3 I/O card.

  12. Install the card that goes over the BlueField-3 card.

  13. Close the rear motherboard compartment.

  14. Slide the motherboard back into the system.

  15. Plug in all cables using the labels as a reference.

  16. Power on the system.

  17. Update the firmware if necessary and test the BlueField-3 I/O card.

  18. Send the failed unit to NVIDIA Enterprise Support using the packaging provided.

Prepare the System for Replacement#

  1. Identify which I/O card to replace.

    Use the nvsm command or network tools to determine the failed card, and then contact NVIDIA Enterprise Support to get a replacement card.

  2. When you receive the replacement card, power off the system.

    To access the rear section of the motherboard, refer to Motherboard Tray - Opening and Closing the I/O Door.

  3. Based on the output from the nvsm command, identify which card needs to be replaced, the card in slot 1 or slot 2.

    _images/case-rear.png

Remove the I/O Card above the BlueField-3 Card to be Replaced#

  1. Pull out the motherboard tray and access the I/O door. Refer to Motherboard Tray - Opening and Closing the I/O Door for information about accessing the IO door.

  2. Remove the I/O card that is above the BlueField-3 card. The card can be the M.2 boot drive assembly or a network interface card.

    • Refer to M.2 Boot Drive Assembly Replacement to remove the M.2 boot drive carrier.

      The images at the preceding link show how to remove the boot drive carrier on the right above the BlueField-3 card in slot 2. If you need to replace the BlueField-3 card in slot 1, follow the instructions but use the thumbscrew on the left side of the motherboard tray.

    • Refer to Network Interface Card Replacement to remove the Ethernet NIC.

Remove the BlueField-3 Card#

  1. Remove the power cable from the BlueField-3 card side only:

    Do not unplug the power cable from the motherboard side.

    _images/dgx-b200-bf3-remove-cable.png

  2. Before pulling the card too far, ensure to unplug the white and black IPEX cables from the card following the instructions in Remove an IPEX Cable.

    The white cable connects to the top of the card and the black cable connects to the bottom (heatsink) of the card:

    _images/dgx-b200-bf3-ipex.png

  3. Pull the card out of the slot and remove it from the system:

    _images/dgx-b200-bf3-remove-card.png

Remove an IPEX Cable#

Repeat this procedure for both the white and black cables.

The following image shows the IPEX cable attached to the connector:

_images/ipex-cable-8.png

  1. Lift the locking door:

    _images/ipex-cable-2.png

  2. Push the cable away from the connector:

    _images/ipex-cable-3.png

Install the BlueField-3 Card#

  1. After you connect the IPEX cables, install the new BlueField-3 card in the bottom slot in the PCI riser:

    _images/dgx-b200-bf3-card-new.png

  2. Attach the IPEX and power cables as shown in Insert an IPEX Cable.

    _images/dgx-b200-bf3-cables-connected.png

    Connect one end of the power cable to the BlueField-3 card and the other end of the power cable to the motherboard.

  3. Insert the BlueField-3 card in the bottom PCI slot:

    _images/dgx-b200-bf3-card-installed.png

Install the I/O Card and Close the System#

  1. Reinstall the I/O card that is above the BlueField-3 card. Refer to one of the two following procedures:

  2. Close the motherboard tray I/O door and insert the motherboard tray. Refer to Motherboard Tray - Opening and Closing the I/O Door for more information.

Power on the System and Confirm the Replacement#

  1. Power on and boot the system.

  2. Update the firmware on the card.

    Refer to the NVIDIA BlueField-3 Networking Platform User Guide.

  3. Reset the Bluefield-3 password.

    Refer to Changing Default Password in the NVIDIA BlueField documentation.

  4. To change passwords other than the one from the BMC user, refer to Changing BMC Password Using bf.cfg in the NVIDIA BlueField documentation.

  5. Confirm that the system is working correctly by using the nvsm command:

    sudo nvsm show health

  6. Use the packaging from the new card to send the failed card to NVIDIA Enterprise Support.