Network Card Replacement

Network Card Replacement Overview

This is a high-level overview of the procedure to replace one or more network cards on the DGX A100 system.

1. Use the nvsm show commands to identify the failed network card.

2. Get a replacement card from NVIDIA Enterprise Support.

3. Shut down the system.

4. Label all motherboard tray cables and unplug them.

5. Remove the motherboard tray and open the lid.

6. Locate the failed network card and remove it.

7. Insert the new card into the slot and secure with the screw.

8. Close the lid on the motherboard tray, then insert the tray into the system.

9. Plug in all cables using the labels as a reference.

10. Power on the system.

11. Verify that the network card is healthy using nvsm.

Identifying the Failed Network Card

Before attempting to replace any of the network cards, be sure to have performed the following:

1. Issue the following.

   $ sudo nvsm show health
   

2. Match the PCIe bus ID for the failed card with the slot ID using the following diagram.

   

   Note the slot ID for ordering and replacing.

3. Order the appropriate card type as indicated in the following table, then follow the corresponding replacement instructions.

   Network Card Slot IDs

   Slot ID	Card Type	Replacement Instructions
   0-3, 6-9	Single port	Replacing the Vertical Network Card
   4	Dual-port vertical	Replacing the Vertical Network Card
   5	Dual-port horizontal	Replacing the Horizontal Network Card

Replacing the Vertical Network Card

Before attempting to replace any of the network cards, be sure to have performed the following:

• Determined the location ID of the faulty network card needing replacement.

  See Identifying the Failed Network Card.

• Obtained the replacement network card and have saved the packaging for use when returning the faulty component.

Caution

Static Sensitive Devices: - Be sure to observe best practices for electrostatic discharge (ESD) protection. This includes making sure personnel and equipment are connected to a common ground, such as by wearing a wrist strap connected to the chassis ground, and placing components on static-free work surfaces.

1. Power down the system.

2. Label all network, monitor, and USB cables connected to the motherboard tray for easy identification when reconnecting.

3. Unplug all power cords, and all network, monitor, and USB cables.

4. Remove the motherboard tray.

   Refer to the instructions in the section Accessing the Motherboard Tray.

5. Lift the network card off the motherboard and replace with the new network card.

   

6. Install the motherboard tray lid and then install the motherboard tray.

   Refer to the instructions in the section Replacing the Motherboard Tray.

7. Connect all the cables to the motherboard tray, connect all power cords, then power on the system and log in.

8. Confirm that the system is healthy.

   $ sudo nvsm show health
   

   There should be no new alerts listed.

Verify that the firmware is up to date according to the instructions in Updating the Mellanox Network Card Firmware.

Replacing the Horizontal Network Card

Before attempting to replace any of the network cards, be sure to have performed the following:

• Determined the location ID of the faulty network card needing replacement.

  See Identifying the Failed Network Card.

• Obtained the replacement network card have saved the packaging for use when returning the faulty component.

Caution

Static Sensitive Devices: - Be sure to observe best practices for electrostatic discharge (ESD) protection. This includes making sure personnel and equipment are connected to a common ground, such as by wearing a wrist strap connected to the chassis ground, and placing components on static-free work surfaces.

1. Power down the system.

2. Label all network, monitor, and USB cables connected to the motherboard tray for easy identification when reconnecting.

3. Unplug all power cords, and all network, monitor, and USB cables.

4. Remove the motherboard tray.

   Refer to the instructions in the section Accessing the Motherboard Tray.

5. Unlock the horizontal network card.

   1. Loosen the black thumbscrew that secures the PCIe card locking mechanism in place.

      

   2. Open the locking mechanism by turning 90 degrees or more.

      

6. Replace the card.

   1. Pull the network card out of the riser card slot.

      

   2. Replace the old network card with the new one.

      

   3. Install the network card into the riser card slot.

      

7. Lock the network card in place.

   1. Close the locking mechanism by turning it back into its slot.

      

   2. Tighten the black thumb screw to secure the card in place.

      

8. Install the motherboard tray lid and then install the motherboard tray.

   Refer to the instructions in the section Replacing the Motherboard Tray.

9. Connect all cables back into the network card ports.

10. Power on the system and log in.

11. Confirm that the system is healthy.

    $ sudo nvsm show health
    

    There should be no new alerts listed.

Verify that the firmware is up to date according to the instructions in Updating the Mellanox Network Card Firmware.