Appendix B: Downgrade Firmware for Mellanox ConnectX-4 Cards

DGX OS 5.0 provides the mlnx-fw-updater package version 5.1-2.4.6.0 which automatically installs firmware version 12.28.2040 on ConnectX-4 devices.

Since 12.28.2006 is the recommended firmware version, on December 15 the updater package was updated to install version 12.28.2006. However, if the firmware has already been updated to 12.28.2040, the updater will not install the downlevel firmware version since a newer version is already installed.

In this case, you will need to force the downgrade as explained in this section.

You can use the mlxfwmanager tool to verify whether ConnectX-4 devices are installed on your DGX system.

Run the following command:

            

            
$ sudo mlxfwmanager
Querying Mellanox devices firmware ... Device #1:
Device Type: ConnectX4
Part Number: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE;
single-port QSFP28; PCIe3.0 x16; ROHS R6
PSID: MT_2180110032
PCI Device Name: /dev/mst/mt4115_pciconf1 Base GUID: 248a070300945e60
Versions: Current Available FW 12.28.2040 N/A
PXE 3.6.0102 N/A
UEFI 14.21.0017 N/A
        

If the output indicates that ConnectX-4 devices are installed, you need to downgrade the firmware.

To downgrade the firmware:

1. Determine the correct firmware package name.

2. Switch to the /opt/Mellanox/mlnx-fw-updater/firmware directory, where the updater installs the firmware files, and list the contents.

   Copy

   Copied!

               
   
               
   $ ls /opt/mellanox/mlnx-fw-updater/firmware
           

3. Identify the correct package from the output.

   Copy

   Copied!

               
   
               
   mlxfwmanager_sriov_dis_x86_64_4115
   mlxfwmanager_sriov_dis_x86_64_4119
   mlxfwmanager_sriov_dis_x86_64_4123
   mlxfwmanager_sriov_dis_x86_64_4127
   mlxfwmanager_sriov_dis_x86_64_41686
   mlxfwmanager_sriov_dis_x86_64_4117
   mlxfwmanager_sriov_dis_x86_64_4121
   mlxfwmanager_sriov_dis_x86_64_4125
   mlxfwmanager_sriov_dis_x86_64_41682
           

4. Execute the firmware package by using the -f flag.

   Copy

   Copied!

               
   
               
   $ cd /opt/mellanox/mlnx-fw-updater/firmware
           

   Copy

   Copied!

               
   
               
   $ sudo ./mlxfwmanager_sriov_dis_x86_64_4115 -f
   
   The software queries the current firmware and then updates
   (downgrades) the firmware.
   Querying Mellanox devices firmware …
   Found 2 device(s) requiring firmware update... Device #1: Updating FW
   …
   Initializing image partition - OK Writing Boot image component - OK
   Done
   Device #2: Updating FW …
   Initializing image partition - OK
   Writing Boot image component - OK Done
           

5. Reboot the system to allow the updates to take effect.

   Copy

   Copied!

               
   
               
   $ sudo reboot