Downgrading ConnectX Firmware

DGX OS 5.0 provides the mlnx-fw-updater package version 5.1-2.4.6.0 which automatically installs firmware version 12.28.2040 on ConnectX-4 devices.

Since 12.28.2006 is the recommended firmware version, on December 15 the updater package was updated to install version 12.28.2006. However, if the firmware has already been updated to 12.28.2040, the updater will not install the downlevel firmware version since a newer version is already installed.

In this case, you will need to force the downgrade as explained in this section.

Checking the Device Type

You can use the mlxfwmanager tool to verify whether ConnectX-4 devices are installed on your DGX system.

Run the following command:

sudo mlxfwmanager
Querying Mellanox devices firmware ... Device #1:
Device Type: ConnectX4
Part Number: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE;
single-port QSFP28; PCIe3.0 x16; ROHS R6
PSID: MT_2180110032
PCI Device Name: /dev/mst/mt4115_pciconf1 Base GUID: 248a070300945e60
Versions: Current Available FW 12.28.2040 N/A
PXE 3.6.0102 N/A
UEFI 14.21.0017 N/A

Downgrading the Firmware

If the output indicates that ConnectX-4 devices are installed, you need to downgrade the firmware.

To downgrade the firmware:

  1. Determine the correct firmware package name.

  2. Switch to the /opt/Mellanox/mlnx-fw-updater/firmware directory, where the updater installs the firmware files, and list the contents.

    ls /opt/mellanox/mlnx-fw-updater/firmware
    
  3. Identify the correct package from the output.

    mlxfwmanager_sriov_dis_x86_64_4115
    mlxfwmanager_sriov_dis_x86_64_4119
    mlxfwmanager_sriov_dis_x86_64_4123
    mlxfwmanager_sriov_dis_x86_64_4127
    mlxfwmanager_sriov_dis_x86_64_41686
    mlxfwmanager_sriov_dis_x86_64_4117
    mlxfwmanager_sriov_dis_x86_64_4121
    mlxfwmanager_sriov_dis_x86_64_4125
    mlxfwmanager_sriov_dis_x86_64_41682
    
  4. Execute the firmware package by using the -f flag.

    cd /opt/mellanox/mlnx-fw-updater/firmware
    
    sudo ./mlxfwmanager_sriov_dis_x86_64_4115 -f
    
    The software queries the current firmware and then updates
    (downgrades) the firmware.
    Querying Mellanox devices firmware …
    Found 2 device(s) requiring firmware update... Device #1: Updating FW
    …
    Initializing image partition - OK Writing Boot image component - OK
    Done
    Device #2: Updating FW …
    Initializing image partition - OK
    Writing Boot image component - OK Done
    
  5. Reboot the system to allow the updates to take effect.

    sudo reboot