Bluefield and CX7 Firmware Update Process#

Prior to installation, copy the binary to the compute tray host or use a shared directory. The binary naming format should look like:

fw-ConnectX7-rel-28_42_1270-900-24768-0002_Ax-UEFI-14.35.15-FlexBoot-3.7.500.signed.bin

The general steps to install NVIDIA Networking firmware are as follows:

  1. Start the MST service

    mst start
    
  2. Query the devices to find the /dev/mst paths of the devices

    mst status -v
    
  3. Read the current version of firmware on a given device

    flint -d /dev/mst/mt4129_pciconf0 q full
    
  4. Flash the firmware on the device

    1. Change to the directory where the binary is stored

    2. Run the flash command:

      flint -d /dev/mst/mt4129_pciconf0 -i fw-ConnectX7-rel-28_42_1270-900-24768-0002_Ax-UEFI-14.35.15-FlexBoot-3.7.500.signed.bin b
      
    3. Repeat this for all 4 CX7 devices

  5. Reset the CX7 and reboot the host

    mlxfwreset -d mlx5_0 reset
    
    PCI devices:
    ------------------
    DEVICE_TYPE      MST                      PCI          RDMA   NET NUMA
    ConnectX7(rev:0) /dev/mst/mt4129_pciconf0 0000:03:00.0 mlx5_0 net-ibp3s0 0
    ConnectX7(rev:0) /dev/mst/mt4129_pciconf1 0002:03:00.0 mlx5_1 net-ibP2p3s0 0
    ConnectX7(rev:0) /dev/mst/mt4129_pciconf2 0010:03:00.0 mlx5_4 net-ibP16p3s0 1
    ConnectX7(rev:0) /dev/mst/mt4129_pciconf3 0012:03:00.0 mlx5_5 net-ibP18p3s0 1
    
  6. For Bluefield 3, the process is the same with the exception of the device being /dev/mst/mt41692.

Combined CX-7 and Bluefield Update#

Run the update script across all GB200 nodes:

pdsh -g category=dgx-gb200 '/home/nvis/(dir where the firmware update is)/nicupdate.sh > /home/nvis/(dir where the firmware update is)/$(hostname)_fw_upgrade_$(date +'%Y%m%d-%H%M%S').log'

Reference Script: NIC Updates (Both BF3 and CX-7) - nicupdate.sh

# CX-7 Update
mst start

flint -d /dev/mst/mt4129_pciconf0 q full
flint -d /dev/mst/mt4129_pciconf1 q full
flint -d /dev/mst/mt4129_pciconf2 q full
flint -d /dev/mst/mt4129_pciconf3 q full

# BF3 Update
flint -d /dev/mst/mt41692_pciconf0 q full
flint -d /dev/mst/mt41692_pciconf1 q full

basedir=/cm/shared/nicfw
bf3file=fw-BlueField-3-rel-32_44_1600-900-9D3B6-00CN-P_Ax-NVME-20.4.1-UEFI-21.4.13-UEFI-22.4.14-UEFI-14.37.14-FlexBoot-3.7.500.signed.bin
cx7file=fw-ConnectX7-rel-28_44_2524-900-24768-0002_Ax-UEFI-14.37.14-FlexBoot-3.7.500.signed.reduced.bin

yes | flint -d /dev/mst/mt4129_pciconf0 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt4129_pciconf1 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt4129_pciconf2 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt4129_pciconf3 -i $basedir/$cx7file b

yes | flint -d /dev/mst/mt41692_pciconf0 -i $basedir/$bf3file b
yes | flint -d /dev/mst/mt41692_pciconf1 -i $basedir/$bf3file b