Bluefield and CX7 Firmware Update Process#
Prior to installation, copy the binary to the compute tray host or use a shared directory. The binary naming format should look like:
fw-ConnectX7-rel-28_42_1270-900-24768-0002_Ax-UEFI-14.35.15-FlexBoot-3.7.500.signed.bin
The general steps to install NVIDIA Networking firmware are as follows:
Start the MST service
mst start
Query the devices to find the /dev/mst paths of the devices
mst status -v
Read the current version of firmware on a given device
flint -d /dev/mst/mt4129_pciconf0 q full
Flash the firmware on the device
Change to the directory where the binary is stored
Run the flash command:
flint -d /dev/mst/mt4129_pciconf0 -i fw-ConnectX7-rel-28_42_1270-900-24768-0002_Ax-UEFI-14.35.15-FlexBoot-3.7.500.signed.bin b
Repeat this for all 4 CX7 devices
Reset the CX7 and reboot the host
mlxfwreset -d mlx5_0 reset PCI devices: ------------------ DEVICE_TYPE MST PCI RDMA NET NUMA ConnectX7(rev:0) /dev/mst/mt4129_pciconf0 0000:03:00.0 mlx5_0 net-ibp3s0 0 ConnectX7(rev:0) /dev/mst/mt4129_pciconf1 0002:03:00.0 mlx5_1 net-ibP2p3s0 0 ConnectX7(rev:0) /dev/mst/mt4129_pciconf2 0010:03:00.0 mlx5_4 net-ibP16p3s0 1 ConnectX7(rev:0) /dev/mst/mt4129_pciconf3 0012:03:00.0 mlx5_5 net-ibP18p3s0 1
For Bluefield 3, the process is the same with the exception of the device being
/dev/mst/mt41692
.
Combined CX-7 and Bluefield Update#
Run the update script across all GB200 nodes:
pdsh -g category=dgx-gb200 '/home/nvis/(dir where the firmware update is)/nicupdate.sh > /home/nvis/(dir where the firmware update is)/$(hostname)_fw_upgrade_$(date +'%Y%m%d-%H%M%S').log'
Reference Script: NIC Updates (Both BF3 and CX-7) - nicupdate.sh
# CX-7 Update
mst start
flint -d /dev/mst/mt4129_pciconf0 q full
flint -d /dev/mst/mt4129_pciconf1 q full
flint -d /dev/mst/mt4129_pciconf2 q full
flint -d /dev/mst/mt4129_pciconf3 q full
# BF3 Update
flint -d /dev/mst/mt41692_pciconf0 q full
flint -d /dev/mst/mt41692_pciconf1 q full
basedir=/cm/shared/nicfw
bf3file=fw-BlueField-3-rel-32_44_1600-900-9D3B6-00CN-P_Ax-NVME-20.4.1-UEFI-21.4.13-UEFI-22.4.14-UEFI-14.37.14-FlexBoot-3.7.500.signed.bin
cx7file=fw-ConnectX7-rel-28_44_2524-900-24768-0002_Ax-UEFI-14.37.14-FlexBoot-3.7.500.signed.reduced.bin
yes | flint -d /dev/mst/mt4129_pciconf0 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt4129_pciconf1 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt4129_pciconf2 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt4129_pciconf3 -i $basedir/$cx7file b
yes | flint -d /dev/mst/mt41692_pciconf0 -i $basedir/$bf3file b
yes | flint -d /dev/mst/mt41692_pciconf1 -i $basedir/$bf3file b