NVIDIA BlueField Reset and Reboot Procedures
Contents:
This section describes the necessary operations to load new NIC firmware, following NVIDIA® BlueField® NIC firmware update. This procedure deprecates the need for full server power cycle.
The following steps are executed in the BlueField OS:
Issue a query command to ascertain whether BlueField system reboot is supported by your environment:
mlxfwreset -d
03:00.0qIf the output includes the following lines, proceed to step 2:
3: Driver restart and PCI reset -Supported (default) ...1: Driver is the owner -Supported (default)NoteIf it says Not Supported instead, then proceed to the instructions under section "BlueField System-level Reset".
Issue a BlueField system reboot:
mlxfwreset -d
03:00.0-y -l3--sync1r
This section describes the way to perform system-level reset (SLR) which is necessary for firmware configuration changes to take effect.
SLR for BlueField running in DPU mode on hosts with separate power control (special use case)
System-level Reset for BlueField in DPU Mode
The following is the high-level flow of the procedure:
Graceful shutdown of BlueField Arm cores.
Query BlueField state to affirm shutdown reached.
InfoIn systems with multiple BlueField networking platforms, repeat steps 1 and 2 for all devices before proceeding.
Warm reboot the server.
Step by step process:
Some of the following steps can be performed using different methods, depending on resource availability and support in the user's environment.
Graceful shutdown of BlueField Arm cores.
InfoThis operation is expected to finish within 15 seconds.
Possible methods:
From the BlueField OS:
shutdown -h now
Or:
mlxfwreset -d /dev/mst/mt*pciconf0 -l
1-t4--sync0rFrom the host OS:
InfoNot relevant when the BlueField is operating in Zero-Trust Mode.
mlxfwreset -d <mst-device> -l
1-t4rUsing the BlueField BMC:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> power softOr using Redfish (BlueField-3 and above):
curl -k -u root:<password> -H
"Content-Type: application/json"-X POST https://<bmc_ip>/redfish/v1/Systems/Bluefield/Actions/ComputerSystem.Reset -d '{"ResetType": "GracefulShutdown"}'
Query BlueField state.
Possible methods:
From the host OS:
InfoNot relevant when the BlueField is operating in Zero-Trust Mode.
echo DISPLAY_LEVEL
2> /dev/rshim0/misc cat /dev/rshim0/miscExpected output:
INFO[BL31]: System Off
Utilizing the BlueField BMC:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> raw0x320xA3Expected output: 06.
Warm reboot the server f rom the host OS:
mlxfwreset -d <mst-device> -l
4rNoteIf multiple BlueField devices are present in the host, this command must run only once. In this case, the MST device can be of any of the BlueFields for which the reset is necessary and participated in step 1.
Or:
reboot
NoteFor external hosts which do not toggle PERST# in their standard reboot command, use the mlxfwreset option.
System-level Reset for BlueField in NIC Mode
Perform warm reboot of the host OS:
mlxfwreset -d <mst-device> -l 4 r
Or:
reboot
For external hosts which do not toggle PERST# in their standard reboot command, use the mlxfwreset option.
System-level Reset for Host with Separate Power Control
This procedure is a special use case relevant only to host platforms with separate power control for the PCIe slot and CPUs, in which the BlueField (running in DPU mode) is provided power while host OS/CPUs may be in shutdown or similar standby state (this allows the BlueField device to be operational while the host CPU is in shutdown/standby state).
The following is the high-level flow of the procedure:
Graceful shutdown of host OS or similar CPU standby.
Graceful shutdown of BlueField Arm cores.
Query BlueField state to affirm shutdown reached.
Full BlueField Reset
Query BlueField state to affirm operational state reached
InfoIn systems with multiple BlueField networking platforms, repeat steps 1 through 5 for all devices before proceeding.
Power on the server.
Step by step process:
Some of the following steps can be performed using different methods, depending on resource availability and support in the user's environment.
Graceful shutdown of host OS by any means preferable.
Graceful shutdown of BlueField Arm cores.
InfoThis step normally takes up to 15 seconds to complete.
From the BlueField OS:
shutdown -h now
Utilizing the BlueField BMC:
Using IPMI:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> power softUsing Redfish (for BlueField-3 and above):
curl -k -u root:<password> -H
"Content-Type: application/json"-X POST https://<bmc_ip>/redfish/v1/Systems/Bluefield/Actions/ComputerSystem.Reset -d '{"ResetType": "GracefulShutdown"}'
Query the BlueField's state utilizing the BlueField BMC:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> raw0x320xA3Expected output: 06.
Perform BlueField hard reset utilizing the BlueField BMC:
InfoThis step takes up to 2 minutes to complete .
Using IPMI:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> power cycleUsing Redfish (for BlueField-3 and above):
curl -k -u root:<password> -H
"Content-Type: application/json"-X POST https://<bmc_ip>/redfish/v1/Systems/Bluefield/Actions/ComputerSystem.Reset -d '{"ResetType" : "PowerCycle"}'
Query BlueField operational state u tilizing the BlueField BMC :
InfoAt this point, the BlueField is expected to b e operational .
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> raw0x320xA3Expected output: 05.
Power on/boot up the host OS.