NVIDIA BlueField Reset and Reboot Procedures
This section describes the necessary operations to load new NIC firmware, following NVIDIA® BlueField® NIC firmware update. This procedure deprecates the need for full server power cycle.
The following steps are executed in the BlueField OS:
Issue a query command to ascertain whether BlueField system reboot is supported by your environment:
mlxfwreset -d
03:00.0qIf the output includes the following lines, proceed to step 2:
3: Driver restart and PCI reset -Supported (default) ...1: Driver is the owner -Supported (default)NoteIf it says
Not Supportedinstead, then proceed to the instructions under section "BlueField System-level Reset".Issue a BlueField system reboot:
mlxfwreset -d
03:00.0-y -l3--sync1r
This section describes the way to perform system-level reset (SLR) which is necessary for firmware configuration changes to take effect.
SLR for BlueField running in DPU mode on hosts with separate power control (special use case)
System-level Reset for BlueField in DPU Mode
The following is the high-level flow of the procedure:
Graceful shutdown of BlueField Arm cores.
Query BlueField state to affirm shutdown reached.
InfoIn systems with multiple BlueField networking platforms, repeat steps 1 and 2 for all devices before proceeding.
Warm reboot the server.
Step by step process:
Some of the following steps can be performed using different methods, depending on resource availability and support in the user's environment.
Graceful shutdown of BlueField Arm cores.
InfoThis operation is expected to finish within 15 seconds.
Possible methods:
From the BlueField OS:
shutdown -h now
Or:
mlxfwreset -d /dev/mst/mt*pciconf0 -l
1-t4--sync0rFrom the host OS:
InfoNot relevant when the BlueField is operating in Zero-Trust Mode.
mlxfwreset -d <mst-device> -l
1-t4rUsing the BlueField BMC:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> power softOr using Redfish (BlueField-3 and above):
curl -k -u root:<password> -H
"Content-Type: application/json"-X POST https://<bmc_ip>/redfish/v1/Systems/Bluefield/Actions/ComputerSystem.Reset -d '{"ResetType": "GracefulShutdown"}'
Query BlueField state.
Possible methods:
From the host OS:
InfoNot relevant when the BlueField is operating in Zero-Trust Mode.
echo DISPLAY_LEVEL
2> /dev/rshim0/misc cat /dev/rshim0/miscExpected output:
INFO[BL31]: System Off
Utilizing the BlueField BMC:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> raw0x320xA3Expected output:
06.
Warm reboot the server f rom the host OS:
mlxfwreset -d <mst-device> -l
4rNoteIf multiple BlueField devices are present in the host, this command must run only once. In this case, the MST device can be of any of the BlueFields for which the reset is necessary and participated in step 1.
Or:
reboot
NoteFor external hosts which do not toggle PERST# in their standard reboot command, use the
mlxfwresetoption.
System-level Reset for BlueField in NIC Mode
Perform warm reboot of the host OS:
mlxfwreset -d <mst-device> -l 4 r
Or:
reboot
For external hosts which do not toggle PERST# in their standard reboot command, use the mlxfwreset option.
System-level Reset for Host with Separate Power Control
This procedure is a special use case relevant only to host platforms with separate power control for the PCIe slot and CPUs, in which the BlueField (running in DPU mode) is provided power while host OS/CPUs may be in shutdown or similar standby state (this allows the BlueField device to be operational while the host CPU is in shutdown/standby state).
The following is the high-level flow of the procedure:
Graceful shutdown of host OS or similar CPU standby.
Graceful shutdown of BlueField Arm cores.
Query BlueField state to affirm shutdown reached.
Full BlueField Reset
Query BlueField state to affirm operational state reached
InfoIn systems with multiple BlueField networking platforms, repeat steps 1 through 5 for all devices before proceeding.
Power on the server.
Step by step process:
Some of the following steps can be performed using different methods, depending on resource availability and support in the user's environment.
Graceful shutdown of host OS by any means preferable.
Graceful shutdown of BlueField Arm cores.
InfoThis step normally takes up to 15 seconds to complete.
From the BlueField OS:
shutdown -h now
Utilizing the BlueField BMC:
Using IPMI:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> power softUsing Redfish (for BlueField-3 and above):
curl -k -u root:<password> -H
"Content-Type: application/json"-X POST https://<bmc_ip>/redfish/v1/Systems/Bluefield/Actions/ComputerSystem.Reset -d '{"ResetType": "GracefulShutdown"}'
Query the BlueField's state utilizing the BlueField BMC:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> raw0x320xA3Expected output:
06.Perform BlueField hard reset utilizing the BlueField BMC:
InfoThis step takes up to 2 minutes to complete .
Using IPMI:
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> power cycleUsing Redfish (for BlueField-3 and above):
curl -k -u root:<password> -H
"Content-Type: application/json"-X POST https://<bmc_ip>/redfish/v1/Systems/Bluefield/Actions/ComputerSystem.Reset -d '{"ResetType" : "PowerCycle"}'
Query BlueField operational state u tilizing the BlueField BMC :
InfoAt this point, the BlueField is expected to b e operational .
ipmitool -C
17-I lanplus -H <bmc_ip> -U root -P <password> raw0x320xA3Expected output:
05.Power on/boot up the host OS.
The BlueField DPU can also be reset from a BMC platform using NC-SI command over I2C. This option is more common in DPU BMC absence, when the BlueField DPU is running in NIC mode or when it is used as a controller.
The reset is performed using the Reset BlueField-3 DPU (Command=0x12, Parameter=0xB), which allows a BMC platform to reset the NVIDIA BlueField-3 DPU device. This command is only applicable to BlueField-3 devices.
The Reset BlueField-3 DPU command is addressed to the package only. When the internal reset is complete, the BMC platform should reconfigure the device.
The Reset BlueField-3 DPU command is supported on BlueField-2 and later devices.
Reset BlueField-3 DPU Format
Bytes/Bits | 31:24 | 23:16 | 15:8 | 7:0 | |
0:15 | NC-SI Header (OEM Command) | ||||
16:19 | NVIDIA Manufacture ID (IANA) = 0x8119 | ||||
20:23 | Command rev=0x00 | MLNX Cmd ID=0x12 | Parameter=0x0B | NICR | Mode |
24:27 | Checksum 31:0 | ||||
The parameter descriptions for Reset BlueField-3 DPU command are provided below.
Reset BlueField-3 DPU Parameters
Field | Description |
NICR |
|
Mode | This field defines the type of conditions to use before performing the internal reset
|
Reset BlueField-3 DPU Response
The ConnectX adapter responds to a Reset BlueField-3 DPU command when the package ID matches, and with no checksum error.
Reset BlueField-3 DPU Response Format
Bytes/Bits | 31:24 | 23:16 | 15:8 | 7:0 | |
0:15 | NC-SI Header (OEM Command) | ||||
16:19 | Response Code | Reason Code | |||
20:23 | NVIDIA Manufacture ID (IANA) = 0x8119 | ||||
24:27 | Command rev=0x00 | MLNX Cmd ID=0x12 | Parameter=0x0B | NICR | Mode |
28:31 | Checksum 31:0 | ||||