CEC and BMC Firmware Operations
HttpPushUri API is obsolete. NVIDIA recommends users migrate to MultipartHttpPushUri API as support for HttpPushUri API will be discontinued in the future.
Firmware upgrade of BMC and CEC components using BMC can be performed from a remote server using the Redfish interface.
Trigger Secure Firmware Update
Required for BMC/CEC update.
Triggers the secure update and starts tracking the secure update progress.
Standard update with HttpPushUri API:
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService/update
Multipart update with MultipartHttpPushUri API:
curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart --form UpdateFile=@<package_path>;type=application/octet-stream --form 'UpdateParameters={}'
Where:
password – password of root user
bmc_ip – BMC IP address
package_path – firmware update package path
Track Secure Firmware Update Progress
Required for BMC/CEC update.
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks
Find the current task ID in the response and use it for checking the progress:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<task_id> | jq -r ' .PercentComplete'
Where:
password – password of root user
bmc_ip – BMC IP address
task_id – Task ID
Reset/Reboot BMC
Required for BMC update.
curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.Reset
Where:
password – password of root user
bmc_ip – BMC IP address
Fetch Running BMC Firmware Version
Required for BMC update.
Fetches the running firmware version from the BMC.
For BlueField-3:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware | jq -r ' .Version'
Where:
password – password of root user
bmc_ip – BMC IP address
For BlueField-2:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory
Fetch the current firmware ID and then perform:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/<firmware_id>_BMC_Firmware | jq -r ' .Version'
Where:
password – password of root user
bmc_ip – BMC IP address
firmware_id – numeric value found in the FwInventory schema only. It is calculated during firmware update by the BMC and used to distinguish between the versions.
Fetch Running CEC Firmware Version
Required for CEC update.
Fetches the running firmware version from CEC.
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/Bluefield_FW_ERoT | jq -r ' .Version'
Where:
password – password of root user
bmc_ip – BMC IP address
Force Update
Required for BMC update.
Relevant for BlueField-3 only.
Forces the update procedure to proceed even if the target version is identical.
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/UpdateService -d '{"HttpPushUriOptions": {"ForceUpdate": true}}'
Where:
password – password of root user
bmc_ip – BMC IP address
Firmware update takes about 12 minutes.
After initiating the BMC secure update with the command #1 from the previous table, a response similar to the following is received depending on whether HttpPushUri or MultipartHttpPushUri is used:
HttpPushUri API:
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService { "@odata.id": "/redfish/v1/TaskService/Tasks/0", "@odata.type": "#Task.v1_4_3.Task", "Id": "<id>", "TaskState": "Running" }
MultipartHttpPushUri API:
curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart --form "UpdateFile=@<package_path>;type=application/octet-stream" --form 'UpdateParameters={}' { "@odata.id": "/redfish/v1/TaskService/Tasks/0", "@odata.type": "#Task.v1_4_3.Task", "Id": "<id>", "TaskState": "Running", "TaskStatus": "OK" }
Where:
package_path – the BMC firmware image file including its path
For example:
curl -k -u root:
'myP@ssword_12345!'https://10.10.10.10/redfish/v1/UpdateService/update-multipart --form UpdateFile=@/root/bf2-bmc-ota-24.01-4-ipn.tar;type=application/octet-stream --form 'UpdateParameters={}'
Command #2 from the previous table can be used to track secure firmware update progress. For instance:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<id> | jq -r ' .PercentComplete'
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 2123 100 2123 0 0 38600 0 --:--:-- --:--:-- --:--:-- 37910
20
Command #2 is used to verify the task has completed because during the update procedure the reboot option is disabled. When PercentComplete reaches 100, command #3 is used to reboot the BMC. For example:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<id> | jq -r ' .PercentComplete'
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 3822 100 3822 0 0 81319 0 --:--:-- --:--:-- --:--:-- 81319
100
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.Reset
{
"@Message.ExtendedInfo": [
{
"@odata.type": "#Message.v1_1_1.Message",
"Message": "The request completed successfully.",
"MessageArgs": [],
"MessageId": "Base.1.13.0.Success",
"MessageSeverity": "OK",
"Resolution": "None"
}
]
}
Command #4 can be used to verify the current BMC firmware version after reboot:
For BlueField-3:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware | jq -r ' .Version' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 513 100 513 0 0 9679 0 --:--:-- --:--:-- --:--:-- 9679
For BlueField-2:
Fetch the firmware ID from FirmwareInventory:
curl -k -u root:'<password>' -X GET https:/<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/ { "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory", "@odata.type": "#SoftwareInventoryCollection.SoftwareInventoryCollection", "Members": [ { "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/8c8549f3_BMC_Firmware" …
Use command #4 with the fetched firmware ID in the previous step:
curl -k -u root:'<password>' -X GET https:/<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/8c8549f3_BMC_Firmware | jq -r ' .Version' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 471 100 471 0 0 622 0 --:--:-- --:--:-- --:--:-- 621 bmc-23.04
For BlueField-3 BMC only, w hen updating to an identical version, it is necessary to modify the ForceUpdate parameter to true using command #6 before initiating the update procedure. Otherwise, the update procedure fails with the message Component image is identical.
curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart --form "UpdateFile=@<package_path>;type=application/octet-stream" --form 'UpdateParameters={}'
{
"@odata.id": "/redfish/v1/TaskService/Tasks/0",
"@odata.type": "#Task.v1_4_3.Task",
"Id": "<id>",
"TaskState": "Running",
"TaskStatus": "OK"
}
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<id>
{
"@odata.id": "/redfish/v1/TaskService/Tasks/<id>",
...
"Messages": [
{
"@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry",
"Message": "The resource property 'BMC_Firmware' has detected errors of type 'Component image is identical'.",
"MessageArgs": [
"BMC_Firmware",
"Component image is identical"
],
"MessageId": "ResourceEvent.1.0.ResourceErrorsDetected",
"Resolution": "Retry firmware update operation with the force flag",
"Severity": "Critical"
}
...
"PercentComplete": 100,
"StartTime": "<time>",
"TaskMonitor": "/redfish/v1/TaskService/Tasks/<id>/Monitor",
"TaskState": "Exception",
"TaskStatus": "Critical"
}
Firmware update takes about 20 seconds.
After initiating the BMC secure update with the command #1 to from the previous table, a response similar to the following is received depending on whether HttpPushUri or MultipartHttpPushUri is used:
HttpPushUri API:
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService { "@odata.id": "/redfish/v1/TaskService/Tasks/0", "@odata.type": "#Task.v1_4_3.Task", "Id": "0", "TaskState": "Running"
MultipartHttpPushUri API:
curl -k -u root:'<password>' https://<bmc_ip>/redfish/v1/UpdateService/update-multipart --form "UpdateFile=@<package_path>;type=application/octet-stream" --form 'UpdateParameters={}' { "@odata.id": "/redfish/v1/TaskService/Tasks/0", "@odata.type": "#Task.v1_4_3.Task", "Id": "0", "TaskState": "Running", "TaskStatus": "OK" }
Where:
package_path – the BMC firmware image file including its path
For example:
curl -k -u root:'myP@ssword_12345!' https://10.10.10.10/redfish/v1/UpdateService/update-multipart --form UpdateFile=@/root/cec_ota_BMGP-04.0f_debug.bin;type=application/octet-stream --form 'UpdateParameters={}'
Command #2 can be used to track the progress of the CEC firmware update. For example:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/0 | jq -r ' .PercentComplete'
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 2123 100 2123 0 0 38600 0 --:--:-- --:--:-- --:--:-- 37910
100
After the CEC secure update operation is complete, a CEC activation and reset should be done (see "Activating New CEC") to apply the changes once the update is finished.
Command #5 can be used to verify the current CEC firmware version after reboot:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/Bluefield_FW_ERoT | jq -r ' .Version'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 421 100 421 0 0 1172 0 --:--:-- --:--:-- --:--:-- 1172
19-4
This is relevant only for BlueField-3 DPUs.
To activate the new CEC firmware, it is necessary to reset the CEC device. The possible options are explained in the following subsections.
Reset CEC and BMC Subsystems Using the CEC Self-reset Command
The CEC self-reset command is supported in CEC version
00.02.0180.0000 and above. The command
activates the new firmware on CEC and
should only be used after the CEC update procedure is complete.
Redfish Command
curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT/Actions/Chassis.Reset
Where:
password – password of root user
bmc_ip – BMC IP address
Command Response Options:
The response in case of success:
{ "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The request completed successfully.", "MessageArgs": [], "MessageId": "Base.1.15.0.Success", "MessageSeverity": "OK", "Resolution": "None" } ] }
The response if there is no pending CEC firmware:
curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT/Actions/Chassis.Reset -d '{"ResetType":"GracefulRestart"}' { "error": { "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The requested resource of type ERoT FW named 'Pending-ERoT-FW' was not found.", "MessageArgs": [ "ERoT FW", "Pending-ERoT-FW" ], "MessageId": "Base.1.15.0.ResourceNotFound", "MessageSeverity": "Critical", "Resolution": "Provide a valid resource identifier and resubmit the request." } ], "code": "Base.1.15.0.ResourceNotFound", "message": "The requested resource of type ERoT FW named 'Pending-ERoT-FW' was not found." } }
The response if the command is not supported by the current CEC firmware:
curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT/Actions/Chassis.Reset -d '{"ResetType":"GracefulRestart"}' { "error": { "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The action ERoT self-reset is not supported by the resource.", "MessageArgs": [ "ERoT self-reset" ], "MessageId": "Base.1.15.0.ActionNotSupported", "MessageSeverity": "Critical", "Resolution": "The action supplied cannot be resubmitted to the implementation. Perhaps the action was invalid, the wrong resource was the target or the implementation documentation may be of assistance." } ], "code": "Base.1.15.0.ActionNotSupported", "message": "The action ERoT self-reset is not supported by the resource." } }
IPMI Command
ipmitool raw 0x32 0xD2
Command Response Options:
In case of success, no response is given. Glacier and BMC reset.
The response if there is no pending CEC firmware:
Unable to send RAW command (channel=0x0 netfn=0x32 lun=0x0 cmd=0xd2 rsp=0xd6): Cannot execute command, command disabled
The response if the command is not supported by the current CEC firmware:
Unable to send RAW command (channel=0x0 netfn=0x32 lun=0x0 cmd=0xd2 rsp=0xd5): Command not supported in present state
Log Event Entries Created per Response Type
Log event entry created in case of success:
{ "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", "@odata.type": "#LogEntry.v1_13_0.LogEntry", "AdditionalDataURI": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>/attachment", "Created": "<Date>", "EntryType": "Event", "Id": "<Id>", "Message": "The request completed successfully.", "MessageArgs": [ "" ], "MessageId": "Base.1.0.Success", "Name": "System Event Log Entry", "Resolution": "Ready for ERoT self Reset", "Resolved": false, "Severity": "OK" }
Log event entry created if there is no pending CEC firmware:
curl -k -u root:'<password>' -H "Content-Type: application/json" -X GET https://<bmc_ip>/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/11 { "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", "@odata.type": "#LogEntry.v1_13_0.LogEntry", "AdditionalDataURI": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>/attachment", "Created": "<Date>", "EntryType": "Event", "Id": "<Id>", "Message": "Awaiting for an action to proceed with installing image '' on 'ERoT'.", "MessageArgs": [ "", "ERoT" ], "MessageId": "Update.1.0.AwaitToUpdate", "Name": "System Event Log Entry", "Resolution": "Cannot perform ERoT self reset: There is no EC FW pending. Perform an ERoT FW update.", "Resolved": false, "Severity": "OK" }
Log event entry created if the command is not supported by the current CEC firmware:
{ "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", "@odata.type": "#LogEntry.v1_13_0.LogEntry", "AdditionalDataURI": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>/attachment", "Created": "<Date>", "EntryType": "Event", "Id": "<Id>", "Message": "The action ERoT self Reset is not supported by the resource.", "MessageArgs": [ "ERoT self Reset" ], "MessageId": "Base.1.0.ActionNotSupported", "Name": "System Event Log Entry", "Resolution": "Cannot perform ERoT self reset: The action is not supported by the current ERoT version.", "Resolved": false, "Severity": "OK" }
Possible log event entries created if an error occurs during the BMC shut down procedure (received after a success log event entry):
{ "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", ... "MessageId": "Base.1.0.InternalError", "Name": "System Event Log Entry", "Resolution": "ERoT Reset: Failed to close services towards ERoT reset", "Resolved": false, "Severity": "Critical" }
{ "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", ... "MessageId": "Base.1.0.InternalError", "Name": "System Event Log Entry", "Resolution": "ERoT Reset: Isolate operation may still be in progress.", "Resolved": false, "Severity": "Critical" }
{ "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", ... "MessageId": "Base.1.0.InternalError", "Name": "System Event Log Entry", "Resolution": "ERoT Reset: Failed to unmount file system towards ERoT reset.", "Resolved": false, "Severity": "Critical" }
{ "@odata.id": "/redfish/v1/Systems/Bluefield/LogServices/EventLog/Entries/<Id>", ... "MessageId": "Base.1.0.InternalError", "Name": "System Event Log Entry", "Resolution": ERoT Reset: Failed to unmount file system. It is still mounted as read-write (rw), "Resolved": false, "Severity": "Critical" }
Reset CEC and BMC Subsystems Using IPMI I2C Command over SMBus Channel Connected to PCIe Golden Finger
This option is valid only for servers which support I2C over SMBus from the host BMC.
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x03 0xFE
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x01 0xFE
sleep <100ms>
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x01 0xFF
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x03 0xFF
The BUS-ID value is system related. It relays how the host BMC is connected to the SMBus of the related DPU.
The format of the ipmitool i2c command is as follows:
ipmitool raw <netfun> <cmd> <bus-id> <addr> <read-count> <write-data1> <write-data2>
Reset Entire BlueField DPU
This option typically involves a full power cycle of the host platform.
This section is relevant only for BlueField-3.
BMC and CEC have an active and inactive copy of the same firmware image on their respective firmware SPI flash. The firmware update is performed on the inactive copy, and on a successful boot from the newly updated and active image, the inactive image (e.g., the previous active image) is updated with the latest image.
Firmware update cannot be initiated if the background copy is in progress.
To check the status of the background update:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT
...
"Oem": {
"Nvidia": {
"@odata.type": "#NvidiaChassis.v1_0_0.NvidiaChassis",
"AutomaticBackgroundCopyEnabled": true,
"BackgroundCopyStatus": "Completed",
"InbandUpdatePolicyEnabled": true
}
}
…
The background update initially indicates InProgress while the inactive copy of the image is being updated.
This section is relevant only for BlueField-3.
|
Fault |
Diagnosis and Possible Solution |
|
Connection to BMC breaks during firmware package transfer |
A new firmware update can be attempted by the Redfish client. |
|
Connection to BMC breaks during firmware update |
A new firmware update can be attempted by the Redfish client. |
|
Two firmware update requests are initiated |
The Redfish server blocks the second firmware update request and returns the following:
Check the status of the ongoing firmware update by looking at the TaskCollection resource. |
|
Redfish task hangs |
A new firmware update can be attempted by the Redfish client. |
|
BMC-EROT communication failure during image transfer |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
Firmware update fails |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
ERoT failure (not responding) |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
Firmware image validation failure |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client might retry the firmware update. |
|
Power loss before activation command is sent |
A new firmware update can be attempted by the Redfish client. |
|
Firmware activation failure |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
Push to BMC firmware package greater than 200 MB |
|