Bare-metal Reprovisioning
Relevant for NVIDIA® BlueField®-3 and later in DPU mode only (not supported in NIC mode).
The re-provisioning flow of the BlueField-3 bare metal system offers a solution for restoring the system to its initial state using built-in resources, eliminating the need for external measures. This approach enables the seamless reloading of the operational image.
To support this functionality, the BMC maintains and manages a golden image for the UEFI and the NIC. This ensures that the UEFI can retrieve the operational image via network protocols such as HTTP or PXE.
The following block diagram provides a high-level overview of the system components and data flow:
The complete flow of network re-provisioning includes the following primary stages:
Initial provisioning – Provisioning the golden images to the BMC, typically performed during system manufacturing.
In-field updates – Updating the golden images using the in-field update process.
OOB network configuration – Configuring network settings through out-of-band (OOB) management.
System recovery – Restoring the system by reinstalling the golden images.
Ensure the BMC is connected to the out-of-band (OOB) network.
Use the
scpcommand to transfer the golden images from your local storage to the BMC's temporary storage directories (/tmp/golden-image-nic/or/tmp/golden-image-arm/).For
golden_image_nic:#host> scp <nic-golden-image-directory>/<nic-golden-image-filename> root@<bmc-ip>:/tmp/golden-image-nic/For
golden_image_arm:#host> scp <arm-golden-image-directory>/<arm-golden-image-filename> root@<bmc-ip>:/tmp/golden-image-arm/InfoAfter the image is copied to the BMC's volatile memory, the version is extracted and stored to enable specific features.
NoteThe NIC firmware version is extracted from the image filename. Ensure the filename follows the standard format of official releases.
Log into the BMC and use the
dpu_golden_imageutility to transfer the golden images from temporary storage to the BMC's non-volatile storage.For
golden_image_nic:#bmc> dpu_golden_image golden_image_nic -w /tmp/golden-image-nic/<nic-golden-image-filename>For
golden_image_arm:#bmc> dpu_golden_image golden_image_arm -w /tmp/golden-image-arm/<arm-golden-image-filename>InfoIf the version of the candidate image matches the one already in non-volatile storage, the update is skipped, and the following message is displayed:
Updating image in memory is cancelled as version will remain unchanged. Force update by adding -f|--force to the command line.
To force the update, use the
--forceflag:For
golden_image_nic:#bmc> dpu_golden_image golden_image_nic -w /tmp/golden-image-nic/<nic-golden-image-filename> --forceFor
golden_image_arm:#bmc> dpu_golden_image golden_image_arm -w /tmp/golden-image-arm/<arm-golden-image-filename> --force
After provisioning, verify the correctness of the golden images using the following commands:
For
golden_image_nic:#bmc> dpu_golden_image -v golden_image_nic bmc> echo $?
Expected output:
0.For
golden_image_arm:bmc> dpu_golden_image -v golden_image_arm bmc> echo $?
Expected output:
0.
This feature is available only for golden images installed following the upgrade of the BMC firmware to version 24.07-14 or later.
To get the human-readable version (MAJOR.MINOR.PATCH.BUILD versioning scheme) of the golden images:
For
golden_image_nic:bmc> dpu_golden_image golden_image_nic -V -H
For
golden_image_arm:bmc> dpu_golden_image golden_image_arm -V -H
To get the sha256sum value:
For
golden_image_nic:bmc> dpu_golden_image golden_image_nic -V
For
golden_image_arm:bmc> dpu_golden_image golden_image_arm -V
This feature is available only for Golden Images installed following the upgrade of the BMC firmware to version 24.07-14 or later.
To get the human-readable version (MAJOR.MINOR.PATCH.BUILD versioning scheme) of the golden images over the Redfish interface, run:
For Arm golden image:
curl -k -u
'<username>':'<password>'-H'Content-type: application/json'-X GET'https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/golden_image_arm'For NIC golden image:
curl -k -u
'<username>':'<password>'-H'Content-type: application/json'-X GET'https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/golden_image_nic'
The process for updating golden images via Redfish includes the following steps.
The process for updating golden images via Redfish includes the following steps:
For NIC golden image:
curl -k -u root:
'<password>'-H"Content-Type: application/json"-X POST -d'{"TransferProtocol":"HTTP", "ImageURI":"<remote-server-ip>/<nic-golden-image-path>", "Targets":["redfish/v1/UpdateService/FirmwareInventory/golden_image_nic"]}'https://<bmc-ip>/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdateFor Arm golden image:
curl -k -u root:
'<password>'-H"Content-Type: application/json"-X POST -d'{"TransferProtocol":"HTTP", "ImageURI":"<remote-server-ip>/<arm-golden-image-path>", "Targets":["redfish/v1/UpdateService/FirmwareInventory/golden_image_arm"]}'https://<bmc-ip>/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdateParameters:
ImageURI– Specify the image location in the format<remote-server-ip>/<golden-image-path>bmc-ip– Specify the IP address of the BMCAfter initiating the update, a new task is created for monitoring progress, with a sample response:
{ "@odata.id": "/redfish/v1/TaskService/Tasks/0", "@odata.type": "#Task.v1_4_3.Task", "Id": "0", "TaskState": "Running", "TaskStatus": "OK" }
Track the update's progress using the following command:
curl -k -u root:
'<password>'-X GET https://<bmc-ip>/redfish/v1/TaskService/Tasks/<task-id>Update states:
0%– Update started.10%– Update in progress.100%– Update complete.Expected output after a successful update:
"PercentComplete": 100,"TaskState":"Completed","TaskStatus":"OK"In case of failure, reboot the BMC and retry the update.
InfoThe golden image update process typically takes 1–3 minutes to complete.
If the candidate image version matches the one stored in the BMC's non-volatile memory, the update is skipped by default, and the following response is returned:
{
...
{
"@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry",
"Message": "The update operation for the component 'BMC' is skipped because 'Component image is identical'.",
"MessageArgs": [
"BMC",
"Component image is identical"
],
"MessageId": "NvidiaUpdate.1.0.ComponentUpdateSkipped",
"Resolution": "Retry firmware update operation with the force flag",
"Severity": "OK"
},
],
...
"TaskState": "Completed",
"TaskStatus": "OK"
}
To override the default behavior and force the update, include the "ForceUpdate": true parameter in the command:
For NIC golden image:
curl -k -u root:
'<password>'-H"Content-Type: application/json"-X POST -d'{"TransferProtocol":"HTTP", "ImageURI":"<remote-server-ip>/<nic-golden-image-path>", "Targets":["redfish/v1/UpdateService/FirmwareInventory/golden_image_nic"], "ForceUpdate": true}'https://<bmc-ip>/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdateFor Arm golden image:
curl -k -u root:
'<password>'-H"Content-Type: application/json"-X POST -d'{"TransferProtocol":"HTTP", "ImageURI":"<remote-server-ip>/<arm-golden-image-path>", "Targets":["redfish/v1/UpdateService/FirmwareInventory/golden_image_arm"], "ForceUpdate": true}'https://<bmc-ip>/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate
To enhance the system's security, a new mechanism has been introduced to control network connectivity over the OOB network. This new feature provides an IPMI command to disable any communication between the BlueField BMC, BlueField, and the OOB management network. A set of IPMI commands are introduced to selectively enable the network on each of the above interfaces. This permits the platform's RoT to have complete control over which network interfaces can be enabled and when.
This IPMI can only be sent by the platform's ROT. OOB and BlueField are blocked.
By default, the OOB interface is enabled. However, for the host BMC to gain control over this interface, it must disable it during the initial boot. Once disabled, the interface remains in that state regardless of BMC reboots or system cold boots.
For more details, refer to "OOB Network 3-Port Switch Control".
The re-provisioning flow is initiated using the following IPMI command:
bmc> ipmitool raw 0x32 0x99 <golden_image_timeout> <timeout_from_network> <verbosity_level> <halt_hard_reset>
This command must be executed from within the BMC, as it can significantly impact the system. Upon execution:
The golden images are extracted from the BlueField BMC's non-volatile memory.
The recovery process is initiated, pushing the golden images to the RShim.
The RShim console output is redirected to the BMC console, allowing the user to monitor the process.
Once the process completes, both the BlueField NIC and ARM execute the designated golden images retrieved from a preconfigured server.
Command parameters:
<golden_image_timeout>– Timeout value (in minutes) for updating the golden images. Default Value:15minutes (input0to use the default).<timeout_from_network>– Timeout value (in minutes) for booting the operational image from the network. Default Value:60minutes (input0to use the default).<verbosity_level>– Controls the level of detail displayed during the reprovisioning process:0– Quiet mode (only error messages are displayed)1– Info mode (error messages and reprovisioning process messages are displayed)2– Full mode (all messages, including BlueField RShim messages, are displayed)
<halt_hard_reset>(Optional) – Specifies whether to halt the reprovisioning process before performing the final hard reset of the BlueField.NoteThe final hard reset is crucial to activate the NIC firmware installed from the network.
0– Perform the hard reset to complete the reprovisioning process (default)1– Halt the process before the final hard resetInfoReprovisioning messages are prefixed with
[<running date> GOLDEN-IMAGE-RECOVERY].
After BFB installation is complete, the BlueField BMC waits for a specific sequence of messages over the RShim log:
NIC firmware update done # Indicates that the firmware update for the NIC subsystem has been successfully completed
Installation finished # Signals the completion of the installation process for the BFB from the network
Linux up # Indicates that the BlueField BMC has acknowledged that the Arm OS has booted up and is ready
BlueField BMC expects these messages in the specified order.
Users can add custom entries to the RShim log from the BlueField Arm OS using the bfrshlog command. The syntax of the command is: bfrshlog <output>.
For example, to add the message "Linux up" to the RShim log, run:
bfrshlog "Linux up"
All output from the BlueField Arm console is redirected to the BlueField BMC console for monitoring purposes.
The steps of the re-provisioning process are printed with
[<running date> GOLDEN-IMAGE-RECOVERY]prefix and are outlined in the following:
[<running date> GOLDEN-IMAGE-RECOVERY] Checking pcie slot is in reset
[<running date> GOLDEN-IMAGE-RECOVERY] Read golden images from flash
[<running date> GOLDEN-IMAGE-RECOVERY] Set FNP to 0
[<running date> GOLDEN-IMAGE-RECOVERY] Checking rshim interface after SOC hard reset
[<running date> GOLDEN-IMAGE-RECOVERY] Starting ATF/UEFI golden image update
[<running date> GOLDEN-IMAGE-RECOVERY] Finished updating ATF/UEFI golden image
[<running date> GOLDEN-IMAGE-RECOVERY] Starting NIC FW golden image update
[<running date> GOLDEN-IMAGE-RECOVERY] Finished updating NIC FW golden image
[<running date> GOLDEN-IMAGE-RECOVERY] Stop Redfish server
[<running date> GOLDEN-IMAGE-RECOVERY] Configure Recovery image to boot from network
[<running date> GOLDEN-IMAGE-RECOVERY] set FNP to 1
[<running date> GOLDEN-IMAGE-RECOVERY] Booting BFB from network
[<running date> GOLDEN-IMAGE-RECOVERY] Start Redfish server
[<running date> GOLDEN-IMAGE-RECOVERY] Set boot option to default
if halt_hard_reset is 0:
[<running date> GOLDEN-IMAGE-RECOVERY] Finished programming image from network. Start DPU hard reset
if halt_hard_reset is 1:
[<running date> GOLDEN-IMAGE-RECOVERY] Finished programming image from network
[<running date> GOLDEN-IMAGE-RECOVERY] The Reprovisioning process was halted at user's request. To complete the process, please power cycle the device
A failed update prints the following:
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: aborting process! PCIE is not in reset.
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: Reading golden_image_nic failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: Reading golden_image_arm failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: rshim has not started successfully
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: pushing ATF/UEFI golden image over rshim failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: programming of ATF/UEFI golden image failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: pushing NIC FW golden image over rshim failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: programming of NIC FW golden image failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: failed to configure image to boot from network
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: programming of image from network failed: NIC firmware update failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: programming of image from network failed: Installation failed
[<running date> GOLDEN-IMAGE-RECOVERY] ERROR: programming of image from network failed: Failed to get Linux up
Due to line buffering in the BlueField Arm console, buffered output lines receive the same timestamp value in <running date> when they are redirected to the BlueField BMC console.