It is recommended to upgrade your BlueField product to the latest software and firmware versions available to benefit from new features and latest bug fixes.
This section assumes that a BlueField DPU has already been installed in a server according to the instructions detailed in the DPU's hardware user guide.
The following table lists an overview of the steps required to install Ubuntu BFB on your DPU:
|
Step |
Procedure |
Direct Link |
|
1 |
Verify that RShim is already running on BMC |
|
|
2 |
Change the default credentials using bf.cfg file (optional) |
|
|
3 |
Install the Ubuntu BFB image |
|
|
4 |
Verify installation completed successfully |
|
|
5 |
Upgrade the firmware on your DPU |
It is is important to learn your DPU's device-id for performing some of the software installations or upgrades in this guide.
To determine the device ID of the DPUs on your setup, run:
mst start
mst status -v
Example output:
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 3b:00.1 mlx5_1 net-ens1f1 0
BlueField2(rev:1) /dev/mst/mt41686_pciconf0 3b:00.0 mlx5_0 net-ens1f0 0
BlueField3(rev:1) /dev/mst/mt41692_pciconf0.1 e2:00.1 mlx5_1 net-ens7f1np1 4
BlueField3(rev:1) /dev/mst/mt41692_pciconf0 e2:00.0 mlx5_0 net-ens7f0np0 4
The device IDs for the BlueField-2 and BlueField-3 DPUs in this example are /dev/mst/mt41686_pciconf0 and /dev/mst/mt41692_pciconf0 respectively.
Display the current setting. Run:
# cat /dev/rshim<N>/misc | grep DEV_NAME
DEV_NAME usb-1.0
This output indicates that the RShim service is ready to use. If you do not receive this output:
Restart RShim service:
sudo systemctl restart rshim
Verify the current setting again. Run:
# cat /dev/rshim<N>/misc | grep DEV_NAME
If DEV_NAME does not appear, then proceed to "RShim driver not loading on DPU with integrated BMC".
Ubuntu users are prompted to change the default password (ubuntu) for the default user (ubuntu) upon first login. Logging in will not be possible even if the login prompt appears until all services are up ("DPU is ready" message appears in /dev/rshim0/misc).
Attempting to log in before all services are up prints the following message: "Permission denied, please try again."
Alternatively, Ubuntu users can provide a unique password that will be applied at the end of the BFB installation. This password would need to be defined in a bf.cfg configuration file. To set the password for the ubuntu user:
Create password hash. Run:
# openssl passwd -1 Password: Verifying - Password: $1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1
Add the password hash in quotes to the bf.cfg file:
# vim bf.cfg ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'
The bf.cfg file will be used with the bfb-install script in the following step.
WarningPassword policy:
Minimum password length – 8
At least one upper-case letter
At least one lower-case letter
At least one numerical character
For comprehensive list of the supported parameters to customize bf.cfg during BFB installation, refer to section "bf.cfg Parameters".
To update the software on the BlueField Arm cores, the DPU must be booted up without mounting the eMMC flash device. This requires an external boot flow where a BFB (which includes ATF, UEFI, Linux kernel, and initramfs) is pushed from an external host via USB or PCIe. On BlueField DPUs with an integrated BMC, the USB interface is internally connected to the BMC and is enabled by default. Therefore, you must verify that the RShim driver is running on the BMC. This provides the ability to push a bootstream over the USB interface to perform an external boot.
Disable RShim on the host. Run on the host:
systemctl stop rshim systemctl disable rshim
Enable RShim on the BMC. Run on the BMC shell:
ipmitool raw 0x32 0x6a 1
Verify that the RShim service is running and that the /dev/rshim0 device is present:
ipmitool raw 0x32 0x69
The expected output for this command is 0x01.
If bf.cfg is needed as part of the boot process, run:
cat <path-to-bfb> bf.cfg > new.bfb
Since the BFB is too large to store on the BMC flash or tmpfs, the image can be written to the RShim device. This can be done by either running SCP directly or using Redfish interface.
SCP
Run:
scp <path_to_bfb> root@<bmc_ip>:/dev/rshim0/boot
Redfish:
Establish Redfish connection session. Run the following on the host:
export token=`curl -k -H "Content-Type: application/json" -X POST https://<bmc_ip>/login -d '{"username":"root", "password":"<password>"}' | grep token | awk '{print $2;}' | tr -d '"'`
Where:
bmc_ip – BMC IP address
password – password of root account
Confirm host identity.
NoteRequired only during first-time setup or after BMC factory reset.
Run the following on the host:
ssh-keyscan -t ed25519 <remote_server_ip>
Where remote_server_ip is the IP address of the server which stores the BFB file.
Retrieve the remote server's public key from the second line of the response then run the following on the host:
curl -k -H "X-Auth-Token: <token>" -H "Content-Type: application/json" -X POST -d '{"Server":"<remote_server_ip>", "KeyType":"ssh-ed25519", "Key":"<remote_server_public_key>"}' https://<bmc_ip>/redfish/v1/UpdateService/Actions/Oem/NvidiaUpdateService.AddRemoteServerPublicKey
Where:
token – session token received when establishing connection
remote_server_ip – remote server's IP address
remote_server_public_key – remote server's public key from ssh-keyscan response
bmc_ip – BMC IP address
Run the following on the host:
curl -k -H "X-Auth-Token: <token>" -H "Content-Type: application/json" -X POST -d '{"TransferProtocol":"SCP", "ImageURI":"<image_uri>", "Username":"<username>", "Password":"<password>"}' https://<bmc_ip>/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate
WarningThis command uses SCP for the image transfer, initiates a soft reset on the BlueField and then pushes the boot stream. For Ubuntu BFBs, the eMMC is flashed automatically once the bootstream is pushed. On success, a "running" message is received with the current task ID.
Where:
token – session token received when establishing connection
image_uri – the image URI format should be <remote_server_ip>/<path_to_bfb>
username – username on the remote server
password – user password on the remote server
bmc_ip – BMC IP addressFor example:
If RShim is disabled:
{ "error": { "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The requested resource of type Target named '/dev/rshim0/boot' was not found.", "MessageArgs": [ "Target", "/dev/rshim0/boot" ], "MessageId": "Base.1.15.0.ResourceNotFound", "MessageSeverity": "Critical", "Resolution": "Provide a valid resource identifier and resubmit the request." } ], "code": "Base.1.15.0.ResourceNotFound", "message": "The requested resource of type Target named '/dev/rshim0/boot' was not found." }
If a username or a password is missing:
{ "error": { "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The value '' for the parameter username/password in the action UpdateService.SimpleUpdate is of a different type than the parameter can accept.", "MessageArgs": [ "", "username", "UpdateService.SimpleUpdate" ], "MessageId": "Base.1.15.0.ActionParameterValueTypeError", "MessageSeverity": "Warning", "Resolution": "Correct the value for the parameter in the request body and resubmit the request if the operation failed." } ], "code": "Base.1.15.0.ActionParameterValueTypeError", "message": "The value '' for the parameter username in the action UpdateService.SimpleUpdate is of a different type than the parameter can accept." }
If the request is valid and a task is created:
{ "@Message.ExtendedInfo": [ { "@odata.type": "#Message.v1_1_1.Message", "Message": "The request completed successfully.", "MessageArgs": [], "MessageId": "Base.1.15.0.Success", "MessageSeverity": "OK", "Resolution": "None" } ], "@odata.id": "/redfish/v1/TaskService/Tasks/<task_id>", "@odata.type": "#Task.v1_4_3.Task", "Id": "<task_id>", "TaskState": "Running", "TaskStatus": "OK" }
Track secure image transfer progress. Run the following on the host:
curl -k -H
"X-Auth-Token: <token>"-X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/<task_id>WarningDuring the transfer, the PercentComplete value remains at 0. If no errors occur, the TaskState is set to Running, and a keep-alive message is generated every 5 minutes with the content "Transfer is still in progress (X minutes elapsed). Please wait". Once the transfer is completed, the PercentComplete is set to 100, and the TaskState is updated to Completed.
Upon failure, a message is generated with the relevant resolution.
Where:
token – session token received when establishing connection
bmc_ip – BMC IP address
task_id – task IDFor example:
If host identity is not confirmed:
{ "@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry", "Message": "Transfer of image '<file_name>' to '/dev/rshim0/boot' failed.", "MessageArgs": [ "<file_name>, "/dev/rshim0/boot" ], "MessageId": "Update.1.0.TransferFailed", "Resolution": "Host is unknown: Please add host public key using AddRemoteServerPublicKey", "Severity": "Critical" } … "PercentComplete": 0, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Exception", "TaskStatus": "Critical"
If SCP fails:
{ "@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry", "Message": "Transfer of image '<file_name>' to '/dev/rshim0/boot' failed.", "MessageArgs": [ "<file_name>", "/dev/rshim0/boot" ], "MessageId": "Update.1.0.TransferFailed", "Resolution": "Failed to launch SCP", "Severity": "Critical" } …. "PercentComplete": 0, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Exception", "TaskStatus": "Critical"
The keep-alive message:
{ "@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry", "Message": " <file_name>' is being transferred to '/dev/rshim0/boot'.", "MessageArgs": [ " <file_name>", "/dev/rshim0/boot" ], "MessageId": "Update.1.0.TransferringToComponent", "Resolution": "Transfer is still in progress (5 minutes passed): Please wait", "Severity": "OK" } … "PercentComplete": 0, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Running", "TaskStatus": "OK"
Upon success:
{ "@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry", "Message": "Device 'DPU' successfully updated with image '<file_name>'.", "MessageArgs": [ "DPU", "<file_name>" ], "MessageId": "Update.1.0.UpdateSuccessful", "Resolution": "None", "Severity": "OK" }, … "PercentComplete": 100, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Completed", "TaskStatus": "OK"
For comprehensive list of the supported parameters to customize bf.cfg during BFB installation, refer to section "bf.cfg Parameters".
After installation of the Ubuntu OS is complete, the following note appears in /dev/rshim0/misc on first boot:
...
INFO[MISC]: Linux up
INFO[MISC]: DPU is ready
"DPU is ready" indicates that all the relevant services are up and users can login the system.
After the installation of the Ubuntu 20.04 BFB, the configuration detailed in the following sections is generated.
Make sure all the services (including cloud-init) are started on BlueField before power cycling the host server.
BlueField OS image version is stored under /etc/mlnx-release in the DPU.
# cat /etc/mlnx-release
DOCA_v1.1_BlueField_OS_Ubuntu_20.04-<version>
mlxfwreset is not supported in this release. Please power cycle the host where mlxfwreset is requested.
To upgrade firmware:
Set a temporary static IP on the host. Run:
sudo ip addr add 192.168.100.1/24 dev tmfifo_net0
SSH to your DPU via 192.168.100.2 (preconfigured). The default credentials for Ubuntu are as follows.
Username
Password
ubuntu
Set during installation
For example:
ssh ubuntu@192.168.100.2 Password: <unique-password>
Upgrade the firmware on the DPU. Run:
sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update
Example output:
Device #1: ---------- Device Type: BlueField-2 [...] Versions: Current Available FW <Old_FW> <New_FW>
WarningImportant! To apply NVConfig changes, stop here and follow the steps in section "Updating NVConfig Params".
Power cycle the host for the changes to take effect.
Reset the nvconfig params to their default values:
# sudo mlxconfig -d /dev/mst/<device-name> -y reset Reset configuration for device /dev/mst/<device-name>? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.
(Optional) Enable NVMe emulation. Run:
sudo mlxconfig -d <device-name> s NVME_EMULATION_ENABLE=1
Skip this step if your BlueField DPU is Ethernet only. Please refer to section "Supported Platforms and Interoperability" under the Release Notes to learn your DPU type.
If you have a VPI DPU, the default link type of the ports will be configured to IB. If you want to change the link type to Ethernet, please run the following configuration:
sudo mlxconfig -d <device-name> s LINK_TYPE_P1=2 LINK_TYPE_P2=2
Power cycle the host for the mlxconfig settings to take effect.