BMC Management
NVIDIA BMC is based on the OpenBMC open-software framework which builds a complete Linux image for a board management controller (BMC). It uses the Yocto project as the underlying building and distro generation framework.
The primary software components of BMC are the following:
U-boot bootloader
Linux kernel
OpenBMC distro
There is a software version for each of the BMC software components. You may retrieve this information by running the following for each component:
Linux version – uname -a command from the Linux prompt
OpenBMC version – cat /etc/os-release from the Linux prompt
Retrieving BMC Version Using Redfish
curl -k -u root:'<password>' -H 'Content-Type: application/json' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware
{
"@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware",
"@odata.type": "#SoftwareInventory.v1_4_0.SoftwareInventory",
"Description": "BMC image",
"Id": "BMC_Firmware",
"Name": "Software Inventory",
"RelatedItem": [],
"RelatedItem@odata.count": 0,
"SoftwareId": "",
"Status": {
"Conditions": [],
"Health": "OK",
"HealthRollup": "OK",
"State": "Enabled"
},
"Updateable": true,
"Version": "BF-23.09-1",
"WriteProtected": false
}
Retrieving BMC Version Using IPMI
# ipmitool mc info
Device ID : 1
Device Revision : 1
Firmware Revision : 23.09
IPMI Version : 2.0
Manufacturer ID : 33049
Manufacturer Name : NVIDIA
Product ID : 4 (0x0004)
Product Name : Bluefield3 BMC
Device Available : yes
Provides Device SDRs : yes
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
IPMB Event Receiver
Chassis Device
Aux Firmware Rev Info :
0x10
0x01
0x00
0x00
Where the BMC version is composed of: [Firmware Revision]-[Aux Firmware Rev Info 2nd byte] in this example 23.9-1.
BMC starts booting through u-boot bootloader once the power supply is powered on.
By default, the BMC automatically boots into Linux. To stop at the u-boot prompt, users must type the password 0penBmc (note the use of the digit zero in 0pen) within 5 seconds. To boot Linux from the u-boot prompt, type boot.
The BMC provides indications of its status during its operation:
Scenario
Message
At the beginning of the boot process of the u-boot
Nvidia Bluefield BMC U-BOOT starting
At the beginning of the OS boot process
Nvidia Bluefield BMC Starting kernel ...
At the login prompt
Nvidia Bluefield BMC OS is up and running
Upon reboot or shutdown
Nvidia Bluefield BMC is shutting down
The default password for the root user, to be typed in once Linux is booted, is 0penBmc.
NoteFor information on password policy, refer to section "BMC Management Interface".
User Management Redfish Commands
General Information
General information about the BMC account services
curl -k -u root:'<password>' -H 'Content-Type: application/json' -X GET https://<IP>/redfish/v1/AccountService
Example output:
{
"@odata.id": "/redfish/v1/AccountService",
"@odata.type": "#AccountService.v1_10_0.AccountService",
"AccountLockoutDuration": 600,
"AccountLockoutThreshold": 4,
"Accounts": {
"@odata.id": "/redfish/v1/AccountService/Accounts"
},
..
"MaxPasswordLength": 20,
"MinPasswordLength": 13,
"Name": "Account Service",
"Oem": {
..
"Roles": {
"@odata.id": "/redfish/v1/AccountService/Roles"
},
"ServiceEnabled": true
}
List Supported User Roles
List supported user roles in the system:
curl -k -u root:'<password>' -H 'Content-Type: application/json' -X GET https://<IP>/redfish/v1/AccountService/Roles
Example output:
{
"@odata.id": "/redfish/v1/AccountService/Roles",
"@odata.type": "#RoleCollection.RoleCollection",
"Description": "BMC User Roles",
"Members": [
{
"@odata.id": "/redfish/v1/AccountService/Roles/Administrator"
},
{
"@odata.id": "/redfish/v1/AccountService/Roles/Operator"
},
{
"@odata.id": "/redfish/v1/AccountService/Roles/ReadOnly"
},
{
"@odata.id": "/redfish/v1/AccountService/Roles/NoAccess"
}
],
"Members@odata.count": 4,
"Name": "Roles Collection"
}
List User Accounts
curl -k -u root:'<password>' -H 'Content-Type: application/json' -X GET https://<IP>/redfish/v1/AccountService/Accounts
Example output:
{
"@odata.id": "/redfish/v1/AccountService/Accounts",
"@odata.type": "#ManagerAccountCollection.ManagerAccountCollection",
"Description": "BMC User Accounts",
"Members": [
{
"@odata.id": "/redfish/v1/AccountService/Accounts/NvdBluefieldUefi"
},
{
"@odata.id": "/redfish/v1/AccountService/Accounts/root"
}
],
"Members@odata.count": 2,
"Name": "Accounts Collection"
}
Create New User
Create a new user on the BMC:
curl -k -u root:'<password>' -H 'Content-Type: application/json' -X POST https://<IP>/redfish/v1/AccountService/Accounts -d '{ "UserName":"<USER>", "Password":"<PASSWORD>", "RoleId":"<ROLE>", "Enabled":true}'
Example output:
{
"@Message.ExtendedInfo": [
{
"@odata.type": "#Message.v1_1_1.Message",
"Message": "The resource has been created successfully.",
"MessageArgs": [],
"MessageId": "Base.1.15.0.Created",
"MessageSeverity": "OK",
"Resolution": "None."
}
]
}
Delete User
Delete user form the system:
curl -k -u root:'<password>' -H 'Content-Type: application/json' -X DELETE https://<IP>/redfish/v1/AccountService/Accounts/<USER>
Example output:
{
"@Message.ExtendedInfo": [
{
"@odata.type": "#Message.v1_1_1.Message",
"Message": "The account was successfully removed.",
"MessageArgs": [],
"MessageId": "Base.1.15.0.AccountRemoved",
"MessageSeverity": "OK",
"Resolution": "No resolution is required."
}
]
}
User Management IPMI Commands
|
# |
Function |
Command |
|
1 |
List the users |
For example:
|
|
2 |
User creation |
For example:
|
|
3 |
Set user password |
For example:
|
|
4 |
Enable user |
For example:
|
|
5 |
Disable user |
For example:
|
|
6 |
Set user privilege |
Where "privilege level":
For example:
|
|
7 |
Enable remote IPMI command functionality for user |
For example:
|
|
8 |
Lanplus commands to execute IPMI commands remotely for users with admin permissions |
For example:
|
|
9 |
Lanplus commands to execute IPMI commands remotely for users with other than administrator roles |
For example:
|
|
10 |
Delete user |
For example:
|
To obtain the BMC's MAC address, refer to the DPU's board label.
BMC management network interface can be configured using Redfish or IPMI. By default, BMC comes up with the DHCP network configuration.
Network configuration functions:
Setting DHCP/Static network mode configuration
Adding/setting IPv4/IPv6 configuration including IP address, gateway, netmask
Adding DNS servers
Adding NTP server
Setting BMC time with NTP server or system RTC
Network Management Redfish Commands
Get Network Protocol Configuration
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/NetworkProtocol
Get Interface Configuration
curl -k -u root:'<password>' -XGET https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0
Enable/Disable Interface
curl -k -u root:'<password>' -XPATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"InterfaceEnabled": <state>}'
Where <state> can be true or false.
Static IPv4 Address Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"IPv4StaticAddresses": [{"Address": "<ip_addr>","SubnetMask": "<netmask>","Gateway":"<gw_ip_addr>"}]}'
Example:
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"IPv4StaticAddresses": [{"Address": "10.7.7.7","SubnetMask": "255.255.0.0","Gateway":"10.7.0.1"}]}'
IPv4 DHCP Enable/Disable Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"DHCPv4": {"DHCPEnabled": <state>}}'
Where <state> can be true or false.
Static DNS server IPv4 and IPv6 Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"StaticNameServers": ["<dns_ip>"]}'
Static IPv6 Address Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"IPv6StaticAddresses": [{"Address": "<ip>", "PrefixLength": <len>}]}'
Example:
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"IPv6StaticAddresses": [{"Address": "fe80::3eec:efff:fe3b:e02f", "PrefixLength": 64}]}'
IPv6 DHCP Enable/Disable Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0 -d '{"DHCPv6": {"OperatingMode": "<state>"}}'
Where <state> can be:
stateful – DHCPv6 stateful mode is used to configure addresses, and when it is enabled, stateless mode is also implicitly enabled.
stateless – DHCPv6 stateless mode allows configuring the interface using DHCP options but does not configure addresses. It is always enabled by default whenever DHCPv6 stateful mode is also enabled.
disabled – DHCPv6 is disabled for this interface.
Enable NTP Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/NetworkProtocol -d '{"NTP": {"ProtocolEnabled": <state>}}'
Where <state> can be true or false.
Static NTP Server IP Configuration
curl -k -u root:'<password>' -X PATCH https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/NetworkProtocol -d '{"NTP": {"NTPServers": ["<ntp_server_ip>"]}}'
Network Management IPMI Commands
The following table lists the available network IPMI commands:
|
No. |
Function |
Command |
Description |
|
1 |
Change mode to Static |
For example:
|
Sets LAN channel 1 IP config mode to static which corresponds to network interface "eth0" |
|
2 |
Change mode to DHCP |
For example:
|
Sets LAN channel 1 IP config mode to DHCP which corresponds to the network interface "eth0" |
|
3 |
Add IPv4 address |
|
Adds IPv4 address, default gateway, and netmask to the network interface "eth0" |
|
4 |
Get IPv4 config |
|
Gets IPv4 network config for channel 1 which corresponds to the network interface "eth0" |
|
5 |
Set IPv6 address |
|
Adds IPv6 address to the network interface "eth0" |
|
6 |
Get IPv6 config |
|
Gets IPv6 network config for channel 1 which corresponds to the network interface "eth0" |
|
7 |
Get DNS server |
Output:
Corresponds to: 10.15.12.67 |
Gets the DNS server |
|
8 |
Add DNS server |
Output:
Corresponds to: 10.15.12.67 |
Adds the DNS server |
|
9 |
Get NTP server |
Output:
Where:
|
Gets NTP server |
|
10 |
Add NTP server |
Where:
|
Adds NTP server |
|
11 |
Enable time sync to NTP server |
Where:
|
Enables NTP time sync |
|
12 |
Enable time sync to system RTC |
Where:
|
Disables NTP time sync |
Reboot BMC Redfish Command
curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.Reset
Reboot BMC IPMI Command
ipmitool mc re cold
The following commands factory reset the BMC configuration.
Factory Reset Redfish Command
curl -k -u root:"<PASSWORD>" -H "Content-Type: application/json" -X POST https:/<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.ResetToDefaults -d '{"ResetToDefaultsType": "ResetAll"}'
Before connecting to the internet, it is important to change the default global password to prevent potential malicious attackers from hacking your system. For information on password policy, refer to section "BMC Management Interface".
Factory Reset IPMI Command
ipmitool raw 0x32 0x66
After issuing the ipmitool raw command for factory reset, you must log into the BMC and reboot it for the factory reset to take effect.
If you have lost your BMC login credentials and cannot login, you may issue the following command from the BlueField Arm:
ipmitool mc reset cold
Before connecting to the internet, it is important to change the default global password to prevent potential malicious attackers from hacking your system. For information on password policy, refer to section "BMC Management Interface".
Firmware upgrade of BMC and CEC components using BMC can be performed from a remote server using the Redfish interface.
|
No. |
Function |
Command |
Required for BMC/CEC Update |
Description |
|
1 |
Establish Redfish connection session |
Where:
|
BMC CEC |
Establish Redfish connection session |
|
2 |
Trigger a secure firmware update |
Where:
|
BMC CEC |
Triggers the secure update and starts tracking the secure update progress |
|
3 |
Track secure firmware update progress |
Find the current task ID in the response and use it for checking the progress:
Where:
|
BMC CEC |
Tracks the firmware update progress |
|
4 |
Reset/reboot a BMC |
Where:
|
BMC |
Resets/reboots the BMC |
|
5 |
Fetch running BMC firmware version |
For BlueField-3:
Where:
|
BMC |
Fetches the running firmware version from BMC |
|
For BlueField-2:
Fetch the current firmware ID and then perform:
Where:
|
||||
|
6 |
Fetch running CEC firmware version |
Where:
|
CEC |
Fetches the running firmware version from CEC |
BMC Update
Firmware update takes about 12 minutes.
After initiating the BMC secure update with the command #2 to from the previous table, a response similar to the following is received:
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService
{
"@odata.id": "/redfish/v1/TaskService/Tasks/0",
"@odata.type": "#Task.v1_4_3.Task",
"Id": "0",
"TaskState": "Running"
}
Command #3 from the previous table can be used to track secure firmware update progress. For instance:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/0 | jq -r ' .PercentComplete'
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 2123 100 2123 0 0 38600 0 --:--:-- --:--:-- --:--:-- 37910
20
Command #3 is used to verify the task has completed because during the update procedure the reboot option is disabled. When "PercentComplete" reaches 100, command #4 is used to reboot the BMC. For example:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/0 | jq -r ' .PercentComplete'
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 3822 100 3822 0 0 81319 0 --:--:-- --:--:-- --:--:-- 81319
100
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -d '{"ResetType": "GracefulRestart"}' https://<bmc_ip>/redfish/v1/Managers/Bluefield_BMC/Actions/Manager.Reset
{
"@Message.ExtendedInfo": [
{
"@odata.type": "#Message.v1_1_1.Message",
"Message": "The request completed successfully.",
"MessageArgs": [],
"MessageId": "Base.1.13.0.Success",
"MessageSeverity": "OK",
"Resolution": "None"
}
]
}
Command #5 can be used to verify the current BMC firmware version after reboot:
For BlueField-3:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware | jq -r ' .Version' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 513 100 513 0 0 9679 0 --:--:-- --:--:-- --:--:-- 9679
For BlueField-2:
Fetch the firmware ID from FirmwareInventory:
curl -k -u root:'<password>' -X GET https:/<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/ { "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory", "@odata.type": "#SoftwareInventoryCollection.SoftwareInventoryCollection", "Members": [ { "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/8c8549f3_BMC_Firmware" …
Use command #5 with the fetched firmware ID in the previous step:
curl -k -u root:'<password>' -X GET https:/<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/8c8549f3_BMC_Firmware | jq -r ' .Version' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 471 100 471 0 0 622 0 --:--:-- --:--:-- --:--:-- 621 bmc-23.04
CEC Update
Firmware update takes about 20 seconds.
After initiating the BMC secure update with the command #2 to from the previous table, a response similar to the following is received:
curl -k -u root:'<password>' -H "Content-Type: application/octet-stream" -X POST -T <package_path> https://<bmc_ip>/redfish/v1/UpdateService
{
"@odata.id": "/redfish/v1/TaskService/Tasks/0",
"@odata.type": "#Task.v1_4_3.Task",
"Id": "0",
"TaskState": "Running"
}
Command #3 can be used to track the progress of the CEC firmware update. For example:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/TaskService/Tasks/0 | jq -r ' .PercentComplete'
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 2123 100 2123 0 0 38600 0 --:--:-- --:--:-- --:--:-- 37910
100
After the CEC secure update operation is complete, a power cycle or cold reset of the BlueField-3 DPU must be manually triggered to apply the changes once the update is finished.
Command #6 can be used to verify the current CEC firmware version after reboot:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/Bluefield_FW_ERoT | jq -r ' .Version'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 421 100 421 0 0 1172 0 --:--:-- --:--:-- --:--:-- 1172
19-4
CEC Activation and Reset
This is relevant only for BlueField-3 DPUs only.
To activate the new CEC firmware, it is necessary to reset the CEC device. The following options are available:
Reset the entire BlueField DPU, which typically involves a full power cycle of the host platform.
Reset the CEC and BMC subsystems only. This can be done using the ipmitool i2c command over the SMBus channel connected to the PCIe golden finger.
WarningThis option is valid only for servers which support I2C over SMBus from the host BMC.
These options provide flexibility in managing the CEC device to apply the firmware update as needed.
To trigger the CEC reset:
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x03 0xFE
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x01 0xFE
sleep <100ms>
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x01 0xFF
ipmitool raw 0x06 0x52 <BUS-ID> 0x82 0x00 0x03 0xFF
The BUS-ID value is system related. It relays how the host BMC is connected to the SMBus of the related DPU.
The format of the ipmitool i2c command is as follows:
ipmitool raw <netfun> <cmd> <bus-id> <addr> <read-count> <write-data1> <write-data2>
CEC Background Update Status
This section is relevant only for BlueField-3.
BMC and CEC have an active and inactive copy of the same firmware image on their respective firmware SPI flash. The firmware update updates the inactive copy, and on a successful boot from the newly updated and active image, the inactive image (e.g., the previous active image) is updated with the latest image.
Firmware update cannot be initiated if the background copy is in progress.
To check the status of the background update:
curl -k -u root:'<password>' -X GET https://<bmc_ip>/redfish/v1/Chassis/Bluefield_ERoT
...
"Oem": {
"Nvidia": {
"@odata.type": "#NvidiaChassis.v1_0_0.NvidiaChassis",
"AutomaticBackgroundCopyEnabled": true,
"BackgroundCopyStatus": "Completed",
"InbandUpdatePolicyEnabled": true
}
}
…
The background update initially indicates InProgress while the inactive copy of the image is being updated with the copy.
Possible Error Codes
This section is relevant only for BlueField-3.
|
Fault |
Diagnosis and Possible Solution |
|
Connection to BMC breaks during firmware package transfer |
A new firmware update can be attempted by the Redfish client. |
|
Connection to BMC breaks during firmware update |
A new firmware update can be attempted by the Redfish client. |
|
Two firmware update requests are initiated |
The Redfish server blocks the second firmware update request and returns the following:
Check the status of the ongoing firmware update by looking at the TaskCollection resource. |
|
Redfish task hangs |
A new firmware update can be attempted by the Redfish client. |
|
BMC-EROT communication failure during image transfer |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
Firmware update fails |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
ERoT failure (not responding) |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
Firmware image validation failure |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client might retry the firmware update. |
|
Power loss before activation command is sent |
A new firmware update can be attempted by the Redfish client. |
|
Firmware activation failure |
The Redfish task monitoring the firmware update indicates a failure:
The Redfish client may retry the firmware update. |
|
Push to BMC firmware package greater than 200 MB |
|
Redfish triggers allow the user to get a journal message when a certain metric crosses a defined threshold for a defined time:
The trigger threshold can only be a numeric threshold
The trigger thresholds are unrelated to the sensor thresholds
The maximum number of triggers allowed in the system is 10
For more details, refer to Redfish Resource and Schema Guide.
|
No. |
Function |
Command |
Description |
|
1 |
Add a numeric trigger |
|
Adds a numeric trigger to the BMC |
|
2 |
Delete a trigger |
|
Deletes a trigger |
Certificate management actions (e.g., getting certificate information, doing atomic replacement of certificates) are found in the CertificateService resource.
The CertificateLocations resource is responsible for providing inventory of all the certificates which the service manages.
More details can be found in the Redfish Certificate Management White Paper.
|
No. |
Function |
Command |
Description |
|
1 |
Get certificate locations |
|
Inventory of all certificates the service is managing |
|
2 |
Get certificate Information |
|
Get certificate info |
|
3 |
Replace existing certificate |
|
Replace certificate |
|
4 |
Generate CSR |
|
Generate certificate signing request |
|
5 |
Install a certificate |
|
Install a certificate |