Redfish
DMTF’s Redfish® is a standard designed to deliver simple and secure management for converged, hybrid IT and the Software Defined Data Center (SDDC). Redfish leverages common Internet and web services standards to expose information directly to the modern tool chain.
In case of Bluefield, Redfish server is located on Bluefield-BMC (aka DPU-BMC) and UEFI Redfish client runs on Bluefield. In the case of Monterey, Redfish server is located on Platform-BMC (aka Server-BMC).
For the most part, BMC acts as a middle-man caching requests and responses between user and UEFI Redfish client.
This page will go over the troubleshooting steps when a Redfish request is not providing desired results due to failure in processing by UEFI Redfish client.
|
Command |
Description |
|
|
Soft reset the BlueField connected on |
|
|
Increase the display log level to |
|
|
Open a serial console to Bluefield on rshim0 and store the |
|
|
Open a serial console to BlueField from its BMC Note
The console logs are stored in
|
RShim Logging
INFO[UEFI]: Redfish enabled Redfish has been enabled
INFO[UEFI]: DPU-BMC RF credentials not found BMC failed to share credentials
INFO[UEFI]: DPU-BMC RF credentials found BMC shared the Redfish credentials with UEFI successfully
ERR[UEFI]: RF DHCPv4 failed BMC DHCP server didn't respond / failed to provide IP address
DHCP Session Start UEFI is attempting DHCP request with BMC
Redfish Requests are Not Processed During DOCA BFB Installation or When BFB is Pushed via RShim to BlueField
The UEFI Redfish client skips execution when a .bfb file is pushed via RShim in order to facilitate the installation process completion. UEFI Redfish client performs a number of HTTP transactions among other things, and hence results in increased boot time, impeding the installation process. Hence execution of UEFI Redfish client is skipped.
For the UEFI Redfish client to process the Redfish requests, simply reboot or soft reset the BlueField.
Redfish Data in BMC Redfish Server is Empty
This may happen due to the fact that the UEFI Redfish client in BlueField has not run since the BMC has booted (following a power cycle or reboot).
For the UEFI Redfish client to run and populate the current settings on the BMC Redfish server, simply reboot or soft reset the BlueField.
Redfish Request is Not Processed Even After Rebooting BlueField
This could happen due to the fact that the request requires additional user action for it to take effect.
Reboot BlueField again to see if that resolves the issue (e.g.,
SecureBootEnablesetting requires an additional reboot of BlueField (1st reboot is for the UEFI Redfish client to execute and process the request, and the 2nd reboot is for the configuration to take effect)If this is a setting request to change mode of operation to NIC or DPU mode, system power cycle is required
No Result Even After Submitting the Redfish Request and Rebooting BlueField Several Times
This may happen due to any of the reasons in the following subsections.
Redfish Not Enabled on BlueField
This can be verified by looking at the RShim logs. In RShim logs, look for "Redfish enabled" log message:
# echo "DISPLAY_LEVEL 2" > /dev/rshim1/misc; cat /dev/rshim1/misc
DISPLAY_LEVEL 2 (0:basic, 1:advanced, 2:log)
BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT 150 (seconds)
DROP_MODE 0 (0:normal, 1:drop)
SW_RESET 0 (1: reset)
DEV_NAME pcie-0000:31:00.5
DEV_INFO BlueField-3(Rev 1)
OPN_STR N/A
UP_TIME 69302(s)
SECURE_NIC_MODE 0 (0:no, 1:yes)
---------------------------------------
Log Messages
---------------------------------------
INFO[PSC]: PSC BL1 START
INFO[BL2]: start
INFO[BL2]: boot mode (emmc)
INFO[BL2]: VDDQ adjustment complete
INFO[BL2]: VDDQ: 1118 mV
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: lifecycle Secured (development)
INFO[BL31]: VDD: 747 mV
INFO[BL31]: runtime
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[BL31]: MB ping success
INFO[UEFI]: UPVS valid
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[UEFI]: UEFI Secure Boot (disabled)
INFO[UEFI]: Redfish enabled
If Redfish is not enabled, access the UEFI menu on the BlueField console and navigate to Device Manager > System Configuration > Redfish Configuration and enable the Redfish setting.
BlueField BMC Credentials Not Found
BlueField requires Redfish credentials to perform redfish transactions. These credentials are retrieved using a separate channel from the BlueField BMC.
If these are not received, then BlueField would not be able to transact Redfish requests with its BMC. To check whether the credentials are found check the RShim logs:
# echo "DISPLAY_LEVEL 2" > /dev/rshim1/misc; cat /dev/rshim1/misc
DISPLAY_LEVEL 2 (0:basic, 1:advanced, 2:log)
BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT 150 (seconds)
DROP_MODE 0 (0:normal, 1:drop)
SW_RESET 0 (1: reset)
DEV_NAME pcie-0000:31:00.5
DEV_INFO BlueField-3(Rev 1)
OPN_STR N/A
UP_TIME 69302(s)
SECURE_NIC_MODE 0 (0:no, 1:yes)
---------------------------------------
Log Messages
---------------------------------------
INFO[PSC]: PSC BL1 START
INFO[BL2]: start
INFO[BL2]: boot mode (emmc)
INFO[BL2]: VDDQ adjustment complete
INFO[BL2]: VDDQ: 1118 mV
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: lifecycle Secured (development)
INFO[BL31]: VDD: 747 mV
INFO[BL31]: runtime
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[BL31]: MB ping success
INFO[UEFI]: UPVS valid
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[UEFI]: UEFI Secure Boot (disabled)
INFO[UEFI]: Redfish enabled
INFO[UEFI]: DPU-BMC RF credentials not found
If the BlueField BMC credentials are not found try rebooting the BMC. If the issue persists, contact NVIDIA Enterprise Support for further assistance.
Redfish DHCP Failure
The first step of Redfish transactions initiated by the UEFI Redfish client is the DHCP request. The UEFI Redfish interface's network must be configured and also information about Redfish host interface must be known, hence the DHCP request. If the DHCP request fails or receives no response, then the UEFI Redfish client aborts and logs the error in RShim log:
# echo "DISPLAY_LEVEL 2" > /dev/rshim1/misc; cat /dev/rshim1/misc
DISPLAY_LEVEL 2 (0:basic, 1:advanced, 2:log)
BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT 150 (seconds)
DROP_MODE 0 (0:normal, 1:drop)
SW_RESET 0 (1: reset)
DEV_NAME pcie-0000:31:00.5
DEV_INFO BlueField-3(Rev 1)
OPN_STR N/A
UP_TIME 134(s)
SECURE_NIC_MODE 0 (0:no, 1:yes)
---------------------------------------
Log Messages
---------------------------------------
INFO[PSC]: PSC BL1 START
INFO[BL2]: start
INFO[BL2]: boot mode (emmc)
INFO[BL2]: VDDQ: 1120 mV
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: lifecycle Secured (development)
INFO[BL31]: VDD: 746 mV
INFO[BL31]: runtime
INFO[BL31]: MB ping success
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[UEFI]: UPVS valid
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[UEFI]: UEFI Secure Boot (disabled)
INFO[UEFI]: Redfish enabled
INFO[UEFI]: DPU-BMC RF credentials found
ERR[UEFI]: RF DHCPv4 failed
Since BlueField BMC is responsible for providing the DHCP service, try rebooting the BlueField BMC. If the issue persists, contact NVIDIA Enterprise Support for further assistance.
NIC Firmware/UNDI Driver Did Not Load
For many of the Redfish vendors, the network function is provided by the universal network device interface (UNDI) driver in UEFI. The UNDI driver is part of the NIC firmware and is loaded by UEFI during PCIe enumeration. UEFI first verifies the signature and then loads the UNDI driver. If the NIC firmware on the BlueField platform is incorrectly installed or corrupt, then the UNDI driver is not loaded in UEFI, and the UEFI Redfish client would not run.
If this indeed is the issue, then checking the RShim logs would show that only the Redfish enabled message is printed, but there is no information about DPU-BMC RF credentials found/not found or about RF DHCPv4/6 failed. This can be resolved by installing the recommended version of NIC firmware as described in the respective manual.
Unknown or Unsupported Vendor in Redfish Service Root
Once the network is set up via DHCP, the next step in the Redfish flow is to read the Redfish vendor information from the Redfish service root (/redfish/v1). No credentials are required to access this URL. Vendor information is critical as it decides the imminent flow (e.g., the authorization type to use to get a token—X-AUTH or O-AUTH—the URLs for different Redfish schemas). For Redfish on the BlueField BMC, the vendor should always be set to Nvidia (case sensitive):
Redfish enabled
DHCP Session Start
Press ESC/F2/DEL twice to enter UEFI Menu.
Press ENTER to skip countdown.
3 seconds remain...
2 seconds remain...
1 seconds remain...
0 seconds remain...
** Redfish GET https://192.168.240.1/redfish/v1/, Success
ProcessVendorIdentification: Vendor: Nvidia
Refer to section "Command Cheat Sheet" for the commands to access the console.
If the vendor information does not match, then UEFI Redfish client aborts further transactions. As the Redfish server is on BMC, a BMC reboot may resolve the issue. If the issue persists, contact NVIDIA Enterprise Support for further assistance.
X-AUTH POST is Not Successful
After vendor identification, the next step in the Redfish flow is to fetch a token by providing the previously retrieved credentials. Based on the vendor, the authorization protocol can be X-AUTH or O-AUTH. It is important that this transaction succeeds. Otherwise, the rest of Redfish transactions are skipped as they rely on the token received during the authorization step.
Redfish enabled
DHCP Session Start
Press ESC/F2/DEL twice to enter UEFI Menu.
Press ENTER to skip countdown.
3 seconds remain...
2 seconds remain...
1 seconds remain...
0 seconds remain...
** Redfish GET https://192.168.240.1/redfish/v1/, Success
ProcessVendorIdentification: Vendor: Nvidia
** XAUTH POST https://192.168.240.1/redfish/v1/SessionService/Sessions, Success
As seen in the example, X-AUTH transaction is Success. If it is not a success (e.g., Aborted, Unsupported), then contact NVIDIA Enterprise Support for further assistance.
UEFI Redfish Client Fails to Process Request
There are a number of Redfish transactions performed by the UEFI Redfish client for different schemas on respective URL. All these transactions can be visible on the BlueField console logs. For example, if a Redfish request is made to change a BIOS attribute, the respective log should be visible on the BlueField. The log should provide enough information about the request process status (success or failure). If the log is missing or the desired behavior is not seen despite a Success, then contact NVIDIA Enterprise Support with RShim and BlueField console logs as this requires further investigation.
The following BlueField console log shows that the UEFI Redfish client successfully read (GET) the BIOS request, processed it, and responded (PATCH).
DHCP Session Start
Press ESC/F2/DEL twice to enter UEFI Menu.
Press ENTER to skip countdown.
3 seconds remain...
2 seconds remain...
1 seconds remain...
0 seconds remain...
** Redfish GET https://192.168.240.1/redfish/v1/, Success
ProcessVendorIdentification: Vendor: Nvidia
** XAUTH POST https://192.168.240.1/redfish/v1/SessionService/Sessions, Success
** Redfish GET https://192.168.240.1/redfish/v1/Registries/BiosAttributeRegistry/BiosAttributeRegistry, Success
** Redfish GET https://192.168.240.1/redfish/v1/Systems/Bluefield, Success
** Redfish GET https://192.168.240.1/redfish/v1/Systems/Bluefield/Settings, Success
** Redfish PATCH https://192.168.240.1/redfish/v1/Systems/Bluefield, Success
** Redfish PATCH https://192.168.240.1/redfish/v1/Systems/Bluefield, Success
** Redfish GET https://192.168.240.1/redfish/v1/Systems/Bluefield/Bios/Settings, Success
** Redfish PATCH https://192.168.240.1/redfish/v1/Systems/Bluefield/Bios, Success
UEFI Redfish Client Does Not Execute for Another Reason
If UEFI is stuck (in a loop) or crashes before the UEFI Redfish client executes, then Redfish transactions fail to take place, causing the Redfish request to not be processed. The details of the crash or hanged state can be seen on RShim logs or from the console log. Contact NVIDIA Enterprise Support with RShim and BlueField console logs as this requires further investigation.