DPU Mode Installation

The following diagram illustrates the sequence of events and actions from first time power-up of the NVIDIA® BlueField® networking platform (DPU or SuperNIC) in the data center environment through provisioning and maintenance.

At the end of this procedure, the BlueField should be configured with an IP address, all required settings, has up-to-date software component versions, and is ready to use.

The BlueField SoC boots to the UEFI BIOS and DHCP DISCOVER is sent

1. BlueField SoC runs UEFI/PXE which sends a DHCP DISCOVER over the 1GbE OOB interface, including vendor class ("NVIDIA/BF/PXE") for BlueField SoC (to allow customer's server to differentiate between BlueField SoC and BlueField BMC), and MAC for identification and discovery. See appendix "BlueField DHCP Discover" for more information.

2. A customer's DHCP server inspects the MAC address and the vendor class, allocates IP, and continues the standard DHCP.

3. DHCP server updates RMC of the new BlueField discovered with detailed information (e.g., MAC, IP address, vendor class).

BlueField BMC issues DHCP DISCOVER over the 1GbE OOB interface, including vendor class ("NVIDIA/BF/BMC") for BlueField-BMC, and MAC for identification and discovery. Example of BlueField BMC DHCP DISCOVER packet structure (note "NVIDIA/BF/BMC" in line 13):

            

            
root@bf-bmc:~# 18:18:10.563269 IP (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 320)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from b8:3f:d2:ca:4b:26 (oui Unknown), length 292, xid 0xfc2acdec, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address b8:3f:d2:ab:cd:ef (oui Unknown)
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether b8:3f:d2:ab:cd:ef
Parameter-Request (55), length 9:
Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), Static-Route (33), NTP (42), Unknown (120)
Classless-Static-Route (121)
MSZ (57), length 2: 576
Hostname (12), length 7: "bf-bmc" Vendor-Class (60), length 13: "NVIDIA/BF/BMC" END (255), length 0
18:18:10.565261 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 353)
(example) dhcp01.XX.YY > ldev-platform-13-043-bmc.bootpc: [no cksum] BOOTP/DHCP, Reply, length 325, hops 1, xid 0xfc2acdec, secs 1, Flags [none] (0x0000)
(example) Your-IP ldev-platform-13-043-bmc.XX.YY
(example) Server-IP l-pxe02.XX.YY
Gateway-IP 10.237.0.255
Client-Ethernet-Address b8:3f:d2:ab:cd:ef (oui Unknown)
file "pxelinux.0" Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: (example) dhcp01.XX.YY
Lease-Time (51), length 4: 43200
Subnet-Mask (1), length 4: 255.255.0.0
Default-Gateway (3), length 4
(example) GW.XX.YY
Hostname (12), length 24: "ldev-platform-13-043-bmc" Domain-Name (15), length 13: "<local domain name>" NTP (42), length 4: (example) NTP.XX.YY
END (255), length 0
18:18:10.565261 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 353)
dhcp01.XX.YY > ldev-platform-13-043-bmc.<local domain name>: [no cksum] BOOTP/DH
        

1. DHCP server inspects the MAC address and the vendor class, allocates IP and continues the standard DHCP flow.

2. DHCP server updates RMC of the new BlueField BMC discovered with detailed information: MAC, IP address, vendor classes, etc.

To communicate with the BlueField BMC, change the default password (0penBmc) by sending the following Redfish schema to the BlueField BMC:

            

            
curl -k -u root:0penBmc -H "Content-Type: application/json" -X PATCH https://<BF-BMC-IP>/redfish/v1/AccountService/Accounts/root -d '{"Password" : "<user-password>"}'
        

Where <BF-BMC-IP> is the IP address for the BlueField BMC (e.g., 10.10.1.2), and <user-password> is the chosen password to log into the BlueField BMC with root privileges.

For example:

            

            
[redfish_scripts] $ curl -k -u root:0penBmc -H "Content-Type: application/json" -X PATCH https://<BF-BMC-IP>/redfish/v1/AccountService/Accounts/root -d '{"Password" : "HelloNvidia3D!"}'
Response:
{
  "@Message.ExtendedInfo": [
    {
      "@odata.type": "#Message.v1_1_1.Message",
      "Message": "The request completed successfully.",
      "MessageArgs": [],
      "MessageId": "Base.1.15.0.Success",
      "MessageSeverity": "OK",
      "Resolution": "None"
    }
  ]
}
        

Upgrade the BlueField firmware components (i.e., ATF, UEFI, NIC-firmware, DPU BMC, and ERoT) and the BSP using the BFB image by following the instructions in section "Installing BFB".

Verify BlueField BSP, BlueField BMC and BlueField NIC firmware versions are up to date.

1. Use the Redfish FirmwareInventory schema over the 1GbE OOB interface to the BlueField's BMC:

   Collapse Source

   Copy

   Copied!

               
   
               
   [redfish_scripts] $ curl -k -u root:<password> -H "Content-Type: application/octet-stream" -X GET https://<BF-BMC-IP>/redfish/v1/UpdateService/FirmwareInventory
   {
     "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory",
     "@odata.type": "#SoftwareInventoryCollection.SoftwareInventoryCollection",
     "Members": [
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/9f7ec75a_BMC_Firmware"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/Bluefield_FW_ERoT"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_ATF"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_BOARD"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_BSP"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_NIC"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_NODE"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_OFED"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_OS"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_SYS_IMAGE"
       },
       {
         "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_UEFI"
       }
     ],
     "Members@odata.count": 11,
     "Name": "Software Inventory Collection"
   }
           

   Response example for DPU_ATF:

   Copy

   Copied!

               
   
               
   > curl -k -u root:<password> -H "Content-Type: application/octet-stream" -X GET https://<BF-BMC-IP>/redfish/v1/UpdateService/FirmwareInventory/DPU_ATF
   {
     "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_ATF",
     "@odata.type": "#SoftwareInventory.v1_4_0.SoftwareInventory",
     "Description": "Host image",
     "Id": "DPU_ATF",
     "Members@odata.count": 1,
     "Name": " "Software Inventory",
     "RelatedItem": [
       {
         "@odata.id": "/redfish/v1/Systems/Bluefield/Bios"
       }
     ],
     "SoftwareId": "",
     "Status": {
       "Health": "OK",
       "HealthRollup": "OK",
       "State": "OK",
     },
     "Updateable": true,
     "Version": "v2.2(release):4.0.2-33-gd9f4ad5"
           

   Info

   This request may also be used to query some of the other previously mentioned components (e.g., 9f7ec75a_BMC_Firmware, Bluefield_FW_ERoT).

2. If the versions are not as expected, upgrade as needed:

   1. Download the latest DOCA (BFB file) versions from the downloader at the bottom of the DOCA product page.

   2. DOCA (BFB) upgrade options (upgrading UEFI, ATF, Arm OS, NIC firmware components):

      • Recommended—BFB upgrade from remote management controller using Redfish UpdateService schema over 1GbE to BlueField BMC:

        Copy

        Copied!

                    
        
                    
        export token=`curl -k -H "Content-Type: application/json" -X POST https://<bmc_ip>/login -d '{"username":"root", "password":"<password>"}' | grep token | awk '{print $2;}' | tr -d '"'`
                

        For more information on deploying BlueField software from the BMC, refer to the "Deploying BlueField Software Using BFB from BMC" page of the NVIDIA BlueField BSP document.

1. Get the BlueField's BMC MAC address using the following Redfish command over the 1GbE OOB port to the BlueField BMC:

   Collapse Source

   Copy

   Copied!

               
   
               
   curl -k -u root:<password> -H 'Content-Type: application/json' -X GET https://<BF-BMC-IP>/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0
    {
      "@odata.id": "/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0",
      "@odata.type": "#EthernetInterface.v1_6_0.EthernetInterface",
      "DHCPv4": {
        "DHCPEnabled": true,
        "UseDNSServers": true,
        "UseDomainName": true,
        "UseNTPServers": true
      },
      "DHCPv6": {
        "OperatingMode": "Stateful",
        "UseDNSServers": true,
        "UseDomainName": true,
        "UseNTPServers": true
      },
      "Description": "Management Network Interface",
      "FQDN": "dpu-bmc",
      "HostName": "BlueField-bmc",
      "IPv4Addresses": [
        {
          "Address": "10.237.40.179",
          "AddressOrigin": "DHCP",
          "Gateway": "0.0.0.0",
          "SubnetMask": "255.255.0.0"
        }
      ],
      "IPv4StaticAddresses": [],
      "IPv6AddressPolicyTable": [],
      "IPv6Addresses": [
        {
          "Address": "fdfd:fdfd:10:237:966d:aeff:fe17:9f5f",
          "AddressOrigin": "DHCPv6",
          "AddressState": null,
          "PrefixLength": 64
        },
        {
          "Address": "fe80::966d:aeff:fe17:9f5f",
          "AddressOrigin": "LinkLocal",
          "AddressState": null,
          "PrefixLength": 64
        }
      ],
      "IPv6DefaultGateway": "fe80::445b:ed80:5f97:8900",
      "IPv6StaticAddresses": [],
      "Id": "eth0",
      "InterfaceEnabled": true,
      "LinkStatus": "LinkUp",
      "MACAddress": "94:6d:ae:17:9f:5f",
      "MTUSize": 1500,
      "Name": "Manager Ethernet Interface",
      "NameServers": [
        "fdfd:fdfd:7:77:250:56ff:fe8b:e4f9"
      ],
      "SpeedMbps": 0,
      "StaticNameServers": [],
      "Status": {
        "Health": "OK",
        "HealthRollup": "OK",
        "State": "Enabled"
      },
      "VLANs": {
        "@odata.id": "/redfish/v1/Managers/Bluefield_BMC/EthernetInterfaces/eth0/VLANs"
      }
    }
           

2. Get the BlueField's high-speed port's MAC addresses using the following Redfish command over the 1GbE OOB port to the BlueField BMC:

   Copy

   Copied!

               
   
               
   curl -k -u root:<password> -H "Content-Type: application/octet-stream" -X GET https://<bmc_ip>/redfish/v1/Chassis/Card1/NetworkAdapters/NvidiaNetworkAdapter/NetworkDeviceFunctions/eth0f0
   {
     "@odata.id": "/redfish/v1/Chassis/Card1/NetworkAdapters/NvidiaNetworkAdapter/NetworkDeviceFunctions/eth0f0",
     "@odata.type": "#NetworkDeviceFunction.v1_9_0.NetworkDeviceFunction",
     "Ethernet": {
       "MACAddress": "02:b1:b6:12:39:05",
       "MTUSize": 1500
     },
     "Id": "eth0f0",
     "Links": {
       "OffloadSystem": {
         "@odata.id": "/redfish/v1/Systems/Bluefield"
       },
       "PhysicalPortAssignment": {
         "@odata.id": "/redfish/v1/Chassis/Card1/NetworkAdapters/NvidiaNetworkAdapter/Ports/eth0"
       }
     },
     "Name": "NetworkDeviceFunction",
     "NetDevFuncCapabilities": [
       "Ethernet"
     ],
     "NetDevFuncType": "Ethernet"
   }
           

Unless it is explicitly desired for the host to be trusted, make sure to disable the host PCIe RShim to protect the BlueField from potential security threats from the host:

1. Use Redfish BIOS settings schema over the 1GbE OOB to the BlueField BMC:

   Copy

   Copied!

               
   
               
   curl -k -X PATCH -d '{"Attributes":{"Internal CPU Model": "Restricted"}}' -u root:<password> https://<BF-BMC-IP>/redfish/v1/Systems/<SystemID>/Bios/Settings | python3 -m json.tool
           

   The available BlueField host privilege levels are Restricted and Privileged. The default is Privileged, where the host has access to BlueField.

2. Change the privilege level to Restricted.

As part of the default settings of the BlueField, UEFI Secure Boot is enabled and requires no special configuration to use it with the bundled Ubuntu OS shipped with the BlueField device. Disabling UEFI Secure Boot may be necessary when running an unsigned Arm OS image, such as a customer OS.

To disable secure boot using the Redfish SecureBoot schema over 1GbE to BlueField BMC, follow the command in section "Setting Secure Boot State".