HowTo Configure NVIDIA BlueField-3 to NIC Mode on VMware vSphere 8.0
Created on Mar 12, 2024
This guide outlines configuring the NVIDIA® BlueField®-3 DPU to NIC Mode on VMware ESXi version 8.0.x.
A server platform with an adapter card utilizing an NVIDIA BlueField-3 device
Administrator privileges are necessary for installation on the target machine
The setup involves an ESXi 8.0 server with one or more NVIDIA BlueField-3 adapter cards.
The setup includes:
VMware ESXi server, vSphere Cluster and vCenter install and configuration is out of the scope of this post.
On a Dell server:
Enter the server's BIOS menu.
Click Device Settings:
Select the desired BlueField device(s).
Verify that the Chip Type is a BlueField device (BlueField-3) and the VMware Distributed Services Engine (DPU) is Disabled.
Click BlueField Internal Cpu Configuration.
Set Internal Cpu Offload Engine setting to Disable (disable is NIC mode).
Repeat steps b-d for all necessary BlueField devices in the server.
Save BIOS settings.
Power cycle the server.
On a Lenovo server:
Enter the server's BIOS menu.
Select the desired BlueField device(s).
Verify that the Chip Type is a BlueField device (e.g., BlueField-3) and the VMware Distributed Services Engine (DPU) is Disabled.
Click BlueField Internal Cpu Configuration.
Set Internal Cpu Offload Engine setting to Disable (i.e., NIC mode).
Repeat steps b-d for all necessary BlueField devices in the server.
Save BIOS settings.
Power cycle the server.
Enable SSH from the vSphere Client to the desired ESXi server.
Access the remote ESXi Shell.
Enter Maintenance Mode with ESXCLI.
Locate the physical NVIDIA network device.
ESXi console
[root@qa-esxio-lnv-06:~] lspci |grep -i mel
Example output:
ESXi console
0000:38:00.0 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7] 0000:38:00.1 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7] 0000:38:00.2 Ethernet controller: Mellanox Technologies MT42822 Family [BlueField Auxiliary Comm Channel] 0000:38:00.3 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface 0000:a8:00.0 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7] 0000:a8:00.1 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7] 0000:a8:00.2 Ethernet controller: Mellanox Technologies MT42822 Family [BlueField Auxiliary Comm Channel] 0000:a8:00.3 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface
Check the NIC link status from the ESXi service console.
ESXi console
[root@qa-esxio-lnv-06:~] esxcli network nic list
Example output :
Query NVIDIA device firmware.
ESXi console
[root@qa-esxio-lnv-06:~] esxcli network nic list
Example output :
WarningIf the following is printed:
ESXi console
/opt/mellanox/bin/mlxfwmanager -E- No devices found or specified, mst might be stopped, run 'mst start' to load MST modules
Then the device is in DPU mode, so MFT does not recognize it.
Verify whether each device is in DPU mode by ensuring that
INTERNAL_CPU_OFFLOAD_ENGINE
is enabled.ESXi console
[root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 q | grep -i offload INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0) [root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf1 q | grep -i offload INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0)
Enter device recovery mode and reboot the server.
ESXi console
[root@qa-esxio-lnv-06:~] esxcli system module parameters set -m nmlx5_core -p mst_recovery=1 [root@qa-esxio-lnv-06:~] reboot -f
Configure the desired device to NIC mode.
ESXi console
[root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 set INTERNAL_CPU_OFFLOAD_ENGINE=1
Example output
Power cycle the system after switching to NIC mode.
Verify that
INTERNAL_CPU_OFFLOAD_ENGINE
isDISABLED
after power cycling the system.ESXi console
[root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 q | grep -i offload
Example output:
We will add a DPU to a VM as a passthrough PCI device and will connect to DPU UEFI to configure DPU in NIC Mode.
Change the management interfaces to passthrough.
Create a virtual machine (VM) and assign the PCIe passthrough devices.
Power on the VM.
Deploy Linux OS from PXE.
Install the driver to get RShim and MFT:
NoteInstalling
minicom
is necessary.For RedHat, run:
ESXi console
yum install minicom yum install pv yum install gcc-gfortran
For Ubuntu, run:
ESXi console
apt install minicom apt install pv
For SLES, use
yast
orzypper
. For example:ESXi console
zypper -i minicom zypper -i pv
Connect to DPU via minicom:
InfoIf you have multiple DPUs on your setup, this means that you have one RShim per DPU (e.g.,
rshim0
for DPU#1,rshim1
for DPU#2, etc.)Open as many threads on VNC to get access to each DPU using the following formula:
ESXi console
minicom -D /dev/rshim<N>/console
Where
<N>
represents the index associated with each DPU.Open a new SSH connection to the same VM and reset each DPU from the host:
ESXi console
echo "SW_RESET 1" > /dev/rshim<N>/misc
Navigate to the console for your DPU and press the
ESC
button key on the keyboard.Define a new password for the DPU.
InfoThe default password for the UEFI BIOS is
bluefield
.Select
Device Manager
.Select
Network Device List
.Select the network device that presents the uplink (i.e., select the device with the uplink MAC address).
Select
NVIDIA Network adapter - $<uplink-mac>
.Select
BlueField Internal Cpu Configuration
.Set NIC mode by setting
Internal CPU Offload Engine
toDisabled
.InfoTo set DPU mode, set
Internal CPU Offload Engine
toEnabled
.Power cycle the system.
![]()
|
Boris Kovalev Boris Kovalev has worked for the past several years as a Solutions Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the Mellanox Documents website. |
This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.