DOCA Documentation v2.10.0

BlueField Modes of Operation

This document describes the modes of operation available for NVIDIA® BlueField® networking platforms (DPUs or SuperNICs).

BlueField devices feature the following modes of operation:

Mode of Operation

External Host Trust Level

Description

Default on SKUs

Can be Configured to All Other Modes?

NIC Mode

Host-trusted

The Arm cores of BlueField are inactive, and the device functions as an NVIDIA® ConnectX® network adapter.

SuperNIC SKUs default mode

Yes 1

DPU Mode

Host-trusted

The Arm cores of BlueField are active, and the embedded Arm system runs services that manage the NIC resources and data path.

DPU SKUs default mode

Yes

Zero Trust (restricted)

The Arm cores of BlueField are active, and the embedded Arm system runs services to manage the NIC resources and data path while enforcing restrictions on the external host (host isolation).

-

Yes

  1. BFx H20 converged accelerator is limited to NIC Mode only.     

NIC Mode

In NIC Mode, BlueField operates as a ConnectX network adapter for the external host. For BlueField-3, the Arm cores are inactive, while for BlueField-2, the Arm cores are active but non-functional.

Operating in NIC Mode on BlueField-3 reduces power consumption, improves network performance, and minimizes the host memory footprint.

Note

BlueField-3 SuperNIC SKUs are shipped in NIC Mode by default.

Note

Multi-host is not supported when BlueField is operating in NIC Mode.


DPU Mode

In this operation mode, Arm cores active, known also as embedded CPU function ownership (ECPF) mode, is the default mode for the BlueField DPU family of SKUs.

In DPU Mode, the NIC resources and functionality are owned and controlled by the embedded Arm subsystem. All network communication to the host flows through a virtual switch control plane hosted on the Arm cores that manages all networking traffic coming and going from the host.

While working in this mode, the BlueField is the trusted function managed by the data center and host administrator for provisioning, management and orchestration (eg. load network drivers, reset an interface, bring an interface up and down, update the firmware, change the mode of operation on BlueField, etc).

Note

BlueField-2 DPU and BlueField-3 DPU SKUs are shipped in DPU Mode by default.

Note

Socket Direct is not supported when BlueField is operating in DPU Mode.

DPU Mode Architecture

In DPU Mode, the BlueField DPU provides the host system with access to network functions, while the host's capabilities are restricted and managed by the Arm processor within the BlueField.

embedded-mode-version-1-modificationdate-1733837115003-api-v2.png

Traffic Management and Paths

The Embedded Control and Processing Framework (ECPF) controls the NIC's embedded switch (eswitch). All network traffic between the host interface and the network initially passes through the BlueField’s Arm processor via Representors. This path, where traffic is processed by the Arm processor, is referred to as the "slow path".

To improve performance, the Arm processor can define rules in the eswitch through the ECPF, allowing packets to bypass the Arm processor and be processed directly by the eswitch. This is known as the "fast path", which reduces latency and increases throughput by offloading traffic processing from the Arm to the eswitch.

A virtual switch running on the Arm processor may integrate both slow path and fast path functionalities by processing and classifying only the first packet of a new flow. For subsequent packets in the flow, the virtual switch defines eswitch rules, enabling fast path processing for the remainder of the traffic.

Initialization Process

At startup, network access to the host is initially blocked. This restriction remains until the virtual switch running on the Arm processor loads the default out-of-box rules to manage the ECPF on the BlueField. Once these rules are loaded, network traffic to the host is automatically enabled.

Note

The driver on the host system can only be loaded after the driver on the BlueField has been loaded and has completed NIC configuration. Additionally, all Interface Configuration Memory (ICM) is allocated by the ECPF and resides in the BlueField's memory.

InfiniBand in DPU Mode

In DPU Mode, when operating with an InfiniBand network, OpenSM must be executed from the BlueField Arm side rather than the host side. Similarly, InfiniBand management tools such as sminfo, ibdev2netdev, and ibnetdiscover can only be used from the BlueField Arm side and are not accessible from the host side.

DPU Mode in Zero Trust

Zero Trust, also known as Restricted Mode, is a specialized variation of DPU Mode that enhances security by preventing the host system administrator from accessing BlueField from the host side. Once Zero Trust mode is enabled, the BlueField must be fully controlled by the data center administrator via the Arm cores or the BMC connection, rather than through the host.

This mode enforces security and isolation by restricting the host from performing operations that could compromise BlueField. The following operations can be restricted individually in Zero Trust mode:

  • Port ownership – The host cannot assign itself as the port owner

  • Hardware counters – The host is denied access to hardware counters

  • Tracer functionality – The tracer functionality is blocked

  • RShim interface – The RShim interface is disabled

  • Firmware flash – firmware flashing from the host is restricted

Zero Trust mode ensures a robust security boundary between the host and BlueField, making it an ideal configuration for environments requiring strict control and isolation.

Transitioning between operation modes can be achieved through various configuration interfaces as presented in the following table.

Note

Some configuration interfaces may be locked out when operating in Zero Trust (Restricted) Mode.

Interface

From

To

Configuration option available by default, for this interface?

Configuration option can be locked-out 2 for this interface?

External Host command line

DPU Mode

NIC Mode

Yes

Yes

NIC Mode

DPU Mode

Yes

Yes

DPU Mode

DPU Mode with Zero Trust

No

Always blocked

DPU Mode with Zero Trust

DPU Mode

No

Always blocked

DPU Mode with Zero Trust

NIC Mode

Not supported

N/A

Host UEFI Menu

DPU Mode

NIC Mode

Yes

Yes

NIC Mode

DPU Mode

Yes

Yes

DPU Mode

DPU Mode with Zero Trust

No

Always blocked

DPU Mode with Zero Trust

DPU Mode

No

Always blocked

DPU Mode with Zero Trust

NIC Mode

Not supported

N/A

Arm OS command line

DPU Mode

NIC Mode

Yes

No

NIC Mode

DPU Mode

N/A

N/A

DPU Mode

DPU Mode with Zero Trust

Yes

No

DPU Mode with Zero Trust

DPU Mode

Yes

No

DPU Mode with Zero Trust

NIC Mode

Not supported

N/A

Arm UEFI Menu

DPU Mode

NIC Mode

Yes

No

NIC Mode

DPU Mode

Yes

No

DPU Mode

DPU Mode with Zero Trust

No 3

No

DPU Mode with Zero Trust

DPU Mode

No 3

No

DPU Mode with Zero Trust

NIC Mode

Not supported

N/A

DPU-BMC Redfish

DPU Mode

NIC Mode

Yes

No

NIC Mode

DPU Mode

Yes

No

DPU Mode

DPU Mode with Zero Trust

No 3

No

DPU Mode with Zero Trust

DPU Mode

No 3

No

DPU Mode with Zero Trust

NIC Mode

Not supported

N/A

Platform-BMC NC-SI OEM commands

DPU Mode

NIC Mode

Yes

No

NIC Mode

DPU Mode

Yes

No

DPU Mode

DPU Mode with Zero Trust

No 3

No

DPU Mode with Zero Trust

DPU Mode

No 3

No

DPU Mode with Zero Trust

NIC Mode

Not supported

N/A

  1. Blocked or restricted     

  2. Roadmap                                        

Identifying Which Mode BlueField is Currently Operating In

Interface

Command

Response

Using external host command line

Copy
Copied!
            

host> sudo mlxconfig -d /dev/mst/<device> q INTERNAL_CPU_OFFLOAD_ENGINE

Output format:

Copy
Copied!
            

RO     INTERNAL_CPU_OFFLOAD_ENGINE     ENABLED(0)

  • ENABLED(0) means BlueField is running in DPU Mode

  • DISABLED(1) means BlueField is running in NIC Mode

  • RO is marked for read only, and indicates DPU Mode with Zero Trust

Using external host UEFI (HII) menu

Navigate to specific device and select BlueField Internal Cpu Configuration.

bluefield-internal-cpu-configuration-version-1-modificationdate-1733837114770-api-v2.png

Internal Cpu Offload Engine indicates the mode BlueField is currently operating in:

  • Disabled – BlueField is operating in NIC Mode

  • Enabled – BlueField is operating in DPU Mode

internal-cpu-offload-engine-version-1-modificationdate-1733837114550-api-v2.png

Using Arm UEFI (GUI) menu

Access the Arm UEFI menu by pressing the Esc button twi ce on the console and n avigate to Device ManagerSystem ConfigurationBlueField Modes .

The NIC Mode menu indicates the mode BlueField is currently operating in:

  • NicMode – BlueField is operating in NIC Mode

  • DpuMode – BlueField is operating in DPU Mode

image-2024-12-23_18-52-9-version-1-modificationdate-1734972728993-api-v2.png

Using DPU BMC Redfish

Copy
Copied!
            

curl -k -u root:'<PASSWORD>' -X GET https://<bmc_ip>/redfish/v1/Systems/Bluefield/Oem/Nvidia

Read Mode field:

Copy
Copied!
            

{ ... "Mode": <"DpuMode"> or <"NicMode"> ... }

  • "Mode": NicMode – BlueField is operating in NIC Mode

  • "Mode":DpuMode – BlueField is operating in DPU Mode

Using platform BMC NC-SI OEM commands

Get Command = 0x13, Parameter = 0x33

Offset

31:24

23:16

15:8

7:0

0:15

NC-SI OEM Header (OEM Command)

16:19

Mellanox Manufacture ID (IANA) = 0x8119

20:23

Command rev=0x00

MLNX Cmd ID= 0x13

Parameter=0x33

Reserved

24:27

Checksum 31:0

Response format:

Offset

31:24

23:16

15:8

7:0

0:15

NC-SI OEM Header (OEM Command)

16:19

Response Code

Reason Code

20:23

Mellanox Manufacture ID (IANA) = 0x8119

24:27

Command rev=0x00

MLNX Cmd ID= 0x12

Parameter=0x33

Reserved

28:31

Offload engine

32:35

Checksum 31:0

Response field:

Field

Size

Offset in NC-SI Command

Description

Offload engine

1 bit

31.0

0x0: Enabled - DPU Mode

0x1: Disabled - NIC Mode


Changing Mode

For the configuration to take effect, Arm and NIC components must undergo a reset. Power cycle is recommended.

Using External Host Command Line

From Mode

To Mode

Command

DPU Mode

NIC Mode

For BlueField-3:

Copy
Copied!
            

host> sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1

For BlueField-2:

Copy
Copied!
            

host> sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=1 INTERNAL_CPU_ESWITCH_MANAGER=1 INTERNAL_CPU_IB_VPORT0=1 INTERNAL_CPU_OFFLOAD_ENGINE=1

NIC Mode

DPU Mode

For BlueField-3:

Copy
Copied!
            

host> sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=0

For BlueField-2:

Copy
Copied!
            

host> sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0 INTERNAL_CPU_OFFLOAD_ENGINE=0

DPU Mode

DPU Mode with Zero Trust

N/A

DPU Mode with Zero Trust

DPU Mode

N/A

DPU Mode with Zero Trust

NIC Mode

Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode.

Warning
  • Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode.

  • Ensure to disable Zero Trust configuration before transitioning to NIC Mode.

  • Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior.


Using External Host UEFI Menu

From Mode

To Mode

Command

DPU Mode

NIC Mode

bluefield-internal-cpu-configuration-version-1-modificationdate-1733837114770-api-v22.png

internal-cpu-offload-engine-version-1-modificationdate-1733837114550-api-v2.png

Select BlueField Internal Cpu Configuration.

To enable NIC Mode, set Internal Cpu Offload Engine to Disabled.

NIC Mode

DPU Mode

bluefield-internal-cpu-configuration-version-1-modificationdate-1733837114770-api-v22.png

internal-cpu-offload-engine-version-1-modificationdate-1733837114550-api-v2.png

Select BlueField Internal Cpu Configuration.

To enable DPU Mode, set Internal Cpu Offload Engine to Enabled.

DPU Mode

DPU Mode with Zero Trust

N/A

DPU Mode with Zero Trust

DPU Mode

N/A

DPU Mode with Zero Trust

NIC Mode

Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode.

Warning
  • Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode.

  • Ensure to disable Zero Trust configuration before transitioning to NIC Mode.

  • Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior.


Using Arm OS command line

From Mode

To Mode

Command

DPU Mode

NIC Mode

For BlueField-3:

Copy
Copied!
            

bf> sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_OFFLOAD_ENGINE=1

For BlueField-2:

Copy
Copied!
            

bf> sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_PAGE_SUPPLIER=1 INTERNAL_CPU_ESWITCH_MANAGER=1 INTERNAL_CPU_IB_VPORT0=1 INTERNAL_CPU_OFFLOAD_ENGINE=1

NIC Mode

DPU Mode

Not Applicable (Arm OS is not available)

DPU Mode

DPU Mode with Zero Trust

Copy
Copied!
            

bf> $ sudo mlxprivhost -d /dev/mst/<device> r --disable_rshim --disable_tracer --disable_counter_rd --disable_port_owner

DPU Mode with Zero Trust

DPU Mode

Copy
Copied!
            

bf> $ sudo mlxprivhost -d /dev/mst/<device> p

DPU Mode with Zero Trust

NIC Mode

Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode.

Warning
  • Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode.

  • Ensure to disable Zero Trust configuration before transitioning to NIC Mode.

  • Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior.


Using Arm UEFI Menu

From Mode

To Mode

Command

DPU Mode

NIC Mode

  1. Access the Arm UEFI menu by pressing the Esc button twice on the console.

  2. Select Device ManagerSystem ConfigurationBlueField Modes.

  3. Set the NIC Mode option to NicMode to enable NIC Mode.

    nic-mode-version-1-modificationdate-1733837114320-api-v2.png

  4. Exit BlueField Modes and System Configuration and make sure to save the settings.

  5. Exit the UEFI setup using the "reset" option. The configuration is not yet applied and BlueField is expected to boot in DPU Mode.

  6. Issue power cycle to apply new configuration.

NIC Mode

DPU Mode

  1. Access the Arm UEFI menu by pressing the Esc button twice on the console.

  2. Select Device ManagerSystem ConfigurationBlueField Modes.

  3. Set the NIC Mode option to DpuMode to enable DPU Mode.

    nic-mode-version-1-modificationdate-1733837114320-api-v2.png

  4. Exit BlueField Modes and System Configuration and make sure to save the settings.

  5. Exit the UEFI setup using the "reset" option. The configuration is not yet applied and BlueField is expected to boot in NIC Mode.

  6. Issue power cycle to apply new configuration.

DPU Mode

DPU Mode with Zero Trust

Roadmap

DPU Mode with Zero Trust

DPU Mode

Roadmap

DPU Mode with Zero Trust

NIC Mode

Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode.

Warning
  • Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode.

  • Ensure to disable Zero Trust configuration before transitioning to NIC Mode.

  • Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior.


Using DPU-BMC Redfish

From Mode

To Mode

Command

DPU Mode

NIC Mode

Copy
Copied!
            

curl -k -u root:'password' -H 'content-type: application/json' -d '{"Mode": "NicMode"}' -X POST https://bmc_ip/redfish/v1/Systems/Bluefield/Oem/Nvidia/Actions/Mode.Set

To apply configuration, two consecutive Arm reboots are required.

NIC Mode

DPU Mode

Copy
Copied!
            

curl -k -u root:'password' -H 'content-type: application/json' -d '{"Mode": "DpuMode"}' -X POST https://bmc_ip/redfish/v1/Systems/Bluefield/Oem/Nvidia/Actions/Mode.Set

To apply configuration, two consecutive Arm reboots are required.

DPU Mode

DPU Mode with Zero Trust

Roadmap

DPU Mode with Zero Trust

DPU Mode

Roadmap

DPU Mode with Zero Trust

NIC Mode

Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode.

Warning
  • Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode.

  • Ensure to disable Zero Trust configuration before transitioning to NIC Mode.

  • Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior.


Using Platform-BMC NC-SI OEM Commands

From Mode

To Mode

Command

DPU Mode

NIC Mode

Command = 0x12, Parameter = 0x33

Offset

31:24

23:16

15:8

7:0

0:15

NC-SI OEM Header (OEM Command)

16:19

Mellanox Manufacture ID (IANA) = 0x8119

20:23

Command rev=0x00

MLNX Cmd ID= 0x12

Parameter=0x33

Reserved

24:27

Offload engine

28:31

Checksum 31:0

Field

Size

Offset in NC-SI Command

Description

Offload engine

1 bit

27.0

0x0: Enabled – DPU Mode

0x1: Disabled – NIC Mode

For NIC Mode, set the offload engine bit to 0x1.

NIC Mode

DPU Mode

Command = 0x12, Parameter = 0x33

bytes/bits

31:24

23:16

15:8

7:0

0:15

NC-SI OEM Header (OEM Command)

16:19

Mellanox Manufacture ID (IANA) = 0x8119

20:23

Command rev=0x00

MLNX Cmd ID= 0x12

Parameter=0x33

Reserved

24:27

Offload engine

28:31

Checksum 31:0

Field

Size

Offset in NC-SI Command

Description

Offload engine

1 bit

27.0

0x0: Enabled - DPU Mode

0x1: Disabled - NIC Mode

For DPU Mode, set the offload engine bit to 0x0.

DPU Mode

DPU Mode with Zero Trust

Roadmap

DPU Mode with Zero Trust

DPU Mode

Roadmap

DPU Mode with Zero Trust

NIC Mode

Not supported. Move from DPU Mode with Zero Trust to DPU Mode first, and then from DPU Mode to NIC Mode.

Warning
  • Perform a system-level reset when moving from Zero Trust to DPU Mode before configuring NIC Mode.

  • Ensure to disable Zero Trust configuration before transitioning to NIC Mode.

  • Operating in NIC Mode with Zero Trust (Restricted) configuration is not supported and may lead to undefined behavior.

Warning

Separated Host Mode is obsolete for BlueField DPU/SuperNIC SKUs and should not be used, even if it remains visible in some configuration menus or options.

Note

Separated Host Mode is available only for BlueField controller SKUs, where the BlueField Arm OS is the sole CPU/OS in the chassis.

© Copyright 2025, NVIDIA. Last updated on Feb 26, 2025.