image image image image image

Warning

YOU ARE VIEWING AN OUTDATED DOCUMENT.

On This Page

The BlueField® SmartNIC has several modes of operation:

  • Separated host mode (symmetric model)
  • Embedded function (ECPF) ownership where the embedded Arm system controls the NIC resources and data path
  • Isolated mode, which is an extension of the ECPF ownership with additional restrictions on the host side

Each one of the modes can be applied individually to each one of the physical ports of the SmartNIC.

Separated Host

This is the default configuration for the BlueField SmartNIC. In this mode, the ECPF and the function exposed to the host are both symmetric. Each one of those functions has its own MAC address and is able to send and receive Ethernet and RDMA over Converged Ethernet (RoCE) traffic. 

There is no dependency between the two functions. They can operate simultaneously or separately. The host can communicate with the embedded function as two separate hosts, each with its own MAC and IP addresses (configured as a standard interface). RDMA connection between the 2 interfaces is supported as well.

There is an equal bandwidth share between the two functions.

The limitations of this mode are as follows:

  • Switchdev (virtual switch offload) mode is not supported on either of the functions
  • SR-IOV is only supported on the host side

Separated host mode is configured per port.

Embedded CPU Function Ownership Mode (SmartNIC Mode)

In ECPF mode, the NIC resources and functionality are owned and controlled by the embedded Arm subsystem. A network function is still exposed to the host, but it has limited privileges. In particular:

  1. The driver on the host side can only be loaded after the driver on the embedded side has loaded and completed NIC configuration.
  2. All ICM (Interface Configuration Memory) is allocated by the ECPF and resides in the embedded host memory.
  3. The ECPF controls and configures the NIC embedded switch which means that traffic to and from the host interface always lands on the Arm side.

There are two ways to pass traffic to the host interface: Either using representors to forward traffic to the host (every packet to/from the host would be handled also by the network interface on the embedded Arm side), or push rules to the embedded switch which allows and offloads this traffic.

Configuring Embedded CPU Function Ownership Mode from Separated Host Mode

To enable this mode:

  1. Start MST (Mellanox Software Tools) driver set service: 

    mst start
  2. Identify the MST device: 

    mst status -v

    Output example: 

    MST modules:
    ------------
        MST PCI module is not loaded
        MST PCI configuration module loaded
    PCI devices:
    ------------
    DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
    BlueField(rev:0)        /dev/mst/mt41682_pciconf0.1   37:00.1   mlx5_1          net-ens1f1                0
    
    BlueField(rev:0)        /dev/mst/mt41682_pciconf0     37:00.0   mlx5_0          net-ens1f0                0
  3. Run the following commands on the Arm:

    mlxconfig -d /dev/mst/mt41682_pciconf0 s INTERNAL_CPU_MODEL=1
    mlxconfig -d /dev/mst/mt41682_pciconf0.1 s INTERNAL_CPU_MODEL=1
  4. Power cycle the server. 

Embedded CPU mode is configured per port.

Configuring Separated Host Mode from Embedded CPU Function Ownership Mode 

On the X86 host, follow this procedure:

  1. Enable separated host mode. Run:

    mst start
    mlxconfig -d /dev/mst/mt41682_pciconf0 s INTERNAL_CPU_MODEL=0
  2. Power cycle.
  3. Verify configuration. Run:

    mst start
    mlxconfig -d /dev/mst/mt41682_pciconf0 q | grep -i model

Restricting SmartNIC Host

By default, the host server has the same permissions as the Arm cores. 

For security and isolation purposes, it is possible to restrict the host from performing operations that can compromise the SmartNIC. Once the SmartNIC host is set in restricted mode, the following operations are restricted:

  • Port ownership  the host cannot assign itself as port owner
  • Hardware counters – the host does not have access to hardware counters
  • Tracer functionality is blocked
  • RShim interface is blocked
  • FW flash is restricted

Enabling Host Restriction

  1. Start the MST service.
  2. Set restricted mode: 

    mlxprivhost -d /dev/mst/mt41682_pciconf0 r --disable_rshim --disable_tracer --disable_counter_rd --disable_port_owner

If RShim is disabled, power cycle is required.

Disabling Host Restriction

Set back to privileged mode: 

mlxprivhost -d /dev/mst/mt41682_pciconf0 p

The configuration takes effect immediately. System reboot is not required.