SoC Management Interface (RShim)

The SoC management interface—formerly known as RShim—enables an external agent such as the host CPU or BMC to operate and monitor the NVIDIA® BlueField® platform (DPU or SuperNIC). This interface supports provisioning BlueField, resetting Arm cores, and retrieving logs.

For installation and upgrade instructions, see "Updating Repo Package on Host Side".

Configuration File

The configuration file for the SoC management interface is located at /etc/rshim.conf and includes the parameters shown in the table below:

To control which RShim index maps to which device, edit the configuration file as follows:

            

            
# Uncomment the 'rshim<N>' line to define the mapping.
#
# Format:
# rshim<N>    pci-device
rshim0        pcie-0000:21:00.2
rshim1        pcie-0000:81:00.2
 
# Ignored devices
# Uncomment the 'none' line to exclude devices.
#
#none          usb-1-1.4
#none          pcie-lf-0000:84:00.0
        

            

            
systemctl restart rshim
        

On the host OS, BlueField registers a “DMA controller” used for management over PCIe. You can verify its presence with the following command:

            

            
#  lspci -d 15b3: | grep 'SoC Management Interface'
        

Example output:

            

            
27:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01)
        

A special driver—currently named RShim—must be installed and running on the host to expose the SoC management interface. This driver is automatically installed as part of the DOCA package.

For installation details, refer to the section Install RShim on Host.

When RShim is operating correctly, the host creates:

• A sysfs device node: /dev/rshim0/*

• A virtual Ethernet interface: tmfifo_net0

To check the driver status:

            

            
systemctl status rshim
        

Example output:

            

            
# systemctl status rshim 
● rshim.service - rshim driver for BlueField SoC
     Loaded: loaded (/lib/systemd/system/rshim.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-05-31 14:57:07 IDT; 1 day 1h ago
       Docs: man:rshim(8)
    Process: 90322 ExecStart=/usr/sbin/rshim $OPTIONS (code=exited, status=0/SUCCESS)
   Main PID: 90323 (rshim)
      Tasks: 11 (limit: 76853)
     Memory: 3.3M
     CGroup: /system.slice/rshim.service
             └─90323 /usr/sbin/rshim
        

Log entries confirm device initialization:

            

            
rshim[90323]: Probing pcie-0000:a3:00.2(vfio)
rshim[90323]: Create rshim pcie-0000:a3:00.2
rshim[90323]: rshim pcie-0000:a3:00.2 enable
rshim[90323]: rshim0 attached
        

Virtual Ethernet Interface

The RShim driver exposes a virtual Ethernet device, tmfifo_net0, which acts as a peer-to-peer tunnel between the host and BlueField OS. On the BlueField side, BFB images configure a static IP address of 192.168.100.2/30. The host side must be configured manually.

To configure the host IP:

            

            
ip addr add dev tmfifo_net0 192.168.100.1/30
        

At this point, to log into the BlueField OS from the host, run the following for example:

            

            
ssh ubuntu@192.168.100.2
        

SoC Management Interface Driver Support for Multiple BlueFields

If multiple BlueField devices are installed on the same host, each will have:

• A unique RShim interface at /dev/rshim<N>

• A corresponding virtual Ethernet device: tmfifo_net<N>

There are two ways to manage multiple tmfifo_net<N> interfaces:

• Bridge mode

  1. Add all tmfifo_net<N> interfaces to a bridge.

  2. Assign a single IP to the bridge.

  3. Each BlueField is configured with a unique IP in the bridge's subnet.

• Point-to-point mode:

  1. Assign a unique subnet IP to each tmfifo_net<N> interface.

  2. Each BlueField is assigned a corresponding IP in that subnet.

Each tmfifo_net<N> interface on the host must have a unique MAC address. To configure it manually:

            

            
ifconfig tmfifo_net0 192.168.100.1/24 hw ether 02:02:02:02:02:02
        

It can also be configured persistently using a custom udev rule.

Similarly, each BlueField-side tmfifo_net interface must be configured with:

• A unique MAC address

• A distinct IP (default is 192.168.100.2 on all devices)

These must be set manually or via BlueField customization scripts during installation.

Multi-board Management Example

This example deals with two BlueField devices installed on the same server (the process is similar for more devices). The example assumes that the RShim package has been installed on the host server.

Configuring Management Interface on Host

1. Create a bf_tmfifo interface under /etc/sysconfig/network-scripts. Run:

   Copy

   Copied!

               
   
               
   vim /etc/sysconfig/network-scripts/ifcfg-br_tmfifo
           

2. Inside ifcfg-br_tmfifo, insert the following content:

   Copy

   Copied!

               
   
               
   DEVICE="br_tmfifo"
   BOOTPROTO="static"
   IPADDR="192.168.100.1"
   NETMASK="255.255.255.0"
   ONBOOT="yes"
   TYPE="Bridge"
           

3. Create a configuration file for the first BlueField, tmfifo_net0. Run:

   Copy

   Copied!

               
   
               
   vim /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0
           

4. Inside ifcfg-tmfifo_net0, insert the following content:

   Copy

   Copied!

               
   
               
   DEVICE=tmfifo_net0
   BOOTPROTO=none
   ONBOOT=yes
   NM_CONTROLLED=no
   BRIDGE=br_tmfifo
           

5. Create a configuration file for the second BlueField, tmfifo_net1. Run:

   Copy

   Copied!

               
   
               
   DEVICE=tmfifo_net1
   BOOTPROTO=none
   ONBOOT=yes
   NM_CONTROLLED=no
   BRIDGE=br_tmfifo
           

6. Create the rules for the tmfifo_net interfaces. Run:

   Copy

   Copied!

               
   
               
   vim /etc/udev/rules.d/91-tmfifo_net.rules
           

7. Restart the network for the changes to take effect. Run:

   Copy

   Copied!

               
   
               
   # /etc/init.d/network restart
   Restarting network (via systemctl):            [  OK  ]
           

Configuring BlueField Side

BlueField devices arrive with the following factory default configurations for tmfifo_net0.

Therefore, if you are working with more than one BlueField, you must change the default MAC and IP addresses.

Updating RShim Network MAC Address

1. Use a Linux console application (e.g. screen or minicom) to log into each BlueField. For example:

   Copy

   Copied!

               
   
               
   # sudo screen /dev/rshim<0|1>/console 115200
           

2. Create a configuration file for tmfifo_net0 MAC address. Run:

   Copy

   Copied!

               
   
               
   # sudo vi /etc/bf.cfg
           

3. Inside bf.cfg, insert the new MAC:

   Copy

   Copied!

               
   
               
   NET_RSHIM_MAC=00:1a:ca:ff:ff:03
           

4. Apply the new MAC address. Run:

   Copy

   Copied!

               
   
               
   sudo bfcfg
           

5. Repeat this procedure for the second BlueField (using a different MAC address).

   Info

   Arm must be rebooted for this configuration to take effect. It is recommended to update the IP address before you do that to avoid unnecessary reboots.

Updating IP Address

For Ubuntu:

1. Access the file 50-cloud-init.yaml and modify the tmfifo_net0 IP address:

   Copy

   Copied!

               
   
               
   sudo vim /etc/netplan/50-cloud-init.yaml
    
                   tmfifo_net0:
                        addresses:
                        - 192.168.100.2/30    ===>>>    192.168.100.3/30
           

2. Reboot the Arm. Run:

   Copy

   Copied!

               
   
               
   sudo reboot
           

3. Repeat this procedure for the second BlueField (using a different IP address).

   Info

   Arm must be rebooted for this configuration to take effect. It is recommended to update the MAC address before you do that to avoid unnecessary reboots.

For CentOS:

1. Access the file ifcfg-tmfifo_net0. Run:

   Copy

   Copied!

               
   
               
   # vim /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0
           

2. Modify the value for IPADDR:

   Copy

   Copied!

               
   
               
   IPADDR=192.168.100.3
           

3. Reboot the Arm. Run:

   Copy

   Copied!

               
   
               
   reboot
           

   Or perform netplan apply.

4. Repeat this procedure for the second BlueField (using a different IP address).

   Info

   Arm must be rebooted for this configuration to take effect. It is recommended to update the MAC address before you do that to avoid unnecessary reboots.

Permanently Changing Arm-side MAC Address

The default MAC address is 00:1a:ca:ff:ff:01. It can be changed using ifconfig or by updating the UEFI variable as follows:

1. Log into Linux from the Arm console.

2. Run:

   Copy

   Copied!

               
   
               
   $ "ls /sys/firmware/efi/efivars".
           

3. If not mounted, run:

   Copy

   Copied!

               
   
               
   $ mount -t efivarfs none /sys/firmware/efi/efivars
   $ chattr -i /sys/firmware/efi/efivars/RshimMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
   $ printf "\x07\x00\x00\x00\x00\x1a\xca\xff\xff\x03" > \
     /sys/firmware/efi/efivars/RshimMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
           

The printf command sets the MAC address to 00:1a:ca:ff:ff:03 (the last six bytes of the printf value). Either reboot the device or reload the tmfifo driver for the change to take effect.

The MAC address can also be updated from the server host side while the Arm-side Linux is running:

1. Enable the configuration. Run:

   Copy

   Copied!

               
   
               
   # echo "DISPLAY_LEVEL 1" > /dev/rshim0/misc
           

2. Display the current setting. Run:

   Copy

   Copied!

               
   
               
   # cat /dev/rshim0/misc
   DISPLAY_LEVEL   1 (0:basic, 1:advanced, 2:log)
   BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
   BOOT_TIMEOUT    300 (seconds)
   DROP_MODE       0 (0:normal, 1:drop)
   SW_RESET        0 (1: reset)
   DEV_NAME        pcie-0000:04:00.2
   DEV_INFO        BlueField-2(Rev 1)
   PEER_MAC        00:1a:ca:ff:ff:01 (rw)
   PXE_ID          0x00000000 (rw)
   VLAN_ID         0 0 (rw)
           

3. Modify the MAC address. Run:

   Copy

   Copied!

               
   
               
   $ echo "PEER_MAC  xx:xx:xx:xx:xx:xx" > /dev/rshim0/misc
           

            

            
bfb-install –r rshim<N> -b <bfb> [-c bf.cfg]
        

            

            
screen /dev/rshim<N>/console 115200 
minicom -D /dev/rshim<N>/console 
        

            

            
ip addr add dev tmfifo_net<N> 192.168.100.1/30
        

            

            
ssh -6  user@fe80::21a:caff:feff:ff01%tmfifo_net<N>
        

            

            
cat /dev/rshim<N>/misc
        

            

            
$ sudo cat /dev/rshim1/misc
DISPLAY_LEVEL   2 (0:basic, 1:advanced, 2:log)
BF_MODE         NIC mode
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT    300 (seconds)
USB_TIMEOUT     40 (seconds)
DROP_MODE       0 (0:normal, 1:drop)
SW_RESET        0 (1: reset)
DEV_NAME        pcie-0000:b1:00.2
DEV_INFO        BlueField-3(Rev 1)
OPN_STR         9009D3B600CVAA
UP_TIME         595371(s)
SECURE_NIC_MODE 0 (0:no, 1:yes)
FORCE_CMD       0 (1: send Force command)
---------------------------------------
             Log Messages
---------------------------------------
 INFO[MISC]: Linux up
 INFO[MISC]: DPU is ready
        

            

            
echo "<command string>" > /dev/rshim<N>/misc
        

            

            
echo "SW_RESERT 1" > /dev/rshim<N>/misc
        

            

            
echo DEV_NAME > /dev/rshim0/misc
        

RShim Misc Interface

            

            
echo "SW_RESET 1" > /dev/rshim0/misc
        

The bf.cfg file contains configuration that can be pushed to customize the installation of the BFB. See "Customizing BlueField Software Deployment" for more information.

The RShim interface may be owned by the BlueField BMC or the host (Windows or Linux). In situations where users do not have access to the host, they would want to transfer RShim ownership to the BMC.

Assuming that /dev/rshim0 is the BlueField requesting ownership over the RShim interface, ownership may be transferred to the BlueField BMC by running the following command from the BMC console:

1. Confirm RShim is not attached to BMC:

   Copy

   Copied!

               
   
               
   root@dpu-bmc:~# systemctl status rshim
           

   This show messages like another backend already attached or rshim0 entering drop mode.

   Info

   If RShim is already attached or RShim is not in drop mode, the ownership transfer command (-F) will fail.

2. Create a directory for the RShim systemd override file if it does not already exist:

   Copy

   Copied!

               
   
               
   root@dpu-bmc:~# mkdir -p /etc/systemd/system/rshim.service.d
           

3. Create the override file:

   Copy

   Copied!

               
   
               
   root@dpu-bmc:~# cat > /etc/systemd/system/rshim.service.d/override.conf
   [Service]
   Environment="OPTIONS=-F"
           

4. Press Ctrl-D to save the file.

5. Reload the systemd manager configuration and restart RShim service.

   Copy

   Copied!

               
   
               
   root@dpu-bmc:~# systemctl daemon-reload
   root@dpu-bmc:~# systemctl restart rshim
   root@dpu-bmc:~# systemctl status rshim
   … …
   Dec 10 20:58:57 dpu-bmc rshim[20561]: rshim0 received ownership transfer ack
   Dec 10 20:59:00 dpu-bmc rshim[20561]: rshim0 regained ownership successfully
           

   RShim ownership is transferred to BMC.

6. Clean up the systemd override file:

   Copy

   Copied!

               
   
               
   root@dpu-bmc:~# rm /etc/systemd/system/rshim.service.d/override.conf
   root@dpu-bmc:~# systemctl daemon-reload
           

Controlling SoC management interface could also be done from the platform BMC using NC-SI OEM command over I²C.

RShim interface could be enabled, disabled, or locked:

• Enabled – RShim over PCIe is supported and ready to use

• Locked – RShim over PCIe is supported, but cannot currently be used. It may be re-activated using the enable command, without the need to perform reset

• Disabled – RShim over PCIe is not supported. You must re-enable it and power cycle the host.

Set Host PCIe RShim Access to BlueField DPU CPU (Command=0x12, Parameter=0x19)

This package command allows a trusted platform BMC to configure the non-volatile enablement state of external Host PCIe RShim access to the BlueField DPU's embedded CPU.

Set Host PCIe RShim Access to BlueField DPU CPU Format

Set Host PCIe RShim Access to BlueField DPU CPU Command Parameters

Set Host PCIe RShim access to BlueField DPU CPU Response

The BlueField DPU always receives and responds to this command when the Package ID matches.

If the command is issued by an untrusted platform BMC, it fails with reason code 0x7FFF (Unsupported command).

Set Host RShim Access to BlueField DPU CPU Response Format

Set Dynamic Host PCIe RShim access to BlueField DPU CPU (Command=0x12, Parameter=0x1B)

This package command allows a trusted platform BMC to configure the runtime (volatile) enablement state of Host PCIe RShim access to the BlueField DPU CPU—without requiring a DPU reset.

The setting applied by this command is volatile and will be overridden upon reset by the non-volatile configuration.

Set Dynamic Host PCIe RShim Access to BlueField DPU CPU Format

Set Dynamic Host PCIe RShim Access to BlueField DPU CPU Command Parameters

Set Dynamic Host RShim Access to BlueField DPU CPU Response

The BlueField DPU processes this command only when:

• The Package ID matches

• The non-volatile setting is Enabled or Locked

If the non-volatile setting is Disabled, the command fails with reason code 0x0002 (Command unavailable).

If issued by an untrusted platform BMC, the command fails with reason code 0x7FFF (Unsupported command).

Set Dynamic Host RShim Access to BlueField DPU CPU Response Format

Get Host PCIe RShim access to BlueField DPU CPU (Command=0x13, Parameter=0x19)

This package command allows the platform BMC to query the BlueField DPU for the current enablement state of Host PCIe RShim access to the embedded CPU.

Get Host PCIe RShim Access to BlueField DPU CPU Format

Get Host PCIe RShim access to BlueField DPU CPU Response

The BlueField DPU always receives and responds to this command when the Package ID matches.

Get Host PCIe RShim Access to BlueField DPU CPU Response

Get Host PCIe RShim Access to BlueField DPU CPU Response Format