RShim Drivers and Usage

Linux Kernel Upstream Release Notes v6.5

This section of the user manual describes installation and operation of NVIDIA® RShim drivers.

Warning

The Rshim drivers will be installed only on Windows Server 2016 and above or Windows Client 10 Operating Systems.

  1. Open the Device Manager when no drivers are installed to make sure a new PCIe device is available as below.

    image2020-7-5_9-20-43.png

  2. Run the installer to install all 3 drivers (MlxRshimBus.sys, MlxRshimCom.sys and MlxRshimEth.sys).

    image2020-7-5_9-21-27.png

  3. Make sure the Bus driver created 2 child devices after the installation (Com port and the Ethernet adapter).

    image2020-7-5_9-22-45.png

    image2020-7-5_9-29-8.png

    image2020-7-5_9-29-22.png

At this time, PuTTY application or any other network utility can be used to communicate with DPU via Virtual Com Port or Virtual Ethernet Adapter (ssh). The Com Port can be used using the 9600 baud-rate and default settings.

Warning

RShim drivers can be connect via PCIe (the drivers we are providing) or via USB (external connection) but not both at the same time. So when the bus driver detects that an external USB is already attached, it will not create the child virtual devices for data access. Access via PCIe is available once the USB connection is removed.

The BlueField DPU can be accessed via PuTTY or any other network utility application to communicate via virtual COM or virtual Ethernet adapter. To use COM:

  1. Open Putty.

  2. Change connection type to Serial.

  3. Run the following command in order to know what to set the "Serial line" field to:

    Copy
    Copied!
                

    C:\Users\username\Desktop> reg query HKLM\HARDWARE\DEVICEMAP\SERIALCOMM | findstr MlxRshim \MlxRshim\COM3 REG-SZ COM3

    In this case use COM3. This name can also be found via Device Manager under "Ports (Com & LPT)".

    device-manager-screenshot.png

  4. Press Open and hit Enter.

    PuTTY.png

To access via BlueField management network adapter, configure an IP address as shown in the example below and run a ping test to confirm configuration.

Registry parameters

The device does not support any type of statefull or stateless offloads. This is indicated to the Operating System accordingly when the driver loads. The MAC address is a pre-defined MAC address (CA-FE-01-CA-FE-02). The following registry keys can be used to change basic settings such as MAC address.

Registry Name

Description

Valid Values

HKLM\SYSTEM\CurrentControlSet\Control\

The size, in bytes, of the largest supported

1514 (default) - 2048

Class\{4d36e972-e325-11ce-

Jumbo Packet (an Ethernet frame that is

bfc1-08002be10318}\<nn>\*JumboPacket

greater than 1514 bytes) that the

hardware can support.

HKLM\SYSTEM\CurrentControlSet\Control\

The network address of the device. The

CA-FE-01-CA-FE-02

Class\{4d36e972-e325-11ce-

format for a MAC address is: XX-XX-XX-XX-

(default)

bfc1-08002be10318}\<nn>\*NetworkAddress

XX-XX.

HKLM\SYSTEM\CurrentControlSet\Control\

The number of receive descriptors used by

16 – 64 (Default)

Class\{4d36e972-e325-11ce-

the miniport adapter.

bfc1-08002be10318}\<nn>\ReceiveBuffers

For instructions on how to find interface index in registry <nn>, please refer to section Finding the Index Value of the Network Interface.

Support of Multiple Bluefield Cards

When the server contains several BlueField cards, several instances of Rshim bus driver will run and each one will create two virtual adapters for COM and network interface. In this case, a unique MAC and IP addresses must be set for the virtual network adapters.

MAC address can be set either using the above registry key or the RshimCmd tool's commands below, whereas the IP address should be set manually, and it should allow communication with DPU's tmfifo_net0 interface. In this case, there is a need to set unique MAC and IP addresses also for the DPUs. The IP address is changed with the help of bf.cfg file.

To check the progress of the DPU reboot, use the COM interface by connecting to the DPU before pushing the new BFB image or resetting of DPU.

To change the way the driver boots, enter the UEFI or GRUB menus after the corresponding boot process prompts.

This driver does all the read/write work to the hardware registers. RShimCmd tool sends IOCTLs to the bus driver to restart the system on chip, to push a new BlueField boot stream image or to perform other actions.

RShimCmd is a command line tool that provides user with several possibilities.

Functionality

Command

Description

Device Enumeration

RshimCmd -EnumDevices

Prints the bus numbers of all BlueField device, found on the machine. All the next numbers contain the ‘bus number’ as a parameter.

Verbosity Level

RshimCmd -SetDisplayLevel [ 0 | 1] -Busnum N

Sets the verbosity level of the Rshim bus drivers replies.

Print of the Driver's and DPU's Variables

RshimCmd -PrintVars -Busnum N

Prints a few of the driver's and DPU's variables

  • When the verbosity is 0 (default), the command prints only the driver variables:
    DISPLAY_LEVEL 1 (0:basic, 1:advanced, 2:log)

  • When the verbosity is 1, the command prints in additional the following DPU variables:
    PEER_MAC 00:1a:ca:ff:ff:07 (rw)

Change the MAC Address of the tmfifo_net0 Interface in DPU

RshimCmd -SetPeerMac xx:xx:xx:xx:xx:xx -Busnum N

Sets new MAC address to tmfifo_net0 interface in DPU.

Reading the MAC address back with ''RshimCmd -PrintVars’ can print either the new or the old value, depending on the OS in the DPU.

Note: In both cases, DPU must be reset for it to accept the new MAC address.

Restart the DPU

RshimCmd -RestartSmartNic 1 -BusNum 11

Restarts the DPU.

Note: This command should be used after changing the MAC address of tmfifo_net0 interface in the DPU.

Push the BFB Image

RshimCmd -PushImage c:\bin\MlnxBootImage.bfb -BusNum 11

A BlueField Boot file (.bfb) is a generated BlueField boot stream file that contains Linux operating system image intended to be run on the DPU. BFB files can be downloaded from the NVIDIA DOCA SDK webpage

IP address of the RShim Ethernet component (called tmfifo_net0 on the BlueField side) is 192.168.100.2/30 by default. Please set the IP address on the Windows side accordingly to be able to communicate via SSH. For example, 192.168.100.1/30.

Once the BFB file is downloaded, the tool starts to print the DPU Log file showing the process happening in the DPU. The prints are usually as follows:

---------------------------------------

Log Messages

---------------------------------------

INFO[BL2]: start

INFO[BL2]: DDR POST passed

INFO[BL2]: UEFI loaded

INFO[BL31]: start

INFO[BL31]: runtime

The printing of Log is performing maximum for 720 seconds.

One can stop it with Ctrl-C.

Tuning of BFB image before Pushing / BlueField UEFI System Boot Customizations during Installation

N/A

The BFB file can be tuned with the help of the bf.cfg configuration file. This file is fully described in BlueField documentation.

To include the bf.cfg file into the BFB installation, you must append the file to the BFB file as follow:

  1. Copy BFB file to a local folder.
    Copy <path>\DOCA_<version>_BSP_<version>_Ubuntu_20.04-5.20220707.bfb c:\bf\MlnxBootImage.bfb

  2. Append the bf.cfg file to the BFB file.

    Cd c:\bf

    Copy /b MlnxBootImage.bfb + bf.cfg MlnxBootImage_with_bf_cfg.bfb

  3. Download the BFB image.
    RshimCmd -PushImage c:\bf\MlnxBootImage_with_bf_cfg.bfb -BusNum 11

Pay attention, that the bf.cfg file is intended for Linux, so it should be created according to Linux rules. For example, the lines of this text file should end in LF and not in CR/LF as is accepted in Windows.

Also, the syntax should be as the accepting OS expects. For example, there should be no spaces in the middle of set statements: NET_RSHIM_MAC=00:1a:ca:ff:ff:05

Printing of DPU Log

RshimCmd -PrintLog -Busnum N

Prints the above DPU Log.

All driver logging is part of the Mellanox-WinOF2-Kernel trace session that comes with the network drivers installation. The default location to the trace is at %SystemRoot%\system32\LogFiles\Mlnx\Mellanox-WinOF2-System.etl.

The following are the Event logs RShim drivers generate:

RShim Bus Driver

Event ID

Severity

Message

2

Informational

RShim Bus driver loaded successfully.

3

Informational

Device successfully stopped.

4

Error

The SmartNic adapter card seems to be stuck as the boot fifo data is not being drained.

5

Error

Driver startup failed due to failure in creation of the child device.

6

Error

Smartnic is in a bad state. Please restart smartnic and reload bus drivers. Please refer to user manual on how to restart smartnic.

7

Warning

Smartnic is in LiveFish mode.

8

Warning

Failed creating child virtual devices as a backend USB device is attached and accessing RShim FIFO. Please refer to user manual for more details.

RShim Serial Driver

Event ID

Severity

Message

2

Informational

RShim Serial driver loaded successfully.

3

Informational

device successfully stopped.

RShim Ethernet Driver

Event ID

Severity

Message

2

Error

MAC Address read from registry is not supported. Please set valid unicast address.

3

Informational

device is successfully stopped.

4

Warning

value read from registry is invalid. Therefore use the default value.

5

Error

SmartNic seems stuck as transmit packets are not being drained.

6

Informational

RShim Ethernet driver loaded successfully.

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.