NVIDIA DOCA SNAP Virtio-fs Service Guide

This guide provides instructions on using the DOCA SNAP Virtio-fs service on top of the NVIDIA® BlueField®-3 DPU.

NVIDIA® BlueField® enables hardware-accelerated software-defined virtio-fs PCIe device emulation. This leverages the power of BlueField networking platforms (DPUs or SuperNICs) to provide high-performance file system access in bare-metal and virtualized environments. Using BlueField, users can offload and accelerate networked file system operations from the host/guest, freeing up resources for other tasks and improving overall system efficiency. In this solution, t he host/guest uses its own standard virtio-fs driver which is fully isolated from the networked filesystem mounted within the BlueField .

Built upon the DOCA and SPDK frameworks, virtio-fs device emulation on BlueField devices offers a comprehensive set of libraries for BlueField-based solutions and for storage solutions. This architecture consists of several key components:

• DOCA DevEmu subsystem and DOCA Virtio-fs library – These core libraries are responsible for the low-level hardware management and the translation of virtio descriptors carrying FUSE (filesystem in userspace) requests into abstract virtio-fs requests, which are then processed by the SPDK virtio-fs DOCA transport component.

• SPDK virtio-fs transport – This component is responsible for the interaction with the low-level DOCA components and translating the incoming abstract DOCA SNAP Virtio-fs requests into generic virtio-fs request which are then processed by the virtio-fs target core.

• SPDK virtio-fs target – This component implements and manages the virtio-fs device, transports, and the interface with a backend file system. Upon arrival on a new generic virtio-fs request from the transport, it processes and translates the requests according to virtio-fs and FUSE specifications, translating FUSE-based commands into the generic filesystem protocol.

• SPDK FSdev – This component provides generic filesystem abstraction and interfaces with the low-level filesystem modules implementing a specific backend filesystem protocol.

DOCA SNAP Virtio-fs as Container

The DOCA SNAP Virtio-fs container image may be downloaded from NVIDIA NGC and easily deployed on the BlueField using a YAML file. The YAML file points to the docker image that includes DOCA SNAP Virtio-fs binaries aligned with the latest spdk.nvda version.

DOCA SNAP Virtio-fs is not pre- installed on the BFB but can be downloaded manually on demand . For instructions on how to install the DOCA SNAP Virtio-fs container, refer to section "DOCA SNAP Virtio-fs Container Deployment".

This section describes how to deploy DOCA SNAP Virtio-fs as a container.

Installing Full DOCA Image on BlueField

To install the BFB on BlueField:

            

            
[host] sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb>
        

For more information, please refer to section "Installing Full DOCA Image on DPU" in the NVIDIA DOCA Installation Guide for Linux.

Firmware Installation

            

            
[dpu] sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update
        

For more information, please refer to section "Upgrading Firmware" in the NVIDIA DOCA Installation Guide for Linux.

Firmware Configuration

1. Clear the firmware config before implementing the required configuration:

   Copy

   Copied!

               
   
               
   [dpu] mst start
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 reset
           

2. Verify the firmware configuration:

   Copy

   Copied!

               
   
               
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 query
           

   Output example:

   Copy

   Copied!

               
   
               
   mlxconfig -d /dev/mst/mt41692_pciconf0 -e query | grep VIRTIO_FS 
   Configurations:                                      Default              Current              Next Boot
   *        VIRTIO_FS_EMULATION_ENABLE                  False(0)             True(1)              True(1)
            VIRTIO_FS_EMULATION_NUM_VF                  0                    0                    0
   *        VIRTIO_FS_EMULATION_NUM_PF                  0                    2                    2
            VIRTIO_FS_EMU_SUBSYSTEM_VENDOR_ID           6900                 6900                 6900
            VIRTIO_FS_EMULATION_SUBSYSTEM_ID            4186                 4186                 4186
   *        VIRTIO_FS_EMULATION_NUM_MSIX                2                    3                    3
           

   The output provides 5 columns (listed from left to right):

   • Non-default configuration marker (*)

   • Firmware configuration name

   • Default firmware value

   • Current firmware value

   • Firmware value after reboot – shows configuration update pending system reboot

3. To enable storage emulation options, BlueField must be set to work in internal CPU model:

   Copy

   Copied!

               
   
               
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_MODEL=1
           

4. To enable the firmware config with virtio-fs emulation PF:

   Copy

   Copied!

               
   
               
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s VIRTIO_FS_EMULATION_ENABLE=1 VIRTIO_FS_EMULATION_NUM_PF=1 VIRTIO_FS_EMULATION_NUM_MSIX=3
           

RDMA/RoCE Firmware Configuration

RoCE communication is blocked for the default interfaces of BlueField OS's (named ECPFs), mlx5_0 and mlx5_1 typically. If RoCE traffic is required, scalable functions (or SFs) must be added which are network functions which support RoCE transport.

To enable RDMA/RoCE:

            

            
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PER_PF_NUM_SF=1
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0.1 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
        

Hot-plug Firmware Configuration

When PCIe switch emulation is enabled, BlueField can support PCI_SWITCH_EMULATION_NUM_PORT-1 hotplug virtio-fs function. These PCIe functions are shared among all BlueField users and applications and may hold hot-plugged devices of type NVMe, virtio-blk, virtio-fs , and more (e.g., virtio-net).

To enable PCIe switch emulation and configure 31 hot-plugged ports to be used, run:

            

            
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PCI_SWITCH_EMULATION_ENABLE=1 PCI_SWITCH_EMULATION_NUM_PORT=32
        

PCI_SWITCH_EMULATION_NUM_PORT equals 1 plus the number of hot-plugged PCIe functions.

DOCA SNAP Virtio-fs Container Deployment

DOCA SNAP Virtio-fs container is available on the DOCA SNAP Virtio-fs NVIDIA™ NGC page.

To deploy DOCA SNAP Virtio-fs container on top of BlueField, the following procedure is required:

1. Setup preparation and DOCA SNAP Virtio-fs resource download for container deployment. See section "Preparation Steps" for details.

2. Adjust the doca_vfs.yaml for advanced configuration if needed according to section "Adjusting YAML Configuration".

3. Deploy the container. The image is automatically pulled from NGC. See section "Spawning DOCA SNAP Virtio-fs Container" for details.

Preparation Steps

Step 1: Allocate Hugepages

Allocate 4GiB hugepages for the DOCA SNAP Virtio-fs container according to the DPU OS's Hugepagesize value:

1. Query the Hugepagesize value:

   Copy

   Copied!

               
   
               
   [dpu] grep Hugepagesize /proc/meminfo
           

   In Ubuntu, the value should be 2048KB.

2. Append the following line to the end of the /etc/sysctl.conf file:

   Copy

   Copied!

               
   
               
   vm.nr_hugepages = 2048
           

3. Run the following:

   Copy

   Copied!

               
   
               
   [dpu] sysctl --system
           

Step 2: Create /etc/virtiofs Folder

The folder /etc/virtiofs is used by the container for automatic configuration after deployment.

Downloading YAML from NGC

The .yaml configuration file for the DOCA SNAP Virtio-fs container, doca_vfs.yaml, is uploaded to DOCA NGC.

Adjusting YAML Configuration

The .yaml file can easily be edited for advanced configuration.

• The DOCA SNAP Virtio-fs .yaml file is configured by default to support Ubuntu setups (i.e., Hugepagesize = 2048 kB) by using hugepages-2Mi.

  To support other setups, edit the hugepages section according to the relevant Hugepagesize value for the BlueField OS. For example, to support CentOS 8.x configure Hugepagesize to 512MB:

  Copy

  Copied!

              
  
              
   limits:
      hugepages-512Mi: "<number-of-hugepages>Gi"
          

• The following example edits the .yaml file to request 8 CPU cores for the DOCA SNAP Virtio-fs container:

  Copy

  Copied!

              
  
              
  resources:
      cpu: "8"
    limits:
      cpu: "8"
  env:
    - name: APP_ARGS
      value: "-m 0xff"
          

  Note

  If all BlueField-3 cores are requested, the user must verify no other containers are in conflict over CPU resources.

• To automatically configure the DOCA SNAP Virtio-fs container upon deployment:

  1. Add the spdk_rpc_init.conf file under /etc/virtiofs/. File example:

     Copy

     Copied!

                 
     
                 
     fsdev_aio_create aio0 /etc/virtiofs/test
     virtio_fs_transport_create -t DOCA
     virtio_fs_transport_start -t DOCA
     virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag docatag --fsdev aio0 --num-request-queues 1 --queue-size 32 --driver-platform x86_64
     virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid "MT2251XZ02WZVFSS0D0F3"
     virtio_fs_device_start --dev-name vfsdev0
             

  2. Edit the .yaml file accordingly (uncomment):

     Copy

     Copied!

                 
     
                 
     env:
       - name: SPDK_RPC_INIT_CONF
         value: "/etc/virtiofs/spdk_rpc_init.conf"
             

     Note

     It is user responsibility to make sure DOCA SNAP Virtio-fs configuration matches firmware configuration. That is, an emulated controller must be opened on all existing (static/hotplug) emulated PCIe functions (either through automatic or manual configuration). A PCIe function without a supporting controller is considered malfunctioned, and host behavior with it is anomalous.

Spawning DOCA SNAP Virtio-fs Container

Run the Kubernetes tool:

            

            
[dpu] systemctl restart containerd 
[dpu] systemctl restart kubelet 
[dpu] systemctl enable kubelet 
[dpu] systemctl enable containerd
        

Copy the updated doca_vfs.yaml file to the /etc/kubelet.d directory.

Kubelet automatically pulls the container image from NGC described in the YAML file and spawns a pod executing the container.

            

            
cp doca_vfs.yaml /etc/kubelet.d/
        

The DOCA SNAP Virtio-fs Service starts initialization immediately, which may take a few seconds.

To verify whether DOCA SNAP Virtio-fs is running, send spdk_rpc.py spdk_get_version to confirm whether DOCA SNAP Virtio-fs is operational or still initializing.

Debug and Log

View currently active pods, and their IDs (it might take up to 20 seconds for the pod to start):

            

            
crictl pods
        

Example output:

            

            
POD ID              CREATED               STATE         NAME
0379ac2c4f34c       About a minute ago    Ready         virtiofs
        

View currently active containers, and their IDs:

            

            
crictl ps
        

View existing containers and their ID:

            

            
crictl ps -a
        

Examine the logs of a given container (virtio-fs logs):

            

            
crictl logs <container_id>
        

Examine the kubelet logs if something does not work as expected:

            

            
journalctl -u kubelet
        

The container log file is saved automatically by Kubelet under /var/log/containers.

Stop, Start, Restart DOCA SNAP Virtio-fs Container

• To stop the container, remove the .yaml file form /etc/kubelet.d/.

• To start the container, copy the .yaml file to the same path:

  Copy

  Copied!

              
  
              
  cp doca_vfs.yaml /etc/kubelet.d
          

• To restart the container (with sig-term), use the -t (timeout) option:

  Copy

  Copied!

              
  
              
  crictl stop -t 10 <container-id>
          

  Note

  After containers in a pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s, 40s, etc.) which is capped at five minutes. Once a container has run for 10 minutes without an issue, the kubelet resets the restart back-off timer for that container.

DOCA SNAP Virtio-fs with SNAP Support

The DOCA SNAP Virtio-fs container, along with associated packages, natively supports DOCA SNAP 4.x.x, which is implemented as an SPDK subsystem module, allowing the concurrent operation of both virtio-fs and virtio-blk as a unified service. Additionally, DOCA SNAP is deployed as part of the DOCA SNAP Virtio-fs deployment.

Like other standard SPDK applications, the remote procedure call (RPC) protocol is used to control the DOCA SNAP Virtio-fs Service and supports JSON-based RPC protocol commands to control any resources and create, delete, query, or modify commands easily from the CLI.

DOCA SNAP Virtio-fs supports all standard SPDK RPC commands in addition to an extended DOCA SNAP Virtio-fs-specific command set. Standard SPDK commands are executed by the spdk_rpc.py tool.

To invoke the extended DOCA SNAP Virtio-fs-specific command set, users must add the --plugin rpc_virtio_fs_tgt flag to the SPDK's rpc.py command. The SPDK RPC plugin rpc_virtio_fs_tgt.py is implemented as an RPC plugin. This flag is not needed when working with containers.

The following is an example of an RPC when using DOCA SNAP Virtio-fs from the source:

            

            
/opt/nvidia/spdk-subsystem/src/spdk/install-$(hostname)/bin/spdk_rpc --plugin rpc_virtio_fs_tgt --help
        

DOCA SNAP Virtio-fs extended commands are detailed in the following subsections.

Using JSON-based RPC Protocol

The JSON-based RPC protocol can be used with the rpc.py script inside the DOCA SNAP Virtio-fs container and crictl tool.

• To query the active container ID:

  Copy

  Copied!

              
  
              
  crictl ps -s running -q --name virtiofs
          

• To post RPCs to the container using crictl:

  Copy

  Copied!

              
  
              
  crictl exec <container-id> spdk_rpc.py -v <RPC-method>
          

  The flag -v controls verbosity. For example:

  Copy

  Copied!

              
  
              
  crictl exec 0379ac2c4f34c spdk_rpc.py -v virtio_fs_doca_get_functions
          

  Alternatively, an alias can be used:

  Copy

  Copied!

              
  
              
  crictl exec -it $(crictl ps -s running -q --name virtiofs) spdk_rpc.py -v virtio_fs_doca_get_functions
          

• To open a bash shell to the container that can be used to post RPCs:

  Copy

  Copied!

              
  
              
  crictl exec -it <container-id> bash
          

PCIe Function Management

Emulated PCIe functions are managed through DOCA devices called emulation managers. Emulation managers have special privileges to control, manipulate, and expose the emulated PCIe devices towards the host PCIe subsystem.

To operate a virtio-fs device/function by the DOCA transport, it is necessary to locate the appropriate emulation manager for it. The emulation manager maintains a list of the emulated PCIe functions it controls. Each of those functions is assigned a globally unique serial called a vendor unique identifier or VUID (e.g., MT2251XZ02WZVFSS0D0F2), which serves as unambiguous reference for identification and tracking purposes.

virtio_fs_doca_get_managers

List existing emulation managers for virtio-fs . This method has no input parameters.

Example response:

            

            
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "name": "mlx5_0"
    }
  ]
}
        

virtio_fs_doca_get_functions

List functions for virtio-fs with their characteristics. The user may specify no parameters to list all emulated virtio-fs functions managed by any emulation manager device, or specify an emulation manager device name to list virtio-fs functions managed by that emulation manager device.

Example response:

            

            
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "manager": "mlx5_0",
      "Function List": [
        {
          "hot pluggable": "false",
          "pci_address": "0000:29:00.2",
          "vuid": "MT2251XZ02WZVFSS0D0F2",
          "function_type": "PF"
        }
      ]
    }
  ]
}
        

virtio_fs_doca_get_possible_managers

List possible emulation managers for virtio-fs. This method has no input parameters.

Example response:

            

            
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "name": "mlx5_0",
    }
  ]
}
        

virtio_fs_doca_manager_create

Create a virtio-fs emulation manager.

virtio_fs_doca_manager_destroy

Destroy a virtio-fs emulation manager.

Hot-pluggable PCIe Functions Management

Hotplug PCIe functions are configured dynamically at runtime using RPCs.

The commands outlined in the following subsections hot plug a new PCIe function to the system.

virtio_fs_doca_get_functions

List DOCA transport functions for virtio-fs with their characteristics.

Users may specify no parameters to list all emulated virtio-fs functions managed by any emulation manager device, or an emulation manager device name to list virtio-fs functions managed by a specific emulation manager device.

virtio_fs_doca_function_create

Create a DOCA virtio FS function. The return value of this method is a VUID. This is n ot needed for static functions as the VUID can be retrieved from virtio_fs_doca_get_functions.

virtio_fs_doca_function_destroy

Destroy a DOCA SNAP Virtio-fs function.

virtio_fs_doca_device_hotplug

Hot plug a DOCA SNAP Virtio-fs device. The virtio-fs device must be started.

virtio_fs_doca_device_hotunplug

Hot unplug a DOCA virtio FS device. The virtio FS device must be started.

SPDK FSdev Module Configuration

fsdev_set_opts

Set SPDK FSdev module options.

fsdev_get_opts

Get SPDK FSdev module options.

SPDK FSDEV Management

DOCA SNAP Virtio-fs uses the SPDK file system ( FSdev ) device framework as a backend for its virtio-fs controllers. Therefore, an SPDK FSdev must created and configured in advance.

Although the SPDK FSdev framework is generic and allows different types of the backend file system devices to be implemented. Currently, the only available backend device is AIO. This is the file system device that provides passthrough access to a local folder using either the Linux-native async I/O or POSIX async I/O.

fsdev_get_fsdevs

Get information about the SPDK filesystem devices (fsdevs). The user may specify no parameters to list all filesystem devices, or a filesystem device may be specified by name.

fsdev_aio_create

Create an SPDK AIO FSdev ,

fsdev_aio_delete

Delete an AIO FSdev .

Virtio-fs Emulation Management

Virtio-fs emulation is a protocol belonging to the virtio family of devices. These mount points are found in virtual environments yet by design look like physical mount points to the user within the virtual machine. Each virtio-fs mount point (e.g., virtio-fs PCIe entry) exposed to the host, whether it is PF or VF, must be backed by a virtio-fs controller.

virtio_fs_transport_create

Create a virtio-fs transport. This RPC includes all the common parameters/options for all transports. The transport becomes operational once it is started.

virtio_fs_transport_destroy

Destroy a virtio-fs transport.

virtio_fs_transport_start

Start a virtio-fs transport. This RPC finalizes the transport configuration. From this point, the transport is fully operational and can be used to create new devices.

virtio_fs_transport_stop

Stop a virtio-fs transport. This RPC makes the transport configurable again.

virtio_fs_get_transports

Display virtio-fs transports or requested transport.

virtio_fs_device_create

Create a virtio-fs device. This RPC creates a device with common parameters which are acceptable to all the transport types. To configure transport-specific parameters, users should use the virtio_fs_doca_device_modify command. The device becomes operational once it is started.

virtio_fs_device_start

Start a virtio-fs device. This RPC finalizes the device configuration. From this point, the transport is fully operational.

virtio_fs_device_stop

Stop a virtio-fs device.

virtio_fs_device_destroy

Destroy a virtio-fs device.

virtio_fs_device_modify

Modify a virtio-fs device. This RPC is used to modify/set common properties of the device which are acceptable to all the transports.

virtio_fs_get_devices

Display virtio-fs devices with their characteristics.

• The user may specify no parameters to list the virtio-fs devices associated with all transports

• The user may specify the name of a transport to list the virtio-fs devices associated with it

• The user may specify the name of a virtio-fs device to display its characteristics

Transport name and device name parameters should be mutually exclusive.

Example response:

            

            
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
           "name": "vfsdev0",
           "transport_name": "DOCA",
           "state": "idle",
          "fsdev": "aio0",
          "tag": "docatag",
          "queue_size": 256,
          "num_request_queues": 1,
          "packed_ring": true
         }
    ]
}
        

virtio_fs_doca_device_modify

Modify a virtio-fs device created from DOCA transport.

Configuration Example

Static Function – Bring up

The following is an example of creating virtio-fs DOCA transport and associating it to a virtio-fs device using a static physical function.

• In BlueField:

  1. Create an AIO FSdev backend:

     Copy

     Copied!

                 
     
                 
     rpc.py fsdev_aio_create aio0 /etc/virtiofs
             

  2. List possible emulation managers:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_possible_managers
             

  3. Create DOCA transport, emulation manager and start transport:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_create -t DOCA
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_create -m mlx5_0
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_start -t DOCA
             

  4. Get transport information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_transports
             

  5. Get managers information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_managers
             

  6. Get function information, including their VUIDs:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_functions
             

  7. Create the virtio-fs device associated with DOCA transport:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag doca_test --fsdev aio0 --num-request-queues 8 --queue-size 256 --driver-platform x86_64
             

  8. Set and modify virtio-fs parameters (VUID must be provided before calling virtio_fs_device_start RPC):

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid MT2333XZ0VJQVFSS0D0F2
             

  9. Start the virtio-fs device:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_start --dev-name vfsdev0
             

  10. Get device information:

      Copy

      Copied!

                  
      
                  
      rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
              

• In VM/host:

  • To mount a device with the tag docatag and load virtio_pci driver if not loaded:

    Copy

    Copied!

                
    
                
    mkdir "/tmp/test"
    modprobe -v virtioFS
    mount -t virtiofs docatag /tmp/test
            

Static Function – Teardown

• In BlueField:

  1. Get device information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
             

  2. Stop and destroy the virtio-fs device:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_stop --dev-name vfsdev0
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_destroy --dev-name vfsdev0
             

  3. Stop the DOCA transport and destroy emulation manager and transport:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_stop -t DOCA
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_destroy -m mlx5_0
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_destroy -t DOCA
             

• In VM/host:

  • To unmount the device:

    Copy

    Copied!

                
    
                
    umount /tmp/test
    modprobe -rv virtiofs
            

Hotplug Function

The following is an example of creating virtio-fs DOCA transport, creating a virtio-fs function, associating it to a virtio-fs device, and hot-plugging it:

• In BlueField:

  1. Create AIO FSdev backend:

     Copy

     Copied!

                 
     
                 
     rpc.py fsdev_aio_create aio0 /etc/virtiofs
             

  2. List possible emulation managers:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_possible_managers
             

  3. Create the DOCA transport and emulation manager and start the transport:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_create -t DOCA
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_create -m mlx5_0
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_start -t DOCA
             

  4. Get transport information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_transports
             

  5. Get managers information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_managers
             

     Some managers would show hotplug capability.

  6. Get functions information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_functions
             

  7. Create virtio-fs function:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_function_create --manager mlx5_0
             

     Returns VUID MT2333XZ0VJQVFSS0D0F2.

  8. Get functions information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_functions
             

     Returns the function that has been created with the appropriate VUID.

  9. Create the virtio-fs device associated with DOCA transport:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag doca_test --fsdev aio0 --num-request-queues 8 --queue-size 256 --driver-platform x86_64
             

  10. Set and modify virtio-fs parameters:

      Copy

      Copied!

                  
      
                  
      rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag doca_test --fsdev aio0 --num-request-queues 8 --queue-size 256 --driver-platform x86_64
              

      The VUID must be provided before calling the virtio_fs_device_start RPC.

  11. Start the virtio FS device:

      Copy

      Copied!

                  
      
                  
      rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_start --dev-name vfsdev0
              

  12. Get device information:

      Copy

      Copied!

                  
      
                  
      rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
              

      The output for vfsdev0 would show it is not yet plugged.

  13. Hot plug the DOCA device to the host and wait until it becomes visible by the host:

      Copy

      Copied!

                  
      
                  
      rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_hotplug --dev-name vfsdev0 --wait-for-done
              

• In VM/host:

  • To mount a device with the tag docatag and load virtio_pci driver if not loaded:

    Copy

    Copied!

                
    
                
    mkdir "/tmp/test"
    modprobe -v virtiofs
    mount -t virtiofs docatag /tmp/test
            

Hot-unplug Function

The following is an example of cleaning up and destroying the flow described under section "Hotplug Function":

• In VM/host:

  • To unmount the device:

    Copy

    Copied!

                
    
                
    umount /tmp/test
    modprobe -rv virtiofs
            

• In BlueField:

  1. Get device information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
             

  2. Hot unplug the DOCA device from the host and wait until it becomes non-visible by the host:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_hotunplug --dev-name vfsdev0 --wait-for-done
             

  3. Get device information:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
             

  4. Stop and destroy the virtio-fs DOCA device:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_stop --dev-name vfsdev0
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_destroy --dev-name vfsdev0
             

  5. Destroy the virtio-fs function:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_function_destroy --manager mlx5_0 --vuid MT2333XZ0VJQVFSS0D0F2
             

  6. Stop the DOCA transport and destroy the emulation manager and transport:

     Copy

     Copied!

                 
     
                 
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_stop -t DOCA
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_destroy -m mlx5_0
     rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_destroy -t DOCA
             

Before configuring DOCA SNAP Virtio-fs, the user must ensure that all firmware configuration requirements are met. By default, virtio-fs is disabled and must be enabled by running both common DOCA SNAP Virtio-fs configurations and additional protocol-specific configurations depending on the expected usage of the application (e.g., hot-plug, SR-IOV, UEFI boot, etc).

After configuration is finished, the host must be power cycled for the changes to take effect.

            

            
mlxconfig -d /dev/mst/mt41692_pciconf0 -e query
        

System Configuration Parameters

RDMA/RoCE Configuration

BlueField's RDMA/RoCE communication is blocked for BlueField's default OS interfaces (nameds ECPFs, typically mlx5_0 and mlx5_1). If RoCE traffic is required, additional network functions (scalable functions) must be added which support RDMA/RoCE traffic.

To enable RoCE interfaces, run the following from within the BlueField device:

            

            
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PER_PF_NUM_SF=1
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0.1 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
        

Virtio-fs Configuration

With Linux environment on host OS, additional kernel boot parameters may be required to support DOCA SNAP Virtio-fs related features:

• To use PCIe hotplug, pci=realloc must be added

• modprobe.blacklist=virtio_pci,virtiofs for the virtio-fs driver which is not built-in

• modprobe.blacklist=virtio_pci for the virtio_pci driver which is not built-in

To view boot parameter values, run:

            

            
cat /proc/cmdline
        

It is recommended to use the following command with virtio-fs :

            

            
[dpu] cat /proc/cmdline BOOT_IMAGE … pci=realloc modprobe.blacklist=virtio_pci,virtiofs
        

Intel Server Performance Optimizations

            

            
cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet pci=realloc modprobe.blacklist=virtio_pci,virtiofs
        

AMD Server Performance Optimizations

            

            
cat /proc/cmdline
cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet pci=realloc modprobe.blacklist=virtio_pci,virtiofs