DOCA SNAP Virtio-fs Service Guide

NVIDIA BlueField enables hardware-accelerated, software-defined virtio-fs PCIe device emulation. This leverages the power of BlueField networking platforms (DPUs or SuperNICs) to provide high-performance file system access in bare-metal and virtualized environments. Using BlueField, users can offload and accelerate networked file system operations from the host/guest, freeing up resources for other tasks and improving overall system efficiency. In this solution, t he host/guest uses its own standard virtio-fs driver which is fully isolated from the networked filesystem mounted within the BlueField .

Built upon the DOCA and SPDK frameworks, virtio-fs device emulation on BlueField devices offers a comprehensive set of libraries for BlueField-based solutions and for storage solutions. This architecture consists of several key components:

• DOCA DevEmu subsystem and DOCA Virtio-fs library – These core libraries are responsible for the low-level hardware management and the translation of virtio descriptors carrying FUSE (filesystem in userspace) requests into abstract virtio-fs requests, which are then processed by the SPDK virtio-fs DOCA transport component.

• SPDK virtio-fs transport – This component is responsible for the interaction with the low-level DOCA components and translating the incoming abstract DOCA SNAP Virtio-fs requests into generic virtio-fs request which are then processed by the virtio-fs target core.

• SPDK virtio-fs target – This component implements and manages the virtio-fs device, transports, and the interface with a backend file system. Upon arrival on a new generic virtio-fs request from the transport, it processes and translates the requests according to virtio-fs and FUSE specifications, translating FUSE-based commands into the generic filesystem protocol.

• SPDK fsdev – This component provides generic filesystem abstraction and interfaces with the low-level filesystem modules implementing a specific backend filesystem protocol.

DOCA SNAP Virtio-fs as Container

The DOCA SNAP Virtio-fs container image may be downloaded from NVIDIA NGC and easily deployed on the BlueField using a YAML file. The YAML file points to the docker image that includes DOCA SNAP Virtio-fs binaries aligned with the latest spdk.nvda version.

DOCA SNAP Virtio-fs is not pre- installed on the BFB but can be downloaded manually on demand . For instructions on how to install the DOCA SNAP Virtio-fs container, refer to DOCA SNAP Virtio-fs Container Deployment.

This section describes how to deploy DOCA SNAP Virtio-fs as a container.

Installing Full DOCA Image on BlueField

To install the BFB on BlueField:

            

            
[host] sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb>
        

For more information, refer to "Installing Full DOCA Image on DPU" in the NVIDIA DOCA Installation Guide for Linux.

Firmware Installation

            

            
[dpu] sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update
        

For more information, refer to "Upgrading Firmware" in the NVIDIA DOCA Installation Guide for Linux.

Firmware Configuration

1. Clear the firmware configuration before implementing the required configuration:

   Copy

   Copied!

               
   
               
   [dpu] mst start
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 reset
           

2. Verify the firmware configuration:

   Copy

   Copied!

               
   
               
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 query
           

   Output example:

   Copy

   Copied!

               
   
               
   mlxconfig -d /dev/mst/mt41692_pciconf0 -e query | grep VIRTIO_FS 
   Configurations:                                      Default              Current              Next Boot
   *        VIRTIO_FS_EMULATION_ENABLE                  False(0)             True(1)              True(1)
            VIRTIO_FS_EMULATION_NUM_VF                  0                    0                    0
   *        VIRTIO_FS_EMULATION_NUM_PF                  0                    2                    2
            VIRTIO_FS_EMU_SUBSYSTEM_VENDOR_ID           6900                 6900                 6900
            VIRTIO_FS_EMULATION_SUBSYSTEM_ID            4186                 4186                 4186
   *        VIRTIO_FS_EMULATION_NUM_MSIX                2                    3                    3
           

   The output provides 5 columns (listed from left to right):

   • Non-default configuration marker (*)

   • Firmware configuration name

   • Default firmware value

   • Current firmware value

   • Firmware value after reboot – shows configuration update pending system reboot

3. To enable storage emulation options, BlueField must be set to work in internal CPU model:

   Copy

   Copied!

               
   
               
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_MODEL=1
           

4. To enable the firmware config with virtio-fs emulation PF:

   Copy

   Copied!

               
   
               
   [dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s VIRTIO_FS_EMULATION_ENABLE=1 VIRTIO_FS_EMULATION_NUM_PF=1 VIRTIO_FS_EMULATION_NUM_MSIX=3
           

RDMA/RoCE Firmware Configuration

RoCE communication is blocked for the default interfaces of BlueField OS's (named ECPFs), mlx5_0 and mlx5_1 typically. If RoCE traffic is required, scalable functions (or SFs) must be added which are network functions which support RoCE transport.

To enable RDMA/RoCE:

            

            
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PER_PF_NUM_SF=1
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0.1 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
        

Hot-plug Firmware Configuration

When PCIe switch emulation is enabled, BlueField can support PCI_SWITCH_EMULATION_NUM_PORT-1 hotplug virtio-fs function. These PCIe functions are shared among all BlueField users and applications and may hold hot-plugged devices of type NVMe, virtio-blk, virtio-fs , and more (e.g., virtio-net).

To enable PCIe switch emulation and configure 31 hot-plugged ports to be used, run:

            

            
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PCI_SWITCH_EMULATION_ENABLE=1 PCI_SWITCH_EMULATION_NUM_PORT=32
        

PCI_SWITCH_EMULATION_NUM_PORT equals 1 plus the number of hot-plugged PCIe functions.

Virtio-fs DOCA Host Drive Installation (Optional)

The NVIDIA DOCA host package includes an optimized and extended version of the Virtio-fs driver; it provides better performance and additional features when compared to the upstream package.

• Optimized performance using Virtio-fs multi-queue with a better locking design.

• Support for a notification queue, allowing for a more accurate cached view of the filesystem by the host when remote changes happen.

• Support for GPU Direct Storage, allowing for zero-copy transfers between storage devices and GPUs.

To install DOCA host, install the following package:

1. Install the doca repo

            

            
[host] apt install doca-host_<version>-ubuntu2404_amd64.deb 
        

1. Update the package cache list

   Copy

   Copied!

               
   
               
   [host] apt update 
           

2. Install NVIDIA virtiofs:

            

            
[host] apt install virtiofs-dkms
        

The Virtio-fs DOCA host driver takes over your system version of the Virtio-fs driver. It does not affect your system version of FUSE. Existing FUSE-based application should run unaffected.

Configuration

Once installed, configure your DPU to run a SNAP Virtio-fs service and reboot the host. The host system will be stuck during the boot process until the SNAP Virtio-fs service becomes available.

You can confirm you are running Virtio-fs DOCA Host driver by looking in your kernel log for mentions of it:

            

            
[host] dmesg | grep 'virtio-fs' virtio-fs: Loading NVIDIA-virtiofs +mq +lockless +nvq
        

You can now mount your Virtio-fs drive as usual:

            

            
[host] mount -t virtiofs <tagname> /mnt/virtiofs/
        

Uninstallation

If you wish to remove the Virtio-fs DOCA Host driver, you can do so with the following command. It will also remove packages automatically installed as dependencies:

            

            
[host] apt purge --autoremove virtiofs-dkms
        

DOCA SNAP Virtio-fs Container Deployment

DOCA SNAP Virtio-fs container is available on the DOCA SNAP Virtio-fs NVIDIA NGC page.

To deploy DOCA SNAP Virtio-fs container on top of BlueField, the following procedure is required:

1. Setup preparation and DOCA SNAP Virtio-fs resource download for container deployment. See section "Preparation Steps" for details.

2. Adjust the doca_vfs.yaml for advanced configuration if needed according to section "Adjusting YAML Configuration".

3. Deploy the container. The image is automatically pulled from NGC. See section "Spawning DOCA SNAP Virtio-fs Container" for details.

Preparation Steps

Step 1: Allocate Hugepages

Allocate 4GiB hugepages for the DOCA SNAP Virtio-fs container according to the DPU OS's Hugepagesize value:

1. Query the Hugepagesize value:

   Copy

   Copied!

               
   
               
   [dpu] grep Hugepagesize /proc/meminfo
           

   In Ubuntu, the value should be 2048KB.

2. Use the doca-hugepages too to configure the requested hugepages:

   Copy

   Copied!

               
   
               
   [dpu] doca-hugepages config --app snap --size 2048 --num 2048
           

3. Run the following to reloads the hugepages configuration for all applications based on the current database settings:

   Copy

   Copied!

               
   
               
   [dpu] doca-hugepages reload
           

Step 2: Create /etc/virtiofs Folder

The folder /etc/virtiofs is used by the container for automatic configuration after deployment.

Downloading YAML from NGC

The .yaml configuration file for the DOCA SNAP Virtio-fs container, doca_vfs.yaml, is uploaded to DOCA NGC.

Adjusting YAML Configuration

The .yaml file can easily be edited for advanced configuration.

• The DOCA SNAP Virtio-fs .yaml file is configured by default to support Ubuntu setups (i.e., Hugepagesize = 2048 kB) by using hugepages-2Mi.

  To support other setups, edit the hugepages section according to the relevant Hugepagesize value for the BlueField OS. For example, to support CentOS 8.x configure Hugepagesize to 512MB:

  Copy

  Copied!

              
  
              
   limits:
      hugepages-512Mi: "<number-of-hugepages>Gi"
          

• The following example edits the .yaml file to request 8 CPU cores for the DOCA SNAP Virtio-fs container:

  Copy

  Copied!

              
  
              
  resources:
      cpu: "8"
    limits:
      cpu: "8"
  env:
    - name: APP_ARGS
      value: "-m 0xff"
          

  Note

  If all BlueField-3 cores are requested, the user must verify no other containers are in conflict over CPU resources.

• To automatically configure the DOCA SNAP Virtio-fs container upon deployment:

  1. Add the spdk_rpc_init.conf file under /etc/virtiofs/. File example:

     Copy

     Copied!

                 
     
                 
     fsdev_aio_create aio0 /etc/virtiofs/test
     virtio_fs_transport_create -t DOCA
     virtio_fs_transport_start -t DOCA
     virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag docatag --fsdev aio0 --num-request-queues 1 --queue-size 32 --driver-platform x86_64
     virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid "MT2251XZ02WZVFSS0D0F3"
     virtio_fs_device_start --dev-name vfsdev0
             

  2. Edit the .yaml file accordingly (uncomment):

     Copy

     Copied!

                 
     
                 
     env:
       - name: SPDK_RPC_INIT_CONF
         value: "/etc/virtiofs/spdk_rpc_init.conf"
             

     Note

     It is user responsibility to make sure DOCA SNAP Virtio-fs configuration matches firmware configuration. That is, an emulated controller must be opened on all existing (static/hotplug) emulated PCIe functions (either through automatic or manual configuration). A PCIe function without a supporting controller is considered malfunctioned, and host behavior with it is anomalous.

Spawning DOCA SNAP Virtio-fs Container

Run the Kubernetes tool:

            

            
[dpu] systemctl restart containerd 
[dpu] systemctl restart kubelet 
[dpu] systemctl enable kubelet 
[dpu] systemctl enable containerd
        

Copy the updated doca_vfs.yaml file to the /etc/kubelet.d directory.

Kubelet automatically pulls the container image from NGC described in the YAML file and spawns a pod executing the container.

            

            
cp doca_vfs.yaml /etc/kubelet.d/
        

The DOCA SNAP Virtio-fs Service starts initialization immediately, which may take a few seconds.

To verify whether DOCA SNAP Virtio-fs is running, send spdk_rpc.py spdk_get_version to confirm whether DOCA SNAP Virtio-fs is operational or still initializing.

Debug and Log

View currently active pods, and their IDs (it might take up to 20 seconds for the pod to start):

            

            
crictl pods
        

Example output:

            

            
POD ID              CREATED               STATE         NAME
0379ac2c4f34c       About a minute ago    Ready         virtiofs
        

View currently active containers, and their IDs:

            

            
crictl ps
        

View existing containers and their ID:

            

            
crictl ps -a
        

Examine the logs of a given container (virtio-fs logs):

            

            
crictl logs <container_id>
        

Examine the kubelet logs if something does not work as expected:

            

            
journalctl -u kubelet
        

The container log file is saved automatically by Kubelet under /var/log/containers.

Stop, Start, Restart DOCA SNAP Virtio-fs Container

• To stop the container, remove the .yaml file form /etc/kubelet.d/.

• To start the container, copy the .yaml file to the same path:

  Copy

  Copied!

              
  
              
  cp doca_vfs.yaml /etc/kubelet.d
          

• To restart the container (with sig-term), use the -t (timeout) option:

  Copy

  Copied!

              
  
              
  crictl stop -t 10 <container-id>
          

  Note

  After containers in a pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s, 40s, etc.) which is capped at five minutes. Once a container has run for 10 minutes without an issue, the kubelet resets the restart back-off timer for that container.

DOCA SNAP Virtio-fs with SNAP Support

The DOCA SNAP virtio-fs container, along with its associated packages, natively supports DOCA SNAP-4 , which is implemented as an SPDK subsystem module. This design enables the concurrent operation of both virtio-fs, virtio-blk and NVMe as a unified service. Additionally, DOCA SNAP is integrated as part of the DOCA SNAP virtio-fs deployment.

DOCA SNAP deployment sets snap, snap_nvme, and snap_vblk as SPDK subsystems which can be disabled as needed.

            

            
spdk_rpc.py  --plugin snap_rpc virtio_blk_controller_create --pf_id 0 --bdev Null0
        

Like other standard SPDK applications, the remote procedure call (RPC) protocol is used to control the DOCA SNAP Virtio-fs Service and supports JSON-based RPC protocol commands to control any resources and create, delete, query, or modify commands easily from the CLI.

DOCA SNAP Virtio-fs supports all standard SPDK RPC commands in addition to an extended DOCA SNAP Virtio-fs-specific command set. Standard SPDK commands are executed by the spdk_rpc.py tool.

To invoke the extended DOCA SNAP Virtio-fs-specific command set, users must add the --plugin rpc_virtio_fs_tgt flag to the SPDK's spdk_rpc.py command. The SPDK RPC plugin rpc_virtio_fs_tgt.py is implemented as an RPC plugin. This flag is not needed when working with containers.

The following is an example of an RPC when using DOCA SNAP Virtio-fs from the source:

            

            
/opt/nvidia/spdk-subsystem/src/spdk/install-$(hostname)/bin/spdk_rpc.py --plugin rpc_virtio_fs_tgt --help
        

DOCA SNAP Virtio-fs extended commands are detailed in the following subsections.

Using JSON-based RPC Protocol

The JSON-based RPC protocol can be used with the spdk_rpc.py script inside the DOCA SNAP Virtio-fs container and crictl tool.

• To query the active container ID:

  Copy

  Copied!

              
  
              
  crictl ps -s running -q --name virtiofs
          

• To post RPCs to the container using crictl:

  Copy

  Copied!

              
  
              
  crictl exec <container-id> spdk_rpc.py -v <RPC-method>
          

  The flag -v controls verbosity. For example:

  Copy

  Copied!

              
  
              
  crictl exec 0379ac2c4f34c spdk_rpc.py -v virtio_fs_doca_get_functions
          

  Alternatively, an alias can be used:

  Copy

  Copied!

              
  
              
  crictl exec -it $(crictl ps -s running -q --name virtiofs) spdk_rpc.py -v virtio_fs_doca_get_functions
          

• To open a bash shell to the container that can be used to post RPCs:

  Copy

  Copied!

              
  
              
  crictl exec -it <container-id> bash
          

PCIe Function Management

Emulated PCIe functions are managed through DOCA devices called emulation managers. Emulation managers have special privileges to control, manipulate, and expose the emulated PCIe devices towards the host PCIe subsystem.

To operate a virtio-fs device/function by the DOCA transport, it is necessary to locate the appropriate emulation manager for it. The emulation manager maintains a list of the emulated PCIe functions it controls. Each of those functions is assigned a globally unique serial called a vendor unique identifier or VUID (e.g., MT2251XZ02WZVFSS0D0F2), which serves as unambiguous reference for identification and tracking purposes.

virtio_fs_doca_get_managers

List existing emulation managers for virtio-fs . This method has no input parameters.

Example response:

            

            
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "name": "mlx5_0"
    }
  ]
}
        

virtio_fs_doca_get_functions

List functions for virtio-fs with their characteristics. The user may specify no parameters to list all emulated virtio-fs functions managed by any emulation manager device, or specify an emulation manager device name to list virtio-fs functions managed by that emulation manager device.

Example response:

            

            
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "manager": "mlx5_0",
      "Function List": [
        {
          "hot pluggable": "false",
          "pci_address": "0000:29:00.2",
          "vuid": "MT2251XZ02WZVFSS0D0F2",
          "function_type": "PF"
        }
      ]
    }
  ]
}
        

virtio_fs_doca_get_possible_managers

List possible emulation managers for virtio-fs. This method has no input parameters.

Example response:

            

            
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "name": "mlx5_0",
    }
  ]
}
        

virtio_fs_doca_manager_create

Create a virtio-fs emulation manager.

virtio_fs_doca_manager_destroy

Destroy a virtio-fs emulation manager.

Hot-pluggable PCIe Functions Management

Hotplug PCIe functions are configured dynamically at runtime using RPCs.

The commands outlined in the following subsections hot plug a new PCIe function to the system.

virtio_fs_doca_get_functions

List DOCA transport functions for virtio-fs with their characteristics.

Users may specify no parameters to list all emulated virtio-fs functions managed by any emulation manager device, or an emulation manager device name to list virtio-fs functions managed by a specific emulation manager device.

virtio_fs_doca_function_create

Create a DOCA virtio FS function. The return value of this method is a VUID. This is n ot needed for static functions as the VUID can be retrieved from virtio_fs_doca_get_functions.

virtio_fs_doca_function_destroy

Destroy a DOCA SNAP Virtio-fs function.

virtio_fs_doca_device_hotplug

Hot plug a DOCA SNAP Virtio-fs device. The virtio-fs device must be started.

virtio_fs_doca_device_hotunplug

Hot unplug a DOCA virtio FS device. The virtio FS device must be started.

SPDK Fsdev Management

DOCA SNAP Virtio-fs uses the SPDK file system ( fsdev ) device framework as a backend for its virtio-fs controllers. Therefore, an SPDK fsdev must created and configured in advance.

Although the SPDK fsdev framework is generic and allows different types of the backend file system devices to be implemented. Currently, the only available backend devices are AIO and NFS.

fsdev_get_fsdevs

Get information about the SPDK filesystem devices (fsdevs). The user may specify no parameters to list all filesystem devices, or a filesystem device may be specified by name.

fsdev_get_iostat

Get I/O statistics of filesystem devices (fsdevs).

If name is not specified, the RPC shows stats for all filesystem devices. If per_channel is not specified, the RPC shows the cumulative per-fsdev stat.

fsdev_reset_iostat

Reset I/O statistics of filesystem devices (fsdevs).

If name is not specified, the RPC resets the stats for all filesystem devices.

fsdev_aio_create

Create an SPDK AIO fsdev. This is the file system device that provides passthrough access to a local folder using either the Linux-native async I/O or POSIX async I/O.

fsdev_aio_delete

Delete an AIO fsdev .

fsdev_nfs_create

Create an SPDK NFS fs dev:

fsdev_nfs_delete

Delete an NFS fsdev .

Virtio-fs Emulation Management

Virtio-fs emulation is a protocol belonging to the virtio family of devices. These mount points are found in virtual environments yet by design look like physical mount points to the user within the virtual machine. Each virtio-fs mount point (e.g., virtio-fs PCIe entry) exposed to the host, whether it is PF or VF, must be backed by a virtio-fs controller.

virtio_fs_transport_create

Create a virtio-fs transport. This RPC includes all the common parameters/options for all transports. The transport becomes operational once it is started.

virtio_fs_transport_destroy

Destroy a virtio-fs transport.

virtio_fs_transport_start

Start a virtio-fs transport. This RPC finalizes the transport configuration. From this point, the transport is fully operational and can be used to create new devices.

virtio_fs_transport_stop

Stop a virtio-fs transport. This RPC makes the transport configurable again.

virtio_fs_get_transports

Display virtio-fs transports or requested transport.

virtio_fs_device_create

Create a virtio-fs device. This RPC creates a device with common parameters which are acceptable to all the transport types. To configure transport-specific parameters, users should use the virtio_fs_doca_device_modify command. The device becomes operational once it is started.

virtio_fs_device_start

Start a virtio-fs device. This RPC finalizes the device configuration. From this point, the transport is fully operational.

virtio_fs_device_stop

Stop a virtio-fs device.

virtio_fs_device_destroy

Destroy a virtio-fs device.

virtio_fs_device_modify

Modify a virtio-fs device. This RPC is used to modify/set common properties of the device which are acceptable to all the transports.

virtio_fs_get_devices

Display virtio-fs devices with their characteristics.

• The user may specify no parameters to list the virtio-fs devices associated with all transports

• The user may specify the name of a transport to list the virtio-fs devices associated with it

• The user may specify the name of a virtio-fs device to display its characteristics

Transport name and device name parameters should be mutually exclusive.

Example response:

            

            
{
 "jsonrpc": "2.0",
  "id": 1,
  "result": {
	"devices" : [
	  {
		"name": "vfsdev0",
		"fsdev_name": "Aio0",
		"state": "running",
		"transport_name": "DOCA"
		"transport_specific" : {
		  "queues": [
			   {
			   "queue_id": 0,
			   "queue_enabled": true,
			   "queue_depth": 128,
			   "queue_inflights: 8,
			   "hw_available_index": 9760,
			   "driver_available_index": 9762,
			   "hw_used_index": 9755, 
			   "driver_used_index": 9750
			   },
			]
		}
	  }
	]
  }
}
        

virtio_fs_doca_device_modify

Modify a virtio-fs device created from DOCA transport.

virtio_fs_device_get_stat

Displays stats for a specified virtio-fs device, or for all devices if none is specified.

Example response:

            

            
{
jsonrpc": "2.0",
  "id": 1,
  "result": {
	  "devices": [
		{
		  "name": "vfsdev0",
		  "fsdev": "aio0",
		  "state": "running",
		  "transport_name": "DOCA",
		  "transport_specific": {
			"queues": [
			  {
				"queue_id": 0,
				"queue_enabled": 1,
				"queue_depth": 256,
				"queue_inflights": 0,
				"hw_available_index": 8,
				"driver_available_index": 8,
				"hw_used_index": 8,
				"driver_used_index": 8
			  },
			  {
				"queue_id": 1,
				"queue_enabled": 1,
				"queue_depth": 256,
				"queue_inflights": 0,
				"hw_available_index": 104,
				"driver_available_index": 104,
				"hw_used_index": 104,
				"driver_used_index": 104
			  }
			]
		  }
		}
	  ]
	}
}
        

Configuration Example

Static Function – Bring up

The following is an example of creating virtio-fs DOCA transport and associating it to a virtio-fs device using a static physical function.

• In BlueField:

  1. Create an AIO fsdev backend:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py fsdev_aio_create aio0 /etc/virtiofs
             

  2. List possible emulation managers:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_possible_managers
             

  3. Create DOCA transport, emulation manager and start transport:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_create -t DOCA
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_create -m mlx5_0
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_start -t DOCA
             

  4. Get transport information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_transports
             

  5. Get managers information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_managers
             

  6. Get function information, including their VUIDs:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_functions
             

  7. Create the virtio-fs device associated with DOCA transport:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag docatag --fsdev aio0 --num-request-queues 8 --queue-size 256 --driver-platform x86_64
             

  8. Set and modify virtio-fs parameters (VUID must be provided before calling virtio_fs_device_start RPC):

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid MT2333XZ0VJQVFSS0D0F2
             

  9. Start the virtio-fs device:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_start --dev-name vfsdev0
             

  10. Get device information:

      Copy

      Copied!

                  
      
                  
      spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
              

• In VM/host:

  • To mount a device with the tag docatag and load virtio_pci driver if not loaded:

    Copy

    Copied!

                
    
                
    mkdir "/tmp/test"
    modprobe -v virtiofs
    mount -t virtiofs docatag /tmp/test
            

Static Function – Teardown

• In BlueField:

  1. Get device information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
             

  2. Stop and destroy the virtio-fs device:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_stop --dev-name vfsdev0
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_destroy --dev-name vfsdev0
             

  3. Stop the DOCA transport and destroy emulation manager and transport:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_stop -t DOCA
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_destroy -m mlx5_0
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_destroy -t DOCA
             

• In VM/host:

  • To unmount the device:

    Copy

    Copied!

                
    
                
    umount /tmp/test
    modprobe -rv virtiofs
            

Hotplug Function

The following is an example of creating virtio-fs DOCA transport, creating a virtio-fs function, associating it to a virtio-fs device, and hot-plugging it:

• In BlueField:

  1. Create AIO fsdev backend:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py fsdev_aio_create aio0 /etc/virtiofs
             

  2. List possible emulation managers:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_possible_managers
             

  3. Create the DOCA transport and emulation manager and start the transport:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_create -t DOCA
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_create -m mlx5_0
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_start -t DOCA
             

  4. Get transport information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_transports
             

  5. Get managers information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_managers
             

     Some managers would show hotplug capability.

  6. Get functions information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_functions
             

  7. Create virtio-fs function:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_function_create --manager mlx5_0
             

     Returns VUID MT2333XZ0VJQVFSS0D0F2.

  8. Get functions information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_get_functions
             

     Returns the function that has been created with the appropriate VUID.

  9. Create the virtio-fs device associated with DOCA transport:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag docatag --fsdev aio0 --num-request-queues 8 --queue-size 256 --driver-platform x86_64
             

  10. Set and modify virtio-fs parameters:

      Copy

      Copied!

                  
      
                  
      spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid MT2333XZ0VJQVFSS0D0F2
              

      The VUID must be provided before calling the virtio_fs_device_start RPC.

  11. Start the virtio FS device:

      Copy

      Copied!

                  
      
                  
      spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_start --dev-name vfsdev0
              

  12. Get device information:

      Copy

      Copied!

                  
      
                  
      spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
              

      The output for vfsdev0 would show it is not yet plugged.

  13. Hot plug the DOCA device to the host and wait until it becomes visible by the host:

      Copy

      Copied!

                  
      
                  
      spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_hotplug --dev-name vfsdev0 --wait-for-done
              

• In VM/host:

  • To mount a device with the tag docatag and load virtio_pci driver if not loaded:

    Copy

    Copied!

                
    
                
    mkdir "/tmp/test"
    modprobe -v virtiofs
    mount -t virtiofs docatag /tmp/test
            

Hot-unplug Function

The following is an example of cleaning up and destroying the flow described under section "Hotplug Function":

• In VM/host:

  • To unmount the device:

    Copy

    Copied!

                
    
                
    umount /tmp/test
    modprobe -rv virtiofs
            

• In BlueField:

  1. Get device information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
             

  2. Hot unplug the DOCA device from the host and wait until it becomes non-visible by the host:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_hotunplug --dev-name vfsdev0 --wait-for-done
             

  3. Get device information:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_get_devices
             

  4. Stop and destroy the virtio-fs DOCA device:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_stop --dev-name vfsdev0
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_device_destroy --dev-name vfsdev0
             

  5. Destroy the virtio-fs function:

     Copy

     Copied!

                 
     
                 
     spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_function_destroy --manager mlx5_0 --vuid MT2333XZ0VJQVFSS0D0F2
             

Stop the DOCA transport and destroy the emulation manager and transport:

            

            
spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_stop -t DOCA
spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_doca_manager_destroy -m mlx5_0
spdk_rpc.py --plugin rpc_virtio_fs_tgt -v virtio_fs_transport_destroy -t DOCA
        

RPC log history (enabled by default) records all RPC requests and their responses in /var/log/virtiofs/rpc-log. This file is also available outside the container at /var/log/snap-log/rpc-log on the DPU.

The file is continuously updated in append mode. Monitor its size, and delete it before launching the SNAP pod if it becomes too large.

To recover and reuse configurations for virtio-fs, you can save the desired configuration and load it later using a saved configuration file. This process allows for quick recovery and ensures consistent device/application setup.

To save the configuration of virtio-fs devices:

1. Use RPCs to configure your virtio-fs device/application as needed.

2. Save the configuration after each configuration cycle using the save_config RPC. This creates a JSON configuration file for future use:

   Copy

   Copied!

               
   
               
   spdk_rpc.py save_config -o <json_config>.json
           

To initiate a configuration cycle, create a new configuration file using the same format as spdk_rpc_init.conf files:

            

            
cat <new_config>.conf | spdk_rpc.py
        

            

            
--plugin rpc_virtio_fs_tgt -v fsdev_aio_create aio0 /tmp/fsdev-test
--plugin rpc_virtio_fs_tgt -v virtio_fs_transport_create -t DOCA
--plugin rpc_virtio_fs_tgt -v virtio_fs_transport_start -t DOCA
--plugin rpc_virtio_fs_tgt -v virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag docatest --fsdev aio0 --num-request-queues 1 --queue-size 256 --driver-platform x86_64
--plugin rpc_virtio_fs_tgt -v virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid MT2306XZ00HKVFSS0D0F4
--plugin rpc_virtio_fs_tgt -v virtio_fs_device_start --dev-name vfsdev0
save_config -o <json_config>.json
        

To load a saved configuration for virtio-fs, use the following methods based on your environment:

• When using virtio-fs from a container, set the following in the YAML configuration file:

  Copy

  Copied!

              
  
              
  SPDK_RPC_INIT_CONF_JSON: <json_config>.json
  APP_ARGS: --json-ignore-init-errors
          

• When using virtio-fs from source, re-run the DOCA SNAP Virtio-fs application, recovering the configuration from the saved file:

  Copy

  Copied!

              
  
              
  /opt/nvidia/spdk-subsystem/src/virtio-fs-target/app/virtio_fs_tgt -c <json_config>.json --json-ignore-init-errors
          

Virtio-fs is defined as an SPDK subsystem and can be disabled or enabled as required.

The Virtio-fs subsystem is enabled by default. To disable it:

1. Start the Virtio-fs application in a waiting state for RPC commands:

   Copy

   Copied!

               
   
               
   /opt/nvidia/spdk-subsystem/src/virtio-fs-target/app/virtio_fs_tgt --wait-for-rpc
           

2. Use the following RPC command to disable the virtio_fs subsystem:

   Copy

   Copied!

               
   
               
   spdk_rpc.py framework_disable_subsystem  virtio_fs
           

3. Restart the SPDK framework initialization:

   Copy

   Copied!

               
   
               
   spdk_rpc.py framework_start_init
           

To view the status of all subsystems:

            

            
spdk_rpc.py framework_get_subsystems
        

This subsystem configuration can be saved in SPDK configuration for future use with the command spdk_rpc.py save_config. This ensures the configuration persists across application restarts.

NVIDIA GPUDirect Storage (GDS) is a technology that enables direct data transfer between storage devices and NVIDIA GPUs. By bypassing the CPU and system memory, GDS significantly reduces data transfer latency and improves bandwidth efficiency for I/O-intensive applications. GDS is particularly beneficial for workloads such as deep learning, high-performance computing (HPC), and large-scale data analytics.

See Getting Started with NVIDIA GPUDirect Storage for more information.

Prerequisites

Make sure the following are available:

• NVIDIA GPU with GPUDirect storage support (via P2P DMA)

• Installed NVIDIA GPU drivers. Refer to the NVIDIA GPUDirect Storage Troubleshooting Guide for additional details.

• Properly configured virtio-fs target

• Linux kernel with CONFIG_PCI_P2PDMA enabled

• NVIDIA-optimized virtio-fs drivers

Enabling GDS Support in Virtio-fs

1. Configure the system for GDS support:

   1. Enable PCI Peer-to-Peer DMA in libcufile:

      Copy

      Copied!

                  
      
                  
      // Edit /usr/local/cuda/gds/cufile.json:
      {
        "use_pci_p2pdma": true
      }
              

   2. Configure NVIDIA Driver Parameters:

      Create a modprobe configuration file to enable the required parameters for the NVIDIA driver:

      Copy

      Copied!

                  
      
                  
      echo 'options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1;FORCE_P2P=1"' | sudo tee /etc/modprobe.d/nvidia-gds.conf
              

   3. Configure Kernel Boot Parameters

      Edit the GRUB_CMDLINE_LINUX_DEFAULT variable in /etc/default/grub to add:

      • nokaslr to disable Kernel Address Space Layout Randomization.

      • Depending on your CPU:

        • For AMD CPUs: amd_iommu=off

        • For Intel CPUs: intel_iommu=off

      Copy

      Copied!

                  
      
                  
      #For example:
      GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 nokaslr intel_iommu=off"
      #After saving the file, update GRUB:
      sudo update-grub
              

      Then, reboot your system for the changes to take effect.

2. Verify System Compatibility:

   1. After rebooting, run the following command to check for GDS support:

      Copy

      Copied!

                  
      
                  
      /usr/local/cuda/gds/tools/gdscheck -p
              

      You should see an output similar to:

      Copy

      Copied!

                  
      
                  
      NVMe P2PDMA : Supported
              

3. Mount the VirtioFS Drive with GDS:

   1. To enable GDS workloads on your VirtioFS drive, mount it with the gds option:

      Edit the GDS configuration file located at /usr/local/cuda/gds/cufile.json and set:

            

            
sudo mount -t virtiofs -o gds <tagname> /mnt/vfs/
        

Replace <tagname> with the appropriate identifier for your setup. Once mounted, test GDS performance using the gdsio tool

            

            
CUFILE_EXPERIMENTAL_FS=1 /usr/local/cuda/gds/tools/gdsio -T 1 -D /mnt/vfs/ -d0 -I1 -s10m -i1m -w1 -x0
        

This command will output performance metrics and generate a cufile.log in the current working directory. Check the log for any errors or issues.

Before configuring DOCA SNAP Virtio-fs, the user must ensure that all firmware configuration requirements are met. By default, virtio-fs is disabled and must be enabled by running both common DOCA SNAP Virtio-fs configurations and additional protocol-specific configurations depending on the expected usage of the application (e.g., hot-plug, SR-IOV, UEFI boot, etc).

After configuration is finished, the host must be power cycled for the changes to take effect.

            

            
mlxconfig -d /dev/mst/mt41692_pciconf0 -e query
        

System Configuration Parameters

RDMA/RoCE Configuration

BlueField's RDMA/RoCE communication is blocked for BlueField's default OS interfaces (nameds ECPFs, typically mlx5_0 and mlx5_1). If RoCE traffic is required, additional network functions (scalable functions) must be added which support RDMA/RoCE traffic.

To enable RoCE interfaces, run the following from within the BlueField device:

            

            
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PER_PF_NUM_SF=1
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0.1 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
        

Virtio-fs Configuration

Before configuring Virtio-fs emulation, ensure the following prerequisites are met:

• A reliable communication channel (e.g., RShim, serial) is available for accessing the Arm subsystem even if the host fails to boot.

  Info

  Refer to NVIDIA BlueField DPU BSP documentation for more details.

• The controller must be created on the static virtio-fs PF using the initial configuration file.

  Info

  For more information, please refer to section "Virtio-fs Emulation Management".

With Linux environment on host OS, additional kernel boot parameters may be required to support DOCA SNAP Virtio-fs related features:

• To use PCIe hotplug, pci=realloc must be added

• modprobe.blacklist=virtio_pci,virtiofs for the virtio-fs driver which is not built-in

• modprobe.blacklist=virtio_pci for the virtio_pci driver which is not built-in

To view boot parameter values, run:

            

            
cat /proc/cmdline
        

It is recommended to use the following command with virtio-fs :

            

            
[dpu] cat /proc/cmdline BOOT_IMAGE … pci=realloc modprobe.blacklist=virtio_pci,virtiofs
        

Intel Server Performance Optimizations

            

            
cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet pci=realloc modprobe.blacklist=virtio_pci,virtiofs
        

AMD Server Performance Optimizations

            

            
cat /proc/cmdline
cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet pci=realloc modprobe.blacklist=virtio_pci,virtiofs
        

This section explains how to extend the Virtio-fs service with custom fsdev modules. Two methods are provided:

• Building a new production container (recommended for production)

• Manual build and run inside a development container (ideal for development and testing)

The Dockerfiles and commands provided in this chapter are intended as examples and are designed to work with the external fsdev module example included with SPDK, located at spdk/test/external_code/fsdev_passthru. These examples demonstrate the integration process and should be adapted to suit real-world use cases, including adjustments for custom fsdev modules, deployment environments, and specific application requirements.

Building a New Production Container

This option allows to create a new container that includes both the original service and the custom fsdev module. The resulting container will run Virtio-fs service with custom functionality built-in.

Environment Preparation

The working directory should contain the spdk sources and the Dockerfile.

            

            
root@bf:/# mkdir -p /opt/build
root@bf:/# cd /opt/build
root@bf:/opt/build# git clone https://github.com/Mellanox/spdk --branch v25.01.1.nvda
root@bf:/opt/build# ll
total 16
drwxr-xr-x  3 root root 4096 Dec  2 14:16 ./
drwxr-xr-x  6 root root 4096 Dec  2 13:30 ../
-rw-r--r--  1 root root  723 Dec  2 14:15 Dockerfile
drwxr-xr-x 37 root root 4096 Dec  2 14:01 spdk/
        

Dockerfile Example

            

            
FROM nvcr.io/nvstaging/doca/doca_vfs:1.2.0-2-doca2.10.0 as builder
 
RUN apt-get update && apt-get install -y autoconf libtool python3-pyelftools libaio-dev libncurses-dev libfuse3-dev patchelf libcmocka-dev make
 
COPY spdk/test/external_code /external_code
 
WORKDIR /external_code
ENV SPDK_HEADER_DIR=/opt/nvidia/spdk-subsystem/virtiofs/include
ENV SPDK_LIB_DIR=/opt/nvidia/spdk-subsystem/virtiofs/lib
ENV DPDK_LIB_DIR=/opt/nvidia/spdk-subsystem/virtiofs/include
RUN make fsdev_passthru_shared
 
FROM nvcr.io/nvstaging/doca/doca_vfs:1.2.0-2-doca2.10.0
 
COPY --from=builder /external_code/fsdev_passthru/libfsdev_passthru_external.so /opt/nvidia/spdk-subsystem/virtiofs/lib/
ENV VIRTIOFS_LD_PRELOAD=/opt/nvidia/spdk-subsystem/virtiofs/lib/libfsdev_passthru_external.so
 
COPY --from=builder /external_code/fsdev_passthru/fsdev_passthru.py /usr/lib/python3/dist-packages/spdk/rpc/
ENV SPDK_RPC_PLUGIN="spdk.rpc.rpc_virtio_fs_tgt:spdk.rpc.fsdev_passthru"
        

Building and Running the Container

1. Build the new container:

   Copy

   Copied!

               
   
               
   docker build -t doca_vfs_custom_fsdev:latest -f Dockerfile .
           

2. Run the new container with Kubelet service:

   Copy

   Copied!

               
   
               
   docker save doca_vfs_custom_fsdev:latest > doca_vfs_custom_fsdev.tar
   ctr -n=k8s.io images import doca_vfs_custom_fsdev.tar
    
   # Copy doca_vfs.yaml into /etc/kubelet.d/
   # Open doca_vfs.yaml and change "image" section:
       image: doca_vfs_custom_fsdev:latest
           

3. To configure running container with custom fsdev module:

   Copy

   Copied!

               
   
               
   # DPU
   mkdir /etc/virtiofs/AIO0
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py fsdev_aio_create AIO0 /etc/virtiofs/AIO0
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py fsdev_passthru_ext_create --name CUSTOM_FSDEV_PT0 --base-fsdev-name AIO0
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py virtio_fs_transport_create -t DOCA
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py virtio_fs_doca_manager_create -m mlx5_0
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py virtio_fs_transport_start -t DOCA
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag CUSTOM_FSDEV_PT0 --fsdev CUSTOM_FSDEV_PT0 --num-request-queues 62 --queue-size 64 --driver-platform x86_64
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid MT2334XZ0LEDVFSS0D0F3
   crictl exec $(crictl ps -s running -q --name virtiofs) spdk_rpc.py virtio_fs_device_start --dev-name vfsdev0
    
   # x86
   modprobe -v virtio_pci
   modprobe -v virtiofs
   test -d /tmp/CUSTOM_FSDEV_PT0 || mkdir -p /tmp/CUSTOM_FSDEV_PT0
   mount -t virtiofs CUSTOM_FSDEV_PT0 /tmp/CUSTOM_FSDEV_PT0
           

Manual Build and Run in a Development Container

This approach is suitable for development workflows where frequent updates, builds, and tests are required. It sets up a development container with the necessary environment to manually build and run the Virtio-fs service with a custom fsdev module.

Environment Preparation

The working directory should contain the spdk sources and the Dockerfile.

            

            
root@bf:/# mkdir -p /opt/build
root@bf:/# cd /opt/build
root@bf:/opt/build# git clone https://github.com/Mellanox/spdk --branch v24.01.4.nvda
root@bf:/opt/build# ll
total 16
drwxr-xr-x  3 root root 4096 Dec  2 14:16 ./
drwxr-xr-x  6 root root 4096 Dec  2 13:30 ../
-rw-r--r--  1 root root  723 Dec  2 14:15 Dockerfile
drwxr-xr-x 37 root root 4096 Dec  2 14:01 spdk/
        

Dockerfile Example

            

            
FROM nvcr.io/nvstaging/doca/doca_vfs:1.2.0-2-doca2.10.0 as builder
 
RUN apt-get update && apt-get install -y autoconf libtool python3-pyelftools libaio-dev libncurses-dev libfuse3-dev patchelf libcmocka-dev make
 
ENTRYPOINT /bin/bash
        

Building and Running the Development Container

1. Build the container using the provided Dockerfile:

   Copy

   Copied!

               
   
               
   docker build -t doca_vfs_custom_fsdev_dev:latest -f Dockerfile .
           

2. Start the container and enter the bash shell:

   Copy

   Copied!

               
   
               
   docker run -ti --privileged --net=host \
       --volume /dev/hugepages:/dev/hugepages \
       --volume /dev/shm:/dev/shm \
       --volume /dev/infiniband:/dev/infiniband \
       --volume /etc/virtiofs:/etc/virtiofs \
       --volume ${PWD}/spdk/test/external_code:/external_code \
       doca_vfs_custom_fsdev_dev:latest
           

3. Once inside the container, one can manually build and test custom module:

   Copy

   Copied!

               
   
               
   cd /external_code/
   export SPDK_HEADER_DIR=/opt/nvidia/spdk-subsystem/virtiofs/include
   export SPDK_LIB_DIR=/opt/nvidia/spdk-subsystem/virtiofs/lib
   export DPDK_LIB_DIR=/opt/nvidia/spdk-subsystem/virtiofs/include
   make fsdev_passthru_shared
           

4. Start the virtio-fs service with the custom module for testing:

   Copy

   Copied!

               
   
               
   cp /external_code/fsdev_passthru/fsdev_passthru.py /usr/bin/
   /opt/nvidia/spdk-subsystem/doca_snap/bin/set_environment_variables.sh
   LD_PRELOAD=/external_code/fsdev_passthru/libfsdev_passthru_external.so \
       /opt/nvidia/spdk-subsystem/virtiofs/bin/virtio_fs_tgt -m 0xff &
           

5. To configure the service with custom fsdev module:

   Copy

   Copied!

               
   
               
   # DPU
   mkdir /etc/virtiofs/AIO0
   spdk_rpc.py fsdev_aio_create AIO0 /etc/virtiofs/AIO0
   spdk_rpc.py --plugin fsdev_passthru fsdev_passthru_ext_create --name CUSTOM_FSDEV_PT0 --base-fsdev-name AIO0
   spdk_rpc.py virtio_fs_transport_create -t DOCA
   spdk_rpc.py virtio_fs_doca_manager_create -m mlx5_0
   spdk_rpc.py virtio_fs_transport_start -t DOCA
   spdk_rpc.py virtio_fs_device_create --transport-name DOCA --dev-name vfsdev0 --tag CUSTOM_FSDEV_PT0 --fsdev CUSTOM_FSDEV_PT0 --num-request-queues 62 --queue-size 64 --driver-platform x86_64
   spdk_rpc.py virtio_fs_doca_device_modify --dev-name vfsdev0 --manager mlx5_0 --vuid MT2334XZ0LEDVFSS0D0F2
   spdk_rpc.py virtio_fs_device_start --dev-name vfsdev0
    
   # x86
   modprobe -v virtio_pci
   modprobe -v virtiofs
   test -d /tmp/CUSTOM_FSDEV_PT0 || mkdir -p /tmp/CUSTOM_FSDEV_PT0
   mount -t virtiofs CUSTOM_FSDEV_PT0 /tmp/CUSTOM_FSDEV_PT0