DOCA DevEmu Virtio FS

DOCA Devemu Virtio FS library is part of the DOCA DevEmu Virtio subsystem. It provides low-level software APIs that provide building blocks for developing and manipulating Virtio Filesystem devices using the device emulation capability of NVIDIA® BlueField®.

DOCA version 2.7.0 introduces support for emulating Virtio FS devices over the PCI bus. The PCI transport is the common transport used for Virtio devices. Configuration, discovery, and features related to PCI (such as MSI-X and PCI device hot plug/unplug) will be managed through the DOCA DevEmu PCI APIs. Configuring common Virtio registers and handling generic Virtio logic (for example: Virtio device reset flow) is handled by the DOCA Virtio common library. This modular design enables each layer within the DOCA Device Emulation subsystem to manage its own business logic. It facilitates seamless integration with the other layers, ensuring independent functionality and operation throughout the system.

The DOCA Devemu Virtio FS library efficiently handles Virtio descriptors, carrying FUSE requests, sent by the device driver, translating them into abstract Virtio FS requests that are then routed to the user. This translation process ensures that the underlying device-specific acceleration details are abstracted away, allowing applications to interact with abstracted Virtio FS requests.

Users of this library are responsible for developing a Virtio FS controller, which manages the underlying DOCA Devemu Virtio FS device alongside an external backend file system that is outside DOCA’s scope. The controller application is designed to receive DOCA Virtio FS requests and process them according to Virtio FS and FUSE specifications, translating FUSE-based commands into the appropriate backend filesystem protocol.

Virtio FS device emulation is part of DOCA DevEmu Virtio subsystem. It is recommended to read the following guides beforehand:

• DOCA Device Emulation

• DOCA DevEmu PCI

• DOCA DevEmu Virtio

DOCA DevEmu Virtio FS is supported only on the BlueField target. The BlueField must meet the following requirements

• DOCA version 2.7.0 or greater

• BlueField-3 firmware 32.41.1000 or higher

Library must be run with root privileges.

Perform the following:

1. Configure DPU to work in DPU mode as described in NVIDIA BlueField Modes of Operation.

2. Enable the emulation capability. This can be done by running the following command on the host or DPU:

   Copy

   Copied!

               
   
               
   host/bf> sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s VIRTIO_FS_EMULATION_ENABLE=1
           

3. Configure the number of static Virtio FS physical functions and the number of MSIX for each physical function to expose. This can be done by running the following command on the DPU:

   Copy

   Copied!

               
   
               
   host/bf> sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s VIRTIO_FS_EMULATION_NUM_PF=2 VIRTIO_FS_EMULATION_NUM_MSIX=18
           

4. Perform a BlueField system reboot for the mlxconfig settings to take effect.

DOCA version 2.7.0 does not support hot plugging Virtio FS physical function devices into the host PCI subsystem or SR-IOV for Virtio FS devices.

The DOCA DevEmu Virtio FS library provides 4 main SW abstractions, the Virtio FS Type, the Virtio FS device, the Virtio FS IO context and the Virtio FS request . The Virtio FS type, that extends the Virtio type, represents common/default Virtio FS configurations of emulated Virtio FS devices, while the Virtio FS device, that extends the Virtio device, represents an instance of an emulated Virtio FS device. The Virtio FS IO context, that extends the Virtio IO context, represents a progress context that is responsible for processing Virtio descriptors, carrying FUSE requests, and their associated Virtio queues (such as hiprio, request, admin and notification queues).

Virtio FS Feature bits

According to the Virtio specification, a Virtio FS device may report support for the VIRTIO_FS_F_NOTIFICATION feature, which indicates the ability to handle FUSE notify messages sent via the notification queue.

However, in DOCA version 2.7.0, there is no support for reporting the VIRTIO_FS_F_NOTIFICATION feature to the driver.

Virtio FS configuration layout

According to the Virtio specification, the Virtio FS configuration structure layout is:

            

            
struct virtio_fs_config {
	char tag[36];
	le32 num_request_queues;
	le32 notify_buf_size;
};
        

The tag and num_request_queues fields are always available. The notify_buf_size field is only available when VIRTIO_FS_F_NOTIFICATION is set.

In DOCA version 2.7.0, there is no support for reporting the VIRTIO_FS_F_NOTIFICATION feature to the driver, therefore, notify_buf_size field is not available in this version.

Virtio FS Type

The Virtio FS type that extends the Virtio type, describes the common/default configuration of emulated Virtio FS devices , including some of the Virtio FS configuration space registers (such as num_request_queues ).
In DOCA version 2.7.0 , the Virtio FS type will be read-only and will introduce only getter APIs to r etrieve information . T he following methods can be use for this purpose:

• doca_devemu_vfs_type_get_num_request_queues - for getting the default initial value of the num_request_queues register for associated Virtio FS devices.

DOCA version 2.7.0 support the default Virtio FS type. In order to find the default Virtio FS type, one should do the following:

• doca_devemu_vfs_is_default_vfs_type_supported – check if the default DOCA Virtio FS type is supported by the device.

• if supported:

  • doca_dev_open – open supported DOCA device

  • doca_devemu_vfs_find_default_vfs_type_by_dev – get the default DOCA Virtio FS type associated with the device.

Virtio FS Device

The Virtio FS device extends the Virtio device. To start using the DOCA DevEmu Virtio FS device i t is recommended to read the guidelines of DOCA DevEmu Virtio device, DOCA DevEmu PCI device, and DOCA Core Context Configuration Phase .

This section describes how to create, configure and operate the Virtio FS device.

Configurations

The Virtio FS emulated device might be in several different visibility levels from host point of view: visible/non-visible to PCI subsystem and visible/non-visible to the Virtio subsystem.

If the device is visible to the PCI subsystem, the user will not be able to configure PCI related parameters, for example: number of MSI-X vector or subsystem_id.

If the device is visible to the Virtio subsystem, the user will not be able to configure Virtio related parameters, for example: number of queues or queue_size.

The flow for creating and configuring a Virtio FS device should be as follows:

1. doca_devemu_vfs_dev_create - Create a new DOCA DevEmu Virtio FS device instance.

2. doca_devemu_vfs_dev_set_tag - Set a unique tag for the device according to the Virtio specification.

3. doca_devemu_vfs_dev_set_num_request_queues - Set the number of requests queues for the device.

4. doca_devemu_vfs_dev_set_vfs_req_user_data_size - Set the user data size of the Virtio FS request. If set, a buffer with this size will be allocated for each DOCA DevEmu Virtio FS on behalf of the user.

5. Configure Virtio related parameters as described in DOCA Virtio configurations. Note that doca_devemu_virtio_dev_set_num_queues should be equal to the number of request queues + 1 (for the hiprio queue) since DOCA version 2.7.0 is not supporting the Virtio FS notification queue.

6. Configure PCI related parameters as described in DOCA DevEmu PCI configurations.

7. doca_ctx_start - Start the Virtio FS device context will finalize the configuration phase. Virtio FS device object follows the DOCA context state machine as described in DOCA Core Context State Machine. The Virtio FS device context will move to “running state” after the initial amount virtio IO context’s will be bounded to it and will be at “running state”, as described at DOCA DevEmu Virtio configurations.

At this point, the DOCA Devemu Virtio FS context is fully operational.

Mandatory Configurations

The mandatory configurations are as follows:

• doca_devemu_vfs_dev_set_tag - Set a unique tag for the device.

Optional Configurations

The optional configurations are as follows:

• doca_devemu_vfs_dev_set_num_request_queues – Set the number of requests queues for the device. If not set, the default value is taken from the Virtio FS Type configuration.

• doca_devemu_vfs_dev_set_vfs_req_user_data_size – Set the user data size of the Virtio FS request. If not set, user data size will be 0.

Events

DOCA DevEmu Virtio FS device exposes asynchronous events to notify about changes that happen out of the blue, according to the DOCA Core architecture.

Common events are described in DOCA DevEmu Virtio device events, DOCA DevEmu PCI Device events and in DOCA Core Event .

Virtio FS IO

The Virtio FS IO context extends the Virtio IO Context. To start using the DOCA DevEmu Virtio FS IO i t is recommended to read the guidelines of DOCA DevEmu Virtio IO and DOCA Core Context Configuration Phase.

This section describes how to create, configure and operate the Virtio FS IO context.

Configurations

The flow for creating and configuring a Virtio FS IO context should be as follows:

1. doca_devemu_vfs_io_create - Create a new DOCA DevEmu Virtio FS IO instance.

2. doca_devemu_vfs_io_event_vfs_req_notice_register - Register event handler for incoming Virtio FS requests.

3. doca_ctx_start - Start the Virtio FS IO context will finalize the configuration phase. Virtio FS IO object follows the DOCA context state machine as described in DOCA Core Context State Machine. The Virtio FS device context will move to “running state” after the initial amount Virtio FS IO context’s will be bounded to it and will be at “running state”, as described at DOCA DevEmu Virtio configurations.

Mandatory Configurations

The mandatory configurations are as follows:

• doca_devemu_vfs_io_event_vfs_req_notice_register – Registering event handler for incoming Virtio FS requests is mandatory.

Virtio FS request

The Virtio FS request object serves as an abstraction for handling requests arriving on Virtio FS queues, including high-priority, request, or notification queues. These requests are initially generated by the device driver through created Virtio queues and then routed to the user via a registered event handler, which is set up using doca_devemu_vfs_io_event_vfs_req_notice_register , on the associated Virtio IO context. This event handler, issued by the DOCA Virtio FS library, ensures that users can receive and process Virtio FS requests effectively within their application. Once the event handler called, The ownership of the Virtio FS request and the associated request user data moves to the user. The request ownership will move back to the associated Virtio IO context once it will be completed by the user by calling doca_devemu_vfs_req_complete.

The following APIs introduced for operating a Virtio FS request:

• doca_devemu_vfs_req_get_datain – Get a DOCA buffer representing the datain of the Virtio FS request. This DOCA buffer is representing the host memory for the device-readable part of the request according to the Virtio specification.

• doca_devemu_vfs_req_get_dataout – Get a DOCA buffer representing the dataout of the Virtio FS request. This DOCA buffer is representing the host memory for the device-writable part of the request according to the Virtio specification.

• doca_devemu_vfs_req_complete – Complete the Virtio FS request. The associated Virtio FS IO context will complete the request toward the device driver according to the Virtio FS specification.

As in the entire DOCA DevEmu subsystem, the emulated Virtio FS PCI functions are represented by a doca_devinfo_rep . In order to find the suitable doca_devinfo_rep that will be used as the input parameter for doca_devemu_vfs_dev_create, one should first discover the existing device representors using the bellow:

1. doca_devinfo_create_list - Get a list of all DOCA devices

2. doca_devemu_vfs_is_default_vfs_type_supported - Check whether the device can manage device associated to Virtio FS type

3. if supported:

   1. doca_dev_open - Get an instance of the DOCA device that can be used as Virtio FS emulation manager

   2. doca_devemu_vfs_find_default_vfs_type_by_dev - Get the default Virtio FS device type

   3. doca_devemu_vfs_type_as_pci_type - Cast Virtio FS type to PCI type

   4. doca_devemu_pci_type_rep_list_create - Create a list of all available representor devices for the Virtio FS type

At this point, the user can choose the preferred representor device, open using doca_dev_rep_open and proceed with the Configurations flow.

This section emphasize the initialization flow of DOCA DevEmu Virtio FS device and one or more DOCA DevEmu Virtio FS IO context’s (4 in our example). During this procedure the user will be setting up and preparing the environment before start receiving control path events (from the Virtio FS device context) and IO requests (from the Virtio FS IO context’s). During initialization, the user should configure various essential components to ensure the correct behavior.

In the bellow example, the user should perform the following configuration:

1. Choose 4 ARM cores for running its application threads.

2. Create 4 DOCA Core Progress Engine objects (pe1, pe2, pe3, pe4).

3. Find the suitable representor device according to the Discovery flow or any other method.

4. Create, configure and start a new Virtio FS device according to the Virtio FS device configurations flow. Let’s assume pe1 progress engine was be associated with the Virtio FS device and doca_devemu_virtio_dev_set_num_required_running_virtio_io_ctxs was set to 4.

5. Create, configure and start 4 new Virtio FS IO context’s according to the Virtio FS IO configurations flow. Let’s assume pe1, pe2, pe3 and pe4 progress engines were be associated with the corresponding Virtio FS IO context’s.

6. At this point, the 4 Virtio FS IO context’s will transition to “running state”, followed by the Virtio FS device context transitioning to the “running state”.

Note: During the initialization flow, it is guaranteed that no Virtio/PCI control path or IO path events will be generated until the Virtio FS device has transitioned to the “running state”.

This section emphasize the teardown flow of DOCA DevEmu Virtio FS device and one or more DOCA DevEmu Virtio FS IO context’s (4 in our example). During this procedure the user will be cleaning all the resources that were allocated in the initialization flow and all the outstanding events and requests.

In the bellow example, the user should perform the following configuration:

1. Start the teardown flow by calling doca_ctx_stop. This will cause the DOCA Virtio FS device context transition to the “stopping state”. It is guaranteed that no Virtio/PCI control path events will be generated during “stopping state”.

2. Call doca_ctx_stop for any DOCA Virtio FS IO context. This will cause the DOCA Virtio FS IO context transition to the “stopping state”. It is guaranteed that no IO path events will be generated during “stopping state”.

3. Flush all outstanding Virtio FS requests to the associated Virtio FS IO context’s by calling doca_devemu_vfs_req_complete. Upon completing all the requests associated with a Virtio FS IO context, the DOCA Virtio FS IO context transition to the “idle state”.

4. At this point it is safe to destroy the Virtio FS IO context by calling doca_devemu_vfs_io_destroy. Destroying a Virtio FS IO context not in the “idle state” will fail.

5. Upon the transition of all the 4 Virtio FS IO context’s associated with the Virtio FS device to the “idle state”, the DOCA Virtio FS device context transition to the “idle state”.

6. At this point it is safe to destroy the Virtio FS device context by calling doca_devemu_vfs_dev_destroy. Destroying a Virtio FS device context not in the “idle state” will fail.

This section describes execution on DPU Arm cores using several DOCA Core Progress Engine objects (one per core).

In the following example, the user should perform the following configuration:

1. Choose 4 Arm cores for running its application threads.

2. Create 4 DOCA Core Progress Engine objects. Application threads should periodically call doca_pe_progress to progress all the DOCA context’s associated with the progress engine.

3. Create, configure and start DOCA Virtio FS device.

4. Create, configure and start 4 DOCA Virtio FS IO context’s.

Control path

The DOCA Virtio FS device context extends the DOCA Virtio device context (which extends the DOCA PCI device context). Therefore, the DOCA Virtio FS device control path is built from all the object it extends - DOCA Context, DOCA DevEmu PCI device and DOCA DevEmu Virtio device.

The following events can be triggered by a Virtio FS device context:

1. DOCA context state change events as described in DOCA Core Context State Machine and in DOCA DevEmu PCI state machine.

2. DOCA DevEmu PCI FLR flow.

3. DOCA DevEmu Virtio reset flow.

The DOCA Virtio FS IO context extends the DOCA Virtio IO context (which extends the DOCA core context). Therefore, the DOCA Virtio FS IO context control path is built from all the object it extends - DOCA Context and DOCA DevEmu Virtio IO.

The following events can be triggered by a Virtio FS IO context:

1. DOCA context state change events as described in DOCA Core Context State Machine.

In addition to the control path events, the DOCA DevEmu Virtio FS IO context also produces IO path events as described in IO path.

IO Path

This section describes the flow for a single Virtio FS request send by the device driver until its completion.

It is assumed that the user properly configured an event handler for incoming Virtio FS request as explained in Virtio FS IO configurations.

It is also assumed that the user is familiar with the Virtio FS specification and has the ability to perform DMA operation to/from the host using DOCA DMA or any other suitable method.

The flow is illustrated in the following diagram: