DOCA DevEmu Virtio
This library is supported at alpha level; backward compatibility is not guaranteed.
DOCA DevEmu Virtio, which is part of the DOCA Device Emulation subsystem, introduces low-level software APIs that provide building blocks for developing and manipulating virtio devices using the device emulation capability of NVIDIA® BlueField®. This subsystem incorporates a core library that handles a common logic for various types of virtio devices, such as virtio-FS. One of its key responsibilities is managing the standard "device reset" procedure outlined in the virtio specification. This core library serves as a foundation for implementing shared functionalities across different virtio device types, ensuring consistency and efficiency in device operations and behaviors.
DOCA provides support for emulating virtio devices over the PCIe bus. The PCIe transport is commonly used for virtio devices. Configuration, discovery, and features related to PCIe (such as MSI-X and PCIe device hot plug/unplug) are managed through the DOCA DevEmu PCI APIs. This modular design enables each layer within the DOCA Device Emulation subsystem to manage its own business logic and facilitates seamless integration with the other layers, ensuring independent functionality and operation throughout the system.
This subsystem also includes device-specific libraries for various virtio device types (e.g., a library for a virtio-FS device).
From the host's perspective, there is no difference between para-virtual, DOCA-emulated, and actual hardware devices. The host uses the same virtio device drivers to operate the device under all circumstances.
Virtio device emulation is part of the DOCA Device Emulation subsystem. It is, therefore, recommended to read the following guides beforehand:
DOCA DevEmu Virtio is supported on the BlueField target only.
The BlueField must meet the following requirements
DOCA version 2.7.0 or greater
BlueField-3 firmware 32.41.1000 or higher
Please refer to the DOCA Backward Compatibility Policy.
Library must be run with root privileges.
The DOCA DevEmu Virtio core library provides the following software abstractions:
Virtio type – extends the PCIe type, represents common/default virtio configurations of emulated virtio devices
Virtio device – extends the PCIe device, represents an instance of an emulated virtio device
Virtio IO context – represents a progress context which is responsible for processing virtio descriptors and their associated virtio queues
DOCA DevEmu Virtio library does not provide APIs to configure the entire BAR layout of the v irtio device as this configuration is done internally. However, the library offers APIs to configure some of the registers within the common configuration structure (see Virtio Device).
Virtio Common Configuration
According to the virtio specification, the common PCIe configuration structure layout is as follows:
virtio_pci_common_cfg
struct
virtio_pci_common_cfg {
/* About the whole device. */
le32 device_feature_select; /* read-write */
le32 device_feature; /* read-only for driver */
le32 driver_feature_select; /* read-write */
le32 driver_feature; /* read-write */
le16 config_msix_vector; /* read-write */
le16 num_queues; /* read-only for driver */
u8 device_status; /* read-write */
u8 config_generation; /* read-only for driver */
/* About a specific virtqueue. */
le16 queue_select; /* read-write */
le16 queue_size; /* read-write */
le16 queue_msix_vector; /* read-write */
le16 queue_enable; /* read-write */
le16 queue_notify_off; /* read-only for driver */
le64 queue_desc; /* read-write */
le64 queue_driver; /* read-write */
le64 queue_device; /* read-write */
le16 queue_notify_data; /* read-only for driver */
le16 queue_reset; /* read-write */
};
The DOCA DevEmu Virtio core library provides the ability to configure some of the listed registers using the appropriate setters.
Virtio Type
The virtio type extends the PCIe type and describes the common/default configuration of emulated virtio devices , including the common virtio configuration space registers (such as num_queues, queue_size, and others).
Virtio type is currently read-only (i.e., only getter APIs are available to r etrieve information) . T he following methods can be used for this purpose:
doca_devemu_virtio_type_get_num_queues – for getting the default initial value of the num_queues register for the associated virtio devices
doca_devemu_virtio_type_get_queue_size – for getting the default initial value of the queue_size register for the associated virtio devices
doca_devemu_virtio_type_get_device_features_63_0 – for getting the default initial values of the device_feature bits (0-63) for the associated virtio devices
doca_devemu_virtio_type_get_config_generation – for getting the default initial value of the config_generation register for the associated virtio devices
The default virtio type is extended by a virtio device's specific type (e.g., virtio-FS type) and cannot be created on demand.
Virtio Device
The virtio device extends the PCIe device. Before using the DOCA DevEmu Virtio device, i t is recommended to read the guidelines of DOCA DevEmu PCI device and DOCA Core context configuration phase .
The virtio device is extended by a virtio-specific device (e.g., virtio FS device) and cannot be created on demand.
Virtio Device Configurations
The virtio device context can be configured to match the application use case and optimize the utilization of system resources.
Mandatory Configurations
The mandatory configurations are as follows:
doca_devemu_virtio_dev_set_num_required_running_virtio_io_ctxs – to set the number of required running virtio IO contexts to be bound to the virtio device context. The virtio device context does not move to running state (according to the DOCA Core context state machine) before having this amount of running virtio IO contexts bound to it.
doca_devemu_virtio_dev_event_reset_register – to register to the virtio device reset event. This configuration is mandatory
Optional Configurations
The optional configurations are as follows:
doca_devemu_virtio_dev_set_device_features_63_0 – to set the values of the device_feature bits (0-63). If not set, the default value is taken from the virtio type configuration.
doca_devemu_virtio_dev_set_num_queues – to set the value of the num_queues register. If not set, the default value is taken from the virtio type configuration.
doca_devemu_virtio_dev_set_queue_size – to set the value of the queue_size register for all virtio queues. If not set, the default value is taken from the virtio type configuration.
Events
DOCA DevEmu Virtio device exposes asynchronous events to notify about sudden changes, according to DOCA Core architecture.
Common events are described in DOCA DevEmu PCI Device events and in DOCA Core Event.
Reset Event
The reset event allows users to receive notifications whenever the device reset flow is initialized by the device driver. Upon receiving this event, it is guaranteed that no further requests are routed to the user via any associated virtio IO context until the reset flow is completed.
To complete the reset flow the user must:
Flush all outstanding requests back to the virtio IO context associated with the request.
Perform one of the following:
Call doca_devemu_virtio_dev_reset_complete .
Follow FLR flow:
doca_ctx_stop – stop the virtio device with its associated virtio IO contexts and wait until the device and its associated virtio IO contexts transition to idle state
doca_ctx_start – start the virtio device with its associated virtio IO contexts and wait until the device and its associated virtio IO contexts transition to running state
Now, the device and its associated virtio IO contexts should be fully operational again, the device is allowed to route new requests via any associated virtio IO context.
Virtio IO
The virtio IO context extends the DOCA Core context. Before using the DOCA DevEmu Virtio IO, i t is recommended to read the guidelines of DOCA Core context configuration phase .
This context is associated with a single DOCA virtio device and is bound to the virtio device context upon start. The virtio IO context is a thread-unsafe object and is progressed by a single DOCA Core progress engine. Usually, users configure a single virtio IO context per BlueField core used by the application service.
The virtio IO context is responsible to route new incoming virtio requests towards the application and to complete handled requests back to the device driver. It can only route requests while in running state and when its associated virtio device is also in running state .