This library is supported at alpha level; backward compatibility is not guaranteed.
DOCA DevEmu Virtio subsystem is part of the DOCA Device Emulation subsystem. It introduces low-level software APIs that provide building blocks for developing and manipulating Virtio devices using the device emulation capability of NVIDIA® BlueField®. This subsystem incorporates a core library that handles common logic for various types of Virtio devices, such as Virtio-FS. One of its key responsibilities is managing the standard “device reset” procedure outlined in the Virtio specification. This core library serves as a foundation for implementing shared functionalities across different Virtio device types, ensuring consistency and efficiency in device operations and behaviors.
DOCA version 2.7.0 introduces support for emulating Virtio devices over the PCI bus. The PCI transport is the common transport used for Virtio devices. Configuration, discovery, and features related to PCI (such as MSI-X and PCI device hot plug/unplug) will be managed through the DOCA DevEmu PCI APIs. This modular design enables each layer within the DOCA Device Emulation subsystem to manage its own business logic. It facilitates seamless integration with the other layers, ensuring independent functionality and operation throughout the system.
This subsystem also include device specific libraries for various Virtio device types ( DOCA version 2.7.0 will include a library for a Virtio-FS device).
From the Host perspective, there is no difference whether the device is a HW/Para-virtual/DOCA-emulated device. The host should use the same Virtio device drivers to operate the device in any case.
Virtio device emulation is part of DOCA Device Emulation subsystem. It is recommended to read the following guides beforehand:
DOCA DevEmu Virtio is supported only on the BlueField target. The BlueField must meet the following requirements
DOCA version 2.7.0 or greater
BlueField-3 firmware 32.41.1000 or higher
Please refer to the DOCA Backward Compatibility Policy.
Library must be run with root privileges.
The DOCA DevEmu Virtio core library provides 3 main SW abstractions, the Virtio Type, the Virtio device, and the Virtio IO context. The Virtio type, that extends the PCI type, represents common/default Virtio configurations of emulated Virtio devices, while the Virtio device, that extends the PCI device, represents an instance of an emulated Virtio device. The Virtio IO context represents a progress context that is responsible for processing Virtio descriptors and their associated Virtio queues.
DOCA DevEmu Virtio library does not provide APIs to configure the entire BAR layout of the Virtio device. This configuration is done internally. However, it offers APIs to configure some of the registers within the common configuration structure (see Virtio Device).
Virtio Common configuration
According to the Virtio specification, the common PCI configuration structure layout is:
virtio_pci_common_cfg
struct virtio_pci_common_cfg {
/* About the whole device. */
le32 device_feature_select; /* read-write */
le32 device_feature; /* read-only for driver */
le32 driver_feature_select; /* read-write */
le32 driver_feature; /* read-write */
le16 config_msix_vector; /* read-write */
le16 num_queues; /* read-only for driver */
u8 device_status; /* read-write */
u8 config_generation; /* read-only for driver */
/* About a specific virtqueue. */
le16 queue_select; /* read-write */
le16 queue_size; /* read-write */
le16 queue_msix_vector; /* read-write */
le16 queue_enable; /* read-write */
le16 queue_notify_off; /* read-only for driver */
le64 queue_desc; /* read-write */
le64 queue_driver; /* read-write */
le64 queue_device; /* read-write */
le16 queue_notify_data; /* read-only for driver */
le16 queue_reset; /* read-write */
};
The DOCA DevEmu Virtio core library provides an ability to configure some of the above registers using the appropriate setters.
Virtio Type
The Virtio type that extends the PCI type, describes the common/default configuration of emulated Virtio devices , including the common Virtio configuration space registers (such as num_queues, queue_size, and others).
In DOCA version 2.7.0 , the Virtio type will be read-only and will introduce only getter APIs to r etrieve information . T he following methods can be use for this purpose:
doca_devemu_virtio_type_get_num_queues - for getting the default initial value of the num_queues register for associated Virtio devices
doca_devemu_virtio_type_get_queue_size - for getting the default initial value of the queue_size register for associated Virtio devices
doca_devemu_virtio_type_get_device_features_63_0 - for getting the default initial values of the device_feature bits (0-63) for associated Virtio devices
doca_devemu_virtio_type_get_config_generation - for getting the default initial value of the config_generation register for associated Virtio devices
The default Virtio type is extended by a Virtio device specific type (e.g., Virtio FS type) and can’t be created on demand.
Virtio Device
The Virtio device extends the PCI device. To start using the DOCA DevEmu Virtio device i t is recommended to read the guidelines of DOCA DevEmu PCI device and DOCA Core Context Configuration Phase .
The Virtio device is extended by a Virtio specific device (e.g., Virtio FS device) and can’t be created on demand.
This section describes how to configure and operate the Virtio device.
Configurations
The Virtio device context can be configured to match the application use case and optimize the utilization of system resources.
Mandatory Configurations
The mandatory configurations are as follows:
doca_devemu_virtio_dev_set_num_required_running_virtio_io_ctxs – for setting the the number of required running Virtio IO context’s to be bounded to the Virtio device context. The Virtio device context will not move to a “running” state (according to the DOCA Core Context state machine) before having this amount of running Virtio IO context’s bounded to it.
doca_devemu_virtio_dev_event_reset_register – for registering to Virtio device reset event. In DOCA version 2.7.0 this configuration is mandatory.
Optional Configurations
The optional configurations are as follows:
doca_devemu_virtio_dev_set_device_features_63_0 - for setting the values of the device_feature bits (0-63). If not set, the default value is taken from the Virtio Type configuration.
doca_devemu_virtio_dev_set_num_queues - for setting the value of the num_queues register. If not set, the default value is taken from the Virtio Type configuration.
doca_devemu_virtio_dev_set_queue_size - for setting the value of the queue_size register for all Virtio queues. If not set, the default value is taken from the Virtio Type configuration.
Events
DOCA DevEmu Virtio device exposes asynchronous events to notify about changes that happen out of the blue, according to the DOCA Core architecture.
Common events are described in DOCA DevEmu PCI Device events and in DOCA Core Event.
Reset Event
The reset event allows users to receive notifications whenever the device reset flow is initialized by the device driver. Upon receiving this event notification, it is guaranteed that no further requests will be routed to the user via any associated Virtio IO context until the reset flow is completed.
In order to complete the reset flow the user must:
Flush all outstanding requests back to the Virtio IO context associated with the request.
Perform one of the following:
Call doca_devemu_virtio_dev_reset_complete
Follow FLR flow:
Stop the Virtio device (and its associated Virtio IO context’s) and wait until the device (and its associated Virtio IO context’s) transitions to ‘idle state’ - doca_ctx_stop
Start the Virtio device (and its associated Virtio IO context’s) and wait until the device (and its associated Virtio IO context’s) transitions to ‘running state’ - doca_ctx_start
After completing the reset flow, the device and its associated Virtio IO context’s are fully operational again and it is allowed to route new requests via any associated Virtio IO context.
Virtio IO
The Virtio IO context extends the DOCA Core Context. To start using the DOCA DevEmu Virtio IO i t is recommended to read the guidelines of DOCA Core Context Configuration Phase .
This context is associated with a single DOCA Virtio device and it will be bounded to the Virtio device context upon start. The Virtio IO context is thread unsafe object and is progressed by a single DOCA Core Progress Engine. Usually, users will configure a single Virtio IO context per DPU core used by the application service.
The Virtio IO context is responsible to route new incoming Virtio requests towards the application and to complete handled requests back to the device driver. It can only route requests while in the ‘running state’ and when its associated Virtio device is also in the " running state”.