DOCA Documentation v2.8.0

DOCA DevEmu Virtio

Note

This library is supported at alpha level; backward compatibility is not guaranteed.

DOCA DevEmu Virtio, which is part of the DOCA Device Emulation subsystem, introduces low-level software APIs that provide building blocks for developing and manipulating virtio devices using the device emulation capability of NVIDIA® BlueField®. This subsystem incorporates a core library that handles a common logic for various types of virtio devices, such as virtio-FS. One of its key responsibilities is managing the standard "device reset" procedure outlined in the virtio specification. This core library serves as a foundation for implementing shared functionalities across different virtio device types, ensuring consistency and efficiency in device operations and behaviors.

DOCA provides support for emulating virtio devices over the PCIe bus. The PCIe transport is commonly used for virtio devices. Configuration, discovery, and features related to PCIe (such as MSI-X and PCIe device hot plug/unplug) are managed through the DOCA DevEmu PCI APIs. This modular design enables each layer within the DOCA Device Emulation subsystem to manage its own business logic and facilitates seamless integration with the other layers, ensuring independent functionality and operation throughout the system.

This subsystem also includes device-specific libraries for various virtio device types (e.g., a library for a virtio-FS device).

From the host's perspective, there is no difference between para-virtual, DOCA-emulated, and actual hardware devices. The host uses the same virtio device drivers to operate the device under all circumstances.

Virtio device emulation is part of the DOCA Device Emulation subsystem. It is, therefore, recommended to read the following guides beforehand:

DOCA DevEmu Virtio is supported on the BlueField target only.

The BlueField must meet the following requirements

  • DOCA version 2.7.0 or greater

  • BlueField-3 firmware 32.41.1000 or higher

Info

Please refer to the DOCA Backward Compatibility Policy.

Library must be run with root privileges.

The DOCA DevEmu Virtio core library provides the following software abstractions:

  • Virtio type – extends the PCIe type, represents common/default virtio configurations of emulated virtio devices

  • Virtio device – extends the PCIe device, represents an instance of an emulated virtio device

  • Virtio IO context – represents a progress context which is responsible for processing virtio descriptors and their associated virtio queues

DOCA DevEmu Virtio library does not provide APIs to configure the entire BAR layout of the v irtio device as this configuration is done internally. However, the library offers APIs to configure some of the registers within the common configuration structure (see Virtio Device).

Virtio Common Configuration

According to the virtio specification, the common PCIe configuration structure layout is as follows:

virtio_pci_common_cfg

Copy
Copied!
            

struct virtio_pci_common_cfg { /* About the whole device. */ le32 device_feature_select; /* read-write */ le32 device_feature; /* read-only for driver */ le32 driver_feature_select; /* read-write */ le32 driver_feature; /* read-write */ le16 config_msix_vector; /* read-write */ le16 num_queues; /* read-only for driver */ u8 device_status; /* read-write */ u8 config_generation; /* read-only for driver */   /* About a specific virtqueue. */ le16 queue_select; /* read-write */ le16 queue_size; /* read-write */ le16 queue_msix_vector; /* read-write */ le16 queue_enable; /* read-write */ le16 queue_notify_off; /* read-only for driver */ le64 queue_desc; /* read-write */ le64 queue_driver; /* read-write */ le64 queue_device; /* read-write */ le16 queue_notify_data; /* read-only for driver */ le16 queue_reset; /* read-write */ };

The DOCA DevEmu Virtio core library provides the ability to configure some of the listed registers using the appropriate setters.

Virtio Type

The virtio type extends the PCIe type and describes the common/default configuration of emulated virtio devices , including the common virtio configuration space registers (such as num_queues, queue_size, and others).

Virtio type is currently read-only (i.e., only getter APIs are available to r etrieve information) . T he following methods can be used for this purpose:

  • doca_devemu_virtio_type_get_num_queues – for getting the default initial value of the num_queues register for the associated virtio devices

  • doca_devemu_virtio_type_get_queue_size – for getting the default initial value of the queue_size register for the associated virtio devices

  • doca_devemu_virtio_type_get_device_features_63_0 – for getting the default initial values of the device_feature bits (0-63) for the associated virtio devices

  • doca_devemu_virtio_type_get_config_generation – for getting the default initial value of the config_generation register for the associated virtio devices

The default virtio type is extended by a virtio device's specific type (e.g., virtio-FS type) and cannot be created on demand.

Virtio Device

The virtio device extends the PCIe device. Before using the DOCA DevEmu Virtio device, i t is recommended to read the guidelines of DOCA DevEmu PCI device and DOCA Core context configuration phase .

The virtio device is extended by a virtio-specific device (e.g., virtio FS device) and cannot be created on demand.

Virtio Device Configurations

The virtio device context can be configured to match the application use case and optimize the utilization of system resources.

Mandatory Configurations

The mandatory configurations are as follows:

  • doca_devemu_virtio_dev_set_num_required_running_virtio_io_ctxs – to set the number of required running virtio IO contexts to be bound to the virtio device context. The virtio device context does not move to running state (according to the DOCA Core context state machine) before having this amount of running virtio IO contexts bound to it.

  • doca_devemu_virtio_dev_event_reset_register – to register to the virtio device reset event. This configuration is mandatory

Optional Configurations

The optional configurations are as follows:

  • doca_devemu_virtio_dev_set_device_features_63_0 – to set the values of the device_feature bits (0-63). If not set, the default value is taken from the virtio type configuration.

  • doca_devemu_virtio_dev_set_num_queues – to set the value of the num_queues register. If not set, the default value is taken from the virtio type configuration.

  • doca_devemu_virtio_dev_set_queue_size – to set the value of the queue_size register for all virtio queues. If not set, the default value is taken from the virtio type configuration.

Events

DOCA DevEmu Virtio device exposes asynchronous events to notify about sudden changes, according to DOCA Core architecture.

Info

Common events are described in DOCA DevEmu PCI Device events and in DOCA Core Event.

Reset Event

The reset event allows users to receive notifications whenever the device reset flow is initialized by the device driver. Upon receiving this event, it is guaranteed that no further requests are routed to the user via any associated virtio IO context until the reset flow is completed.

To complete the reset flow the user must:

  1. Flush all outstanding requests back to the virtio IO context associated with the request.

  2. Perform one of the following:

    • Call doca_devemu_virtio_dev_reset_complete .

    • Follow FLR flow:

      1. doca_ctx_stop – stop the virtio device with its associated virtio IO contexts and wait until the device and its associated virtio IO contexts transition to idle state

      2. doca_ctx_start – start the virtio device with its associated virtio IO contexts and wait until the device and its associated virtio IO contexts transition to running state

Now, the device and its associated virtio IO contexts should be fully operational again, the device is allowed to route new requests via any associated virtio IO context.

Virtio IO

The virtio IO context extends the DOCA Core context. Before using the DOCA DevEmu Virtio IO, i t is recommended to read the guidelines of DOCA Core context configuration phase .

This context is associated with a single DOCA virtio device and is bound to the virtio device context upon start. The virtio IO context is a thread-unsafe object and is progressed by a single DOCA Core progress engine. Usually, users configure a single virtio IO context per BlueField core used by the application service.

The virtio IO context is responsible to route new incoming virtio requests towards the application and to complete handled requests back to the device driver. It can only route requests while in running state and when its associated virtio device is also in running state .

© Copyright 2024, NVIDIA. Last updated on Aug 21, 2024.