DOCA Verbs
This guide provides an overview and configuration instructions for the DOCA Verbs API.
DOCA Verbs is a Verbs-like API that exposes the low-level features of Remote Direct Memory Access (RDMA). By enabling direct access to remote machine memory without CPU interruptions, DOCA Verbs reduces context switching for I/O operations, resulting in lower latency and higher bandwidth compared to traditional network communication methods.
DOCA Verbs supports both InfiniBand and Ethernet using RoCE.
Key features:
Vendor Agnostic: Supports multiple RDMA protocols.
Verbs Compatibility: Built on established Verbs interfaces for easy migration from existing Verbs-based applications.
Low-Level Control: Provides direct access to hardware capabilities for optimal performance tuning.
Generic Design: Offers a unified API interface for various RDMA protocols.
DOCA Verbs operates on Verbs objects and is designed to ensure compatibility while extending capabilities for new RDMA protocols and hardware features. The library includes both native DOCA Verbs APIs and bridge APIs for seamless integration with existing IB Verbs applications.
For detailed API documentation and implementation, refer to the doca_verbs.h and doca_verbs_bridge.h header files.
This library follows the architecture of DOCA Core and Verbs, it is recommended to read the following sections before proceeding:
Libibverbs external guides
DOCA Verbs-based applications can run on either the host machine or the NVIDIA® BlueField® networking platform (DPU or SuperNIC).
The DOCA Verbs API is centered around Verbs objects, designed to utilize the RDMA NIC's capabilities for high-performance data transfer. Each object plays a specific role in enabling and managing RDMA operations.
The API is categorized into two main function types:
Control Path APIs: For initializing and configuring RDMA resources based on user requirements.
Data Path APIs: For performing RDMA operations like sending, receiving, reading, and writing data. These functions are bridge functions relying on IB Verbs structures ( e.g.,
ibv_send_wr,ibv_wc).
Objects
DOCA Verbs Device Attributes
For DOCA Device documentation, read DOCA Core Device.
A DOCA Device represents a hardware device that supports RDMA operations through the DOCA Verbs library, serving as the foundation for creating Verbs contexts, protection domains, and other necessary resources.
DOCA Verbs Device Attributes provide detailed information about the device's capabilities and limitations, including:
Maximum number of supported Queue Pairs (QPs), Completion Queues (CQs), Memory Regions (MRs), Protection Domains (PDs), Address Handles (AHs), and Shared Receive Queues (SRQs)
Support for different QP types (RC, UC)
Support for atomic operations
Support for specialized data paths like DPA or GPU acceleration
These attributes are typically queried during application initialization to validate resource requirements against hardware capabilities.
The doca_verbs_query_device function is used to retrieve device attributes:
doca_error_t doca_verbs_query_device(struct doca_verbs_context *context, struct doca_verbs_device_attr **verbs_device_attr)
DOCA Verbs Context
A DOCA Verbs Context represents a logical connection to an RDMA device, providing access to the RDMA NIC and serving as the root for other Verbs-related objects. It enables centralized cleanup and resource management for objects created under the Verbs library.
DOCA Verbs context can be created or imported (more information on IB Verbs Bridge section):
Create context from
devinfooribv_device:doca_error_t doca_verbs_context_create(struct doca_devinfo *devinfo, uint32_t flags, struct doca_verbs_context **verbs_context) doca_error_t doca_verbs_bridge_verbs_context_create(struct ibv_device *ibv_dev, uint32_t flags, struct doca_verbs_context **verbs_context)
DOCA manages the resources and is independent of the IBV resources.
The context can also be imported from an already existing IB Verbs context using the RDMA bridge:
doca_error_t doca_verbs_bridge_verbs_context_import(struct ibv_context *ibv_ctx, uint32_t flags, struct doca_verbs_context **verbs_context)
The user is responsible for cleanup operations, as the library does not manage the lifecycle of imported contexts.
NoteWhen importing using the bridge method, the IB context must remain open until the Verbs context is closed.
DOCA Verbs Protection Domain (PD)
A DOCA Verbs Protection Domain (PD) defines a logical security boundary within an RDMA device. It acts as a sandbox for grouping RDMA resources and ensures hardware-enforced isolation between applications or components sharing the same RDMA device.
The PD guarantees that all RDMA objects created within the same domain can trust and safely interact with each other, while any access from outside the domain is explicitly prohibited. This isolation creates a secure environment for RDMA operations.
Key features:
Memory Access Control: Restricts RDMA operations to authorized memory regions.
Hardware-Enforced Security: Utilizes hardware mechanisms to prevent unauthorized access.
Application Isolation: Enforces separation between different applications using the same RDMA device.
Safe Resource Grouping: Organizes related RDMA resources (e.g., memory regions, queue pairs) within a trusted context.
Interoperability with IB Verbs
You can import an existing
ibv_pdinstance into DOCA Verbs using:doca_error_t doca_verbs_bridge_verbs_pd_import(struct ibv_pd *pd, struct doca_verbs_pd **verbs_pd)
For use cases such as registering memory with
ibv_reg_mr(), retrieve the underlying IB Verbs PD:struct ibv_pd *doca_verbs_bridge_verbs_pd_get_ibv_pd(
conststruct doca_verbs_pd *verbs_pd)
PD Creation and Usage
Create a DOCA Verbs PD
doca_error_t doca_verbs_pd_create(struct doca_verbs_context *verbs_context, struct doca_verbs_pd **verbs_pd);
When initializing a queue pair (QP), associate it with the PD:
doca_error_t doca_verbs_qp_init_attr_set_pd(struct doca_verbs_qp_init_attr *verbs_qp_init_attr, struct doca_verbs_pd *pd);
Memory Region (MR)
A Memory Region (MR) is a block of memory registered with the RDMA NIC to enable direct memory access through RDMA operations. Registering a memory region allows the NIC to perform zero-copy data transfers while enforcing access permissions and hardware-level protections.
Access Permissions
When registering an MR, the following access permissions can be enabled:
Local Write – Allows local RDMA operations to write to the memory.
Remote Read – Allows remote RDMA peers to read from the memory.
Remote Write – Allows remote RDMA peers to write to the memory.
Atomic Operations – Enables support for remote atomic operations (e.g., compare-and-swap, fetch-and-add).
Memory Keys
Each MR is associated with two keys used to authorize RDMA access:
Local Key (lkey) – Used by the local RDMA NIC for internal access to the memory.
Remote Key (rkey) – Shared with remote peers to grant them access, based on the permissions set during registration.
Registration
Memory regions are registered using the ibv_reg_mr() function from the RDMA Core API. This call:
Allocates the necessary hardware resources for the memory region.
Returns the local and remote keys (lkey and rkey).
Enforces the specified access permissions for RDMA operations.
DOCA Verbs Queue Pair
A DOCA Verbs Queue Pair (QP) represents a connection between two RDMA-capable nodes. Each QP consists of a Send Queue and a Receive Queue, which process Work Requests (WRs) for RDMA operations.
QP Types
RC (Reliable Connection) – Guarantees reliable, ordered delivery of messages.
UC (Unreliable Connection) – Provides best-effort delivery without guaranteed ordering or reliability.
QP Initialization
Before creating a QP, users must first configure the desired attributes using a doca_verbs_qp_init_attr object. This object specifies:
Send and receive queue sizes
Associated Completion Queues (CQs)
Protection Domain (PD)
Optional data-path configuration
The configured initialization object is then used to instantiate the QP.
doca_error_t doca_verbs_qp_init_attr_create(struct doca_verbs_qp_init_attr **verbs_qp_init_attr);
doca_error_t doca_verbs_qp_create(struct doca_verbs_context *verbs_context,
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_verbs_qp **verbs_qp);
QP Modification
After creation, QPs may require state transitions or updates to their configuration. This is done using a doca_verbs_qp_attr object.
Common use cases include transitioning a QP from INIT → RTR (Ready to Receive) → RTS (Ready to Send) or updating addressing and connection parameters.
doca_error_t doca_verbs_qp_attr_create(struct doca_verbs_qp_attr **verbs_qp_attr);
doca_error_t doca_verbs_qp_modify(struct doca_verbs_qp *verbs_qp,
struct doca_verbs_qp_attr *verbs_qp_attr,
int attr_mask);
QP Query
To inspect the current configuration or state of an existing QP, use the doca_verbs_qp_query() function. This retrieves attributes and initialization parameters according to the specified mask.
doca_error_t doca_verbs_qp_query(struct doca_verbs_qp *verbs_qp,
struct doca_verbs_qp_attr *verbs_qp_attr,
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
int *attr_mask);
External Data Path Mode
DOCA Verbs supports an external data path mode, allowing the data path to be implemented by the user or a third-party library (e.g., doca_dpa or doca_gpunetio). To enable this mode:
doca_error_t doca_verbs_qp_init_attr_set_external_datapath_en(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
uint8_t external_datapath_en);
Refer to the Bridge Execution Phase and Alternative Data Path Options sections for more details on integrating external data-path logic.
DOCA Verbs Completion Queue
A DOCA Verbs Completion Queue (CQ) is a data structure that collects notifications for completed RDMA operations. CQs are essential for tracking the status of Send and Receive Work Requests (WRs) submitted to a Queue Pair (QP).
Associating CQs with a QP
During QP initialization, the application must specify the CQs for send and receive operations:
doca_error_t doca_verbs_qp_init_attr_set_send_cq(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_verbs_cq *send_cq);
doca_error_t doca_verbs_qp_init_attr_set_receive_cq(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_verbs_cq *receive_cq);
CQ Initialization
A doca_verbs_cq_attr object defines the configuration for the CQ, including:
Queue size
Entry size
Completion channel binding
Data-path mode settings
To create and configure a CQ:
doca_error_t doca_verbs_cq_attr_create(struct doca_verbs_cq_attr **verbs_cq_attr);
doca_error_t doca_verbs_cq_create(struct doca_verbs_context *verbs_context,
struct doca_verbs_cq_attr *verbs_cq_attr,
struct doca_verbs_cq **verbs_cq);
External Data Path Mode
DOCA Verbs supports an external data path mode for advanced integration with external processing frameworks (e.g., doca_dpa, doca_gpunetio). In this mode, CQ data management and notification handling are offloaded outside of DOCA Verbs.
To enable this mode:
doca_error_t doca_verbs_cq_attr_set_external_datapath_en(
struct doca_verbs_cq_attr *cq_attr,
uint8_t external_datapath_en);
Refer to the Bridge Execution Phase and Alternative Data Path Options sections for more details.
DOCA Verbs Address Handle Attribute
A DOCA Verbs Address Handle Attribute (AH) is a data structure that encapsulates the addressing information required to establish communication with remote Queue Pairs (QPs). It is typically used during QP state transitions—such as INIT → RTR (Ready to Receive)—to define the destination addressing parameters.
Key Attributes
The AH structure may include the following fields depending on the address type and transport:
Address Type – Specifies the type of addressing:
IPv4
IPv6
InfiniBand (with or without GRH)
GID (Global Identifier) – Used for global routing in RoCE or IB with GRH.
LID (Local Identifier) – Used for local routing in InfiniBand.
Hop Limit – Specifies the maximum number of network hops (TTL equivalent).
Service Level (SL) – Indicates QoS priority level on the fabric.
Traffic Class – Classifies packets for differentiated services in IP-based networks.
Creating and Associating AH Attributes
To define an address handle and attach it to a QP modification request:
doca_error_t doca_verbs_ah_attr_create(
struct doca_verbs_context *verbs_context,
struct doca_verbs_ah_attr **verbs_ah);
doca_error_t doca_verbs_qp_attr_set_ah_attr(
struct doca_verbs_qp_attr *verbs_qp_attr,
struct doca_verbs_ah_attr *ah_attr);
The configured AH attribute can then be used during QP state transitions, particularly in INIT2RTR operations, to ensure proper remote addressing.
DOCA Verbs Completion Channel
A DOCA Verbs Completion Channel (CC) is a mechanism for managing asynchronous completion events from one or more associated Completion Queues (CQs). Completion channels enable efficient event-driven processing, reducing the need for continuous polling.
Creation and Association
To create a completion channel and associate it with a CQ:
doca_error_t doca_verbs_comp_channel_create(
struct doca_verbs_context *verbs_context,
struct doca_verbs_comp_channel **verbs_comp_channel);
doca_error_t doca_verbs_cq_attr_set_comp_channel(
struct doca_verbs_cq_attr *cq_attr,
struct doca_verbs_comp_channel *comp_channel);
Receiving and Acknowledging Events
To retrieve the next event (i.e., notification of completed work) from the completion channel:
doca_error_t doca_verbs_get_cq_event(
struct doca_verbs_comp_channel *verbs_comp_channel,
struct doca_verbs_cq **verbs_cq,
void **cq_context);
Each event retrieved using doca_verbs_get_cq_event() must be acknowledged using:
void doca_verbs_ack_cq_events(struct doca_verbs_cq *verbs_cq, unsigned int nevents);
Integration with Epoll
The completion channel can be integrated into a standard event loop using epoll for scalable I/O multiplexing. To retrieve the file descriptor:
doca_event_handle_t doca_verbs_comp_channel_get_handle(
struct doca_verbs_comp_channel *verbs_comp_channel);
This handle can be registered with epoll to receive readiness notifications, enabling synchronization across multiple RDMA and non-RDMA events.
DOCA Verbs Shared Received Queue (SRQ)
A Shared Receive Queue (SRQ) allows multiple RDMA Queue Pairs (QPs) to pull receive buffers from a common pool. Instead of maintaining separate receive queues per QP, the SRQ model supports scalable, efficient buffer sharing—especially useful in high-connection-count scenarios.
When work requests (WRs) are posted to the SRQ, any associated QP can consume them upon receiving data. Receive completions are still reported via the Completion Queue (CQ) attached to the individual QP, not the SRQ.
Advantages of SRQ-Based Design
Scalable Design – Reduces the overhead of managing thousands of per-QP receive queues.
Efficient Buffer Utilization – Ideal for servers with many clients where only a subset are active simultaneously.
Lower Memory Footprint – A shared buffer pool minimizes overall memory usage.
Simplified Buffer Management – Centralized posting reduces CPU and management complexity.
SRQ Types
Linked List SRQ
Receive Work Queue (WQ) is organized as a linked list.
Supports out-of-order (OOO) processing across different QPs.
Supported by DOCA Verbs data path.
Contiguous SRQ
Receive WQ is managed like a regular per-QP Receive Queue (RQ).
Supported only in external data-path mode (e.g., via
doca_dpa,doca_gpunetio).
Associating SRQ with a QP
During QP initialization, an SRQ can be linked to a QP to enable shared receive functionality:
doca_error_t doca_verbs_qp_init_attr_set_srq(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_verbs_srq *srq);
SRQ Initialization
An srq_init_attr object is used to configure SRQ creation parameters, including:
Queue size
Maximum scatter/gather elements per WR
SRQ type (linked list or contiguous)
Data-path configuration
To create and configure an SRQ:
doca_error_t doca_verbs_srq_init_attr_create(
struct doca_verbs_srq_init_attr **verbs_srq_init_attr);
doca_error_t doca_verbs_srq_create(
struct doca_verbs_context *verbs_context,
struct doca_verbs_srq_init_attr *verbs_srq_init_attr,
struct doca_verbs_srq **verbs_srq);
Query SRQ Attributes:
Use srq_query() to retrieve the current configuration and parameters of an SRQ:
doca_error_t doca_verbs_srq_query(
struct doca_verbs_srq *verbs_srq,
struct doca_verbs_srq_init_attr *verbs_srq_init_attr);
External Data Path Mode
To offload the SRQ data path to an external engine (e.g., doca_dpa, doca_gpunetio), enable external data-path mode:
doca_error_t doca_verbs_srq_init_attr_set_external_datapath_en(
struct doca_verbs_srq_init_attr *verbs_srq_init_attr,
uint8_t external_datapath_en);
Refer to the Alternative Data Path Options section for further details.
DOCA Verbs Asynchronous Events
DOCA Verbs asynchronous events (async events) are non-blocking notifications that inform the application of significant hardware events or error conditions that occur during RDMA operations. These events are essential for detecting issues such as QP errors, memory registration failures, or transport problems.
Async events enable proactive error handling and health monitoring of RDMA resources without interrupting the primary application flow.
Retrieving and Acknowledging Events
To receive the next asynchronous event:
doca_error_t doca_verbs_get_async_event(
struct doca_verbs_context *verbs_context,
struct doca_verbs_async_event *async_event);
Each retrieved event must be acknowledged using the following function to release internal resources and prevent leaks:
void doca_verbs_ack_async_event(struct doca_verbs_async_event *async_event);
Event Handling Best Practices
Applications should use the event handle obtained from the DOCA Verbs context to integrate async event monitoring into their event loop (e.g., using
epoll).Async events should be processed promptly to avoid overflows in the internal event queue.
Acknowledgment is mandatory for each event to ensure proper resource cleanup.
DOCA Verbs External Resources
DOCA Verbs supports both DOCA-managed and user-managed (external) modes for managing the low-level hardware resources used in RDMA operations. These include memory regions for work queues, user access regions (UARs), and doorbell registers.
For background on memory management concepts, refer to the DOCA Core Memory Subsystem documentation.
Key External Resources
External UAR (User Access Region) – A memory-mapped region that provides direct access to hardware doorbell registers. UARs are used to signal the RDMA device when new work requests or completions are ready.
External UMEM (User Memory) – Represents the physical memory region backing the Work Queue (WQ) structures, which hold posted Send and Receive Work Requests (WRs).
External DBR UMEM (Doorbell Register UMEM) – A dedicated memory region used for doorbell registers that notify the hardware of new entries in the work queues.
Usage Modes
Mode | Description |
DOCA-managed (default) | DOCA internally allocates and manages all UAR, UMEM, and DBR UMEM resources. The application is abstracted from memory alignment, sizing, and cleanup. |
External mode | The application provides and manages its own UAR, UMEM, and DBR UMEM resources. This offers greater flexibility but requires a deep understanding of hardware constraints and careful memory management. External mode responsibilities:
|
Setting External Resources
To configure external resource pointers during QP initialization:
doca_error_t doca_verbs_qp_init_attr_set_external_umem(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_umem *external_umem,
uint64_t external_umem_offset);
doca_error_t doca_verbs_qp_init_attr_set_external_dbr_umem(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_umem *external_dbr_umem,
uint64_t external_dbr_umem_offset);
doca_error_t doca_verbs_qp_init_attr_set_external_uar(
struct doca_verbs_qp_init_attr *verbs_qp_init_attr,
struct doca_uar *external_uar);
IB Verbs Bridge
DOCA Verbs provides a bridge between the traditional InfiniBand Verbs (IB Verbs) interface and DOCA abstractions, enabling a layered and interoperable architecture for RDMA operations. This bridge allows applications to integrate DOCA's enhanced features while preserving compatibility with the low-level IB Verbs API.
Bridge Configuration Phase
This dual-layer architecture allows applications to mix and match DOCA Verbs APIs and IB Verbs APIs based on their requirements. Developers can, for example, manage resources using DOCA abstractions while executing operations using native IB Verbs data structures.
The
doca_rdma_verbs_contextis created usingdoca_devinfoand provides the entry point to the DOCA Verbs layer.The underlying
ibv_contextis created from anibv_deviceusing the standard IB Verbs API.The bridge components span across several API libraries:
doca_dev,doca_verbs,doca_verbs_bridge,doca_rdma_bridge, andlibibverbs.
The recommended sequence is to first create the DOCA Verbs context, and then retrieve the DOCA device handle from it—not the other way around.
The following diagram illustrates the relationships between components (not included here, but referenced):
Bridge Execution Phase
The bridge also supports execution-time interoperability. Applications can use IB Verbs work requests (ibv_send_wr, ibv_recv_wr) and completions (ibv_wc) with DOCA-managed QPs and CQs.
Posting Work Requests
Use IB Verbs WRs with DOCA Verbs QPs to submit send or receive operations:
int doca_verbs_bridge_post_send(
struct doca_verbs_qp *verbs_qp,
struct ibv_send_wr *wr,
struct ibv_send_wr **bad_wr);
int doca_verbs_bridge_post_recv(
struct doca_verbs_qp *verbs_qp,
struct ibv_recv_wr *wr,
struct ibv_recv_wr **bad_wr);
Polling Completions
Poll a DOCA Verbs CQ for completions and retrieve results as an array of ibv_wc entries:
int doca_verbs_bridge_poll_cq(
struct doca_verbs_cq *verbs_cq,
int num_entries,
struct ibv_wc *ibv_wc);
This enables applications to maintain compatibility with existing RDMA workflows while benefiting from DOCA's abstraction, resource management, and hardware integration.
DPA Data Path
The DPA Data Path enables offloading RDMA operations from the host CPU to the device (e.g., BlueField DPU), allowing RDMA work to be initiated and completed entirely in the data plane. This mode is especially useful for accelerating networking applications by reducing host CPU involvement.
Configuration Phase
The DPA data path enables execution of RDMA operations directly on the device, bypassing the host CPU. This offload model improves performance and scalability for data-centric applications.
Key differences from host-side (CPU) configuration:
External data path must be explicitly enabled.
All RDMA operations, including buffer allocation and queue management, are performed on the device.
RDMA queues are created to operate entirely within the DPA context.
To create a doca_verbs_qp in DPA mode, use one of the following approaches:
Set DPA context in QP initialization attributes:
doca_error_t doca_verbs_qp_init_attr_set_dpa( struct doca_verbs_qp_init_attr *verbs_qp_init_attr, struct doca_dpa *dpa_ctx);
In this mode, external UMEM (for WQ and DBR), external UAR, and PD are created or retrieved internally using the provided DPA context. This is the simplified configuration mode, where DOCA manages all required resources.
Set external UMEM/UAR manually using the standard external resource APIs: In this advanced configuration mode, the application allocates external UMEM regions for WQ and DBR, and provides its own UAR. This offers maximum control over memory layout, alignment, and resource lifecycle.
Use a DPA completion context instead of a traditional Verbs CQ:
doca_error_t doca_verbs_qp_init_attr_set_send_dpa_completion( struct doca_verbs_qp_init_attr *verbs_qp_init_attr, struct doca_dpa_completion *send_dpa_completion);
To use the QP from within the DPA kernel, retrieve the DPA QP handle using:
doca_error_t doca_verbs_qp_get_dpa_handle(
struct doca_verbs_qp *verbs_qp,
struct doca_dpa *dpa_ctx,
doca_dpa_dev_verbs_qp_t *dpa_qp_handle);
The returned dpa_qp_handle can then be used to post send and receive work requests from DPA-side execution flows.
Execution Phase
Refer to section "GPU Functions – Verbs" for information.
GPU Data Path
The GPU data path enables RDMA operations to be initiated and completed directly on the GPU, bypassing the host CPU entirely. This mode is essential for accelerating GPU-to-GPU and GPU-to-network communication in high-performance computing and AI workloads.
Configuration Phase
Main differences from host-side (CPU) configuration:
External data path must be explicitly enabled.
All RDMA operations, including buffer management, are performed on the device side.
The application must explicitly allocate GPU memory and bind it to DOCA Verbs structures.
To create a doca_verbs_qp for GPU use:
Allocate a GPU buffer and use it to create a UMEM object, then assign it to the QP as external UMEM:
doca_error_t doca_verbs_qp_init_attr_set_external_umem( struct doca_verbs_qp_init_attr *verbs_qp_init_attr, struct doca_umem *external_umem, uint64_t external_umem_offset);
Allocate another GPU buffer and use it to set an external UAR:
doca_error_t doca_verbs_qp_init_attr_set_external_uar( struct doca_verbs_qp_init_attr *verbs_qp_init_attr, struct doca_uar *external_uar);
To create a doca_verbs_cq for GPU use:
Allocate a GPU buffer, wrap it as a UMEM object, and assign it as the external UMEM for the CQ:
doca_error_t doca_verbs_cq_attr_set_external_umem( struct doca_verbs_cq_attr *cq_attr, struct doca_umem *external_umem, uint64_t external_umem_offset);
Enable CQ overrun mode, as it is required by most GPU-side data path operations:
doca_error_t doca_verbs_cq_attr_set_cq_overrun( struct doca_verbs_cq_attr *cq_attr, uint8_t overrun);
To use the QP on the GPU side, export a handle using DOCA GPUNetIO :
doca_error_t doca_gpu_verbs_export_qp(
struct doca_gpu *gpu_dev,
struct doca_dev *dev,
struct doca_verbs_qp *qp,
enum doca_gpu_dev_verbs_nic_handler nic_handler,
void *gpu_qp_umem_dev_ptr,
struct doca_verbs_cq *cq_sq,
struct doca_verbs_cq *cq_rq,
struct doca_gpu_verbs_qp **qp_out);
The exported doca_gpu_verbs_qp handle enables GPU kernels to post send and receive work requests directly from device code, facilitating low-latency, high-throughput RDMA operations within GPU execution contexts.
Execution Phase
Refer to section "GPU Functions – Verbs" for information.
Verbs Server Client
The doca_verbs_server_client sample demonstrates how to perform RDMA verbs operations using DOCA Verbs for a server-client mechanism. The sample queries the device port and executes the data path according to the link layer configuration (IB or RoCE).
Sample logic:
Resource setup:
Creating
verbs_ctxfrom device name given as an argument .Creating
verbs_pdfromverbs_ctx.Creating and setting
cq_attrto createverbs_cq.Creating
verbs_ahand setting attributes .Creating and setting
qp_init_attrto createverbs_qp.Allocating local memory to execute the RDMA operations using
verbs_pdandibv_reg_mr.
Establishing connection between server and client using TCP socket.
Exchanging RDMA parameters: Server and client exchange local buffer addresses, MKEYs, QP numbers and GID/LID addresses
QP connection: Setting QP attributes, Modify QP states – RST →INIT → RTR → RTS
Data Path Execution: Performing RDMA operations according to user flags:
Send/receive operation –
Client side – post send WR (opcode
IBV_WR_RDMA_SEND), poll CQ completionServer side – post receive WR, poll CQ completion, read the message from the client via local buffer
Write operation –
Client side – post send WR (opcode
IBV_WR_RDMA_WRITEand remote address), poll CQ completionServer side – poll the local buffer and wait for the expected message
Read operation –
Client side – post send WR (opcode
IBV_WR_RDMA_READand remote address), poll CQ completionServer side – no action needed
Syncing completions of local and remote peer via socket message.
Verifying completions or received messages.
Cleanup: Destroying QP, AH, CQ, PD, Verbs Context, close device, TCP connection and free memory buffers.
Sample usage:
Usage: doca_verbs_server_client [DOCA Flags] [Program Flags]
DOCA Flags:
-h, --help Print a help synopsis
-v, --version Print program version information
-l, --log-level Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
--sdk-log-level Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
Program Flags:
-c, --client <Sample is client, requires server OOB IP> Sample is client, requires server OOB IP
-d, --device <IB device name> IB device name
-gid, --gid-index GID index for DOCA RDMA. Default value 0
-ro, --rdma-oper-type RDMA operation type: Send, Write, Read. Default type Send
-s, --buffer-size Buffer size. Default value 64
-qt, --qp-type QP Type: RC. Default type RC
-n, --iters Number of iterations. Default value 1
-b, --send-burst Send all messages at once in a burst
Reference:
/opt/mellanox/doca/samples/doca_verbs/verbs_server_client/verbs_server_client_sample.c/opt/mellanox/doca/samples/doca_verbs/verbs_server_client/verbs_server_client_main.c/opt/mellanox/doca/samples/doca_verbs/verbs_server_client/meson.build
Verbs Wait CQ WR
This sample illustrates how to use the wait_cq_pi work request to chain messages and force order of execution.
Sample logic:
Locating DOCA device.
Initializing required DOCA Core structures.
Creating two pairs of QPs, each pair connected between them (QP1, QP2, QP3 and QP4 where QP1 is connected to QP2, QP3 is connected to QP4).
Creating three buffers and populate the first with data.
The send flows:
QP1 sends data from the first buffer to QP2, which receives the data in the second buffer.
QP3 sends the data from the second buffer to QP4, which receives the data in the third buffer.
NoteFor the data to arrive properly in the third buffer, the send operation on QP3 must be done only after the send operation on QP1 finishes successfully.
Posting recv WRs on QP2 and QP4 to prepare for receiving the data.
Posting a
wait_cq_piWR on QP3 to wait with the send execution until QP1 finishes sending its data.Posting the send WR on QP3 which is only executed after the
wait_cq_piWR finishes waiting.Posting the send WR on QP1 to start the data transfer.
Waiting for all WRs to complete.
Assuring the data on the third buffer is identical to the data on the first, indicating the WRs were executed in the correct order.
Sample usa ge:
Usage: wait_cq_wr [DOCA Flags] [Program Flags]
DOCA Flags:
-h, --help Print a help synopsis
-v, --version Print program version information
-l, --log-level Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
--sdk-log-level Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
Program Flags:
-d, --device-name IBV device name to be used.
-gi, --gid-index GID index to use, deafult: 0.
--ipv6 Use IPv6 instead of IPv4
References:
/opt/mellanox/doca/samples/doca_verbs/verbs_wait_cq_wr/verbs_wait_cq_wr_sample.c/opt/mellanox/doca/samples/doca_verbs/verbs_wait_cq_wr/verbs_wait_cq_wr_main.c/opt/mellanox/doca/samples/doca_verbs/verbs_wait_cq_wr/meson.build