RDMA Aware Networks Programming User Manual

Protection Domain Operations

Once you have established a protection domain (PD), you may create objects within that domain. This section describes operations available on a PD. These include registering memory regions (MR), creating queue pairs (QP) or shared receive queues (SRQ) and address handles (AH).

Template: struct ibv_mr {}ibv_reg_mr{*}(struct ibv_pd *pd, void *addr, size_t length, enum ibv_access_flags access)

Input Parameters:

pd protection domain, struct ibv_pd from ibv_alloc_pd

addr memory base address

length length of memory region in bytes

access access flags

Output Parameters: none

Return Value: pointer to created memory region (MR) or NULL on failure.

Description:
ibv_reg_mr registers a memory region (MR), associates it with a protection domain (PD), and assigns it local and remote keys (lkey, rkey). All VPI commands that use memory require the memory to be registered via this command. The same physical memory may be mapped to differ- ent MRs even allowing different permissions or PDs to be assigned to the same memory, depending on user requirements.
Access flags may be bitwise or one of the following enumerations:

IBV_ACCESS_LOCAL_WRITE Allow local host write access

IBV_ACCESS_REMOTE_WRITE Allow remote hosts write access

IBV_ACCESS_REMOTE_READ Allow remote hosts read access

IBV_ACCESS_REMOTE_ATOMIC Allow remote hosts atomic access

IBV_ACCESS_MW_BIND Allow memory windows on this MR

Local read access is implied and automatic.
Any VPI operation that violates the access permissions of the given memory operation will fail. Note that the queue pair (QP) attributes must also have the correct permissions or the operation will fail.
If IBV_ACCESS_REMOTE_WRITE or IBV_ACCESS_REMOTE_ATOMIC is set, then IBV_ACCESS_LOCAL_WRITE must be set as well.

struct ibv_mr is defined as follows:

Copy
Copied!
            

struct ibv_mr { struct ibv_context*context; struct ibv_pd *pd; void*addr; size_tlength; uint32_thandle; uint32_tlkey; uint32_trkey; };

Template: int ibv_dereg_mr(struct ibv_mr *mr)

Input Parameters: mr struct ibv_mr from ibv_reg_mr

Output Parameters: none

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_dereg_mr frees a memory region (MR). The operation will fail if any memory windows (MW) are still bound to the MR.

Template: struct ibv_qp {}ibv_create_qp{*}(struct ibv_pd *pd, struct ibv_qp_init_attr *qp_init_attr)

Input Parameters: pd struct ibv_pd from ibv_alloc_pd qp_init_attrinitial attributes of queue pair

Output Parameters: qp_init_attractual values are filled in

Return Value: pointer to created queue pair (QP) or NULL on failure.

Description: ibv_create_qp creates a QP. When a QP is created, it is put into the RESET state.

struct qp_init_attr is defined as follows:

Copy
Copied!
            

struct ibv_qp_init_attr { void *qp_context; struct ibv_cq *send_cq; struct ibv_cq *recv_cq; struct ibv_srq *srq; struct ibv_qp_cap cap; enum ibv_qp_type qp_type; int sq_sig_all; struct ibv_xrc_domain *xrc_domain; };   qp_context (optional) user defined value associated with QP. send_cq send CQ. This must be created by the user prior to calling ibv_create_qp. recv_cq receive CQ. This must be created by the user prior to calling ibv_create_qp. It may be the same as send_cq. srq (optional) shared receive queue. Only used for SRQ QP’s. cap defined below. qp_type must be one of the following: IBV_QPT_RC = 2, IBV_QPT_UC, IBV_QPT_UD, IBV_QPT_XRC, IBV_QPT_RAW_PACKET = 8, IBV_QPT_RAW_ETH = 8 sq_sig_all If this value is set to 1, all send requests (WR) will generate completion queue events (CQE). If this value is set to 0, only WRs that are flagged will generate CQE’s (see ibv_post_send). xrc_domain (Optional) Only used for XRC operations.   struct ibv_qp_cap is defined as follows:   struct ibv_qp_cap { uint32_t max_send_wr; uint32_t max_recv_wr; uint32_t max_send_sge; uint32_t max_recv_sge; uint32_t max_inline_data; };   max_send_wr Maximum number of outstanding send requests in the send queue. max_recv_wr Maximum number of outstanding receive requests (buffers) in the receive queue. max_send_sge Maximum number of scatter/gather elements (SGE) in a WR on the send queue. max_recv_sge Maximum number of SGEs in a WR on the receive queue. max_inline_data Maximum size in bytes of inline data on the send queue.

Template: int ibv_destroy_qp(struct ibv_qp *qp)

Input Parameters: qp struct ibv_qp from ibv_create_qp

Output Parameters: none

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_destroy_qp frees a queue pair (QP).

Template: struct ibv_srq *ibv_create_srq(struct ibv_pd *pd, struct ibv_srq_init_attr *srq_init_attr)

Input Parameters:

pd The protection domain associated with the shared receive queue (SRQ)

srq_init_attr A list of initial attributes required to create the SRQ

Output Parameters: ibv_srq attrActual values of the struct are set

Return Value: A pointer to the created SRQ or NULL on failure

Description: ibv_create_srq creates a shared receive queue (SRQ). srq_attr->max_wr and srq_attr->max_sge are read to determine the requested size of the SRQ, and set to the actual values allocated on return. If ibv_create_srq succeeds, then max_wr and max_sge will be at least as large as the requested values.

struct ibv_srq is defined as follows:

Copy
Copied!
            

struct ibv_srq { struct ibv_context *context; struct ibv_context from ibv_open_device void *srq_context; struct ibv_pd *pd; Protection domain uint32_t handle; pthread_mutex_t mutex; pthread_cond_t cond; uint32_t events_completed; } struct ibv_srq_init_attr is defined as follows: struct ibv_srq_init_attr { void *srq_context; struct ibv_srq_attr attr; };   srq_context struct ibv_context from ibv_open_device attr An ibv_srq_attr struct defined as follows:       struct ibv_srq_attr is defined as follows:   struct ibv_srq_attr { uint32_t max_wr; uint32_t max_sge; uint32_t srq_limit; };   max_wr Requested maximum number of outstanding WRs in the SRQ max_sge Requested number of scatter elements per WR srq_limit; The limit value of the SRQ (irrelevant for ibv_create_srq)

Template: int ibv_modify_srq (struct ibv_srq *srq, struct ibv_srq_attr *srq_attr, int srq_attr_mask)

Input Parameters:

srq The SRQ to modify

srq_attr Specifies the SRQ to modify (input)/the current values of the selected SRQ attributes are returned (output)

srq_attr_mask A bit-mask used to specify which SRQ attributes are being modified

Output Parameters: srq_attr The struct ibv_srq_attr is returned with the updated values

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_modify_srq modifies the attributes of the SRQ srq using the attribute values in srq_attr based on the mask srq_attr_mask. srq_attr is an ibv_srq_attr struct as defined above under the verb ibv_create_srq. The argument srq_attr_mask specifies the SRQ attributes to be modified. It is either 0 or the bitwise OR of one or more of the flags:

IBV_SRQ_MAX_WR Resize the SRQ

IBV_SRQ_LIMIT Set the SRQ limit

If any of the attributes to be modified is invalid, none of the attributes will be modified. Also, not all devices support resizing SRQs. To check if a device supports resizing, check if the IBV_DE- VICE_SRQ_RESIZE bit is set in the device capabilities flags.
Modifying the SRQ limit arms the SRQ to produce an IBV_EVENT_SRQ_LIMIT_REACHED 'low watermark' async event once the number of WRs in the SRQ drops below the SRQ limit.

Template: int ibv_destroy_srq(struct ibv_srq *srq)

Input Parameters:

srq The SRQ to destroy

Output Parameters: none

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_destroy_srq destroys the specified SRQ. It will fail if any queue pair is still associated with this SRQ.

Template: struct ibv_xrc_domain *ibv_open_xrc_domain(struct ibv_context *context, int fd, int oflag)

Input Parameters:

context struct ibv_context from ibv_open_device

fd The file descriptor to be associated with the XRC domain

oflag The desired file creation attributes

Output Parameters: A file descriptor associated with the opened XRC domain

Return Value: A reference to an opened XRC domain or NULL

Description: ibv_open_xrc_domain opens an eXtended Reliable Connection (XRC) domain for the RDMA device context. The desired file creation attributes oflag can either be 0 or the bitwise OR of O_CREAT and O_EXCL. If a domain belonging to the device named by the context is already associated with the inode, then the O_CREAT flag has no effect. If both O_CREAT and O_XCL are set, open will fail if a domain associated with the inode already exists. Otherwise a new XRC domain will be created and associated with the inode specified by fd.
Please note that the check for the existence of the domain and creation of the domain if it does not exist is atomic with respect to other processes executing open with fd naming the same inode.
If fd equals -1, then no inode is associated with the domain, and the only valid value for oflag is O_CREAT.
Since each ibv_open_xrc_domain call increments the xrc_domain object's reference count, each such call must have a corresponding ibv_close_xrc_domain call to decrement the xrc_domain object's reference count.

Template:

struct ibv_srq {}ibv_create_xrc_srq{*}(struct ibv_pd *pd,

struct ibv_xrc_domain *xrc_domain,

struct ibv_cq *xrc_cq,

struct ibv_srq_init_attr *srq_init_attr)

Input Parameters:

pd The protection domain associated with the shared receive queue

xrc_domain The XRC domain

xrc_cq The CQ which will hold the XRC completion

srq_init_attr A list of initial attributes required to create the SRQ (described above)

Output Parameters: ibv_srq_attrActual values of the struct are set

Return Value: A pointer to the created SRQ or NULL on failure

Description: ibv_create_xrc_srq creates an XRC shared receive queue (SRQ) associated with the protection domain pd, the XRC domain domain_xrc and the CQ which will hold the completion xrc_cq

struct ibv_xrc_domain is defined as follows:

Copy
Copied!
            

struct ibv_xrc_domain { struct ibv_context *context; struct ibv_context from ibv_open_device uint64_t handle; }

Template: int ibv_close_xrc_domain(struct ibv_xrc_domain *d)

Input Parameters:

d A pointer to the XRC domain the user wishes to close

Output Parameters: none

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_close_xrc_domain closes the XRC domain, d. If this happens to be the last reference, then the XRC domain will be destroyed. This function decrements a reference count and may fail if any QP or SRQ are still associated with the XRC domain being closed.

Template: int ibv_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, uint32_t *xrc_rcv_qpn)

Input Parameters:

init_attr The structure to be populated with QP information

xrc_rcv_qpn The QP number associated with the receive QP to be created

Output Parameters:

init_attr Populated with the XRC domain information the QP will be associated with

xrc_rcv_qpn The QP number associated with the receive QP being created

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_create_xrc_rcv_qp creates an XRC queue pair (QP) to serve as a receive side only QP and returns the QP number through xrc_rcv_qpn. This number must be passed to the remote (sender) node. The remote node will use xrc_rcv_qpn in ibv_post_send when it sends messages to an XRC SRQ on this host in the same xrc domain as the XRC receive QP.
The QP with number xrc_rcv_qpn is created in kernel space and persists until the last process registered for the QP called ibv_unreg_xrc_rcv_qp, at which point the QP is destroyed. The pro- cess which creates this QP is automatically registered for it and should also call ibv_unreg_x- rc_rcv_qp at some point to unregister.
Any process which wishes to receive on an XRC SRQ via this QP must call ibv_reg_xrc_rcv_qp for this QP to ensure that the QP will not be destroyed while they are still using it.
Please note that because the QP xrc_rcv_qpn is a receive only QP, the send queue in the init_attr struct is ignored.

Template: int ibv_modify_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num, struct ibv_qp_attr *attr, int attr_mask)

Input Parameters:

xrc_domain The XRC domain associated with this QP xrc_qp_num The queue pair number to identify this QP

attr The attributes to use to modify the XRC receive QP attr_mask The mask to use for modifying the QP attributes

Output Parameters: None

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_modify_xrc_rcv_qp modifies the attributes of an XRC receive QP with the number xrc_qp_num which is associated with the attributes in the struct attr according to the mask attr_mask. It then moves the QP through the following transitions: Reset->Init->RTR
At least the following masks must be set (the user may add optional attributes as needed)

Next State

Next State Required attributes

Init

IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS

RTR

IBV_QP_STATE, IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN, IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER

Warning

Please note that if any attribute to modify is invalid or if the mask as invalid values, then none of the attributes will be modified, including the QP state.

Template: int ibv_reg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num)

Input Parameters:

xrc_domain The XRC domain associated with the receive QP

xrc_qp_num The number associated with the created QP to which the user process is to be registered

Output Parameters: None

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_reg_xrc_rcv_qp registers a user process with the XRC receive QP whose number is xrc_qp_num associated with the XRC domain xrc_domain.
This function may fail if the number xrc_qp_num is not the number of a valid XRC receive QP (for example if the QP is not allocated or it is the number of a non-XRC QP), or the XRC receive QP was created with an XRC domain other than xrc_domain.

Template: int ibv_unreg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num)

Input Parameters:

xrc_domain The XRC domain associated with the XRC receive QP from which the user wishes to unregister

xrc_qp_num TheQPnumberfromwhich the user process istobe unregistered

Output Parameters: None

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_unreg_xrc_rcv_qp unregisters a user process from the XRC receive QP number xrc_qp_- num which is associated with the XRC domain xrc_domain. When the number of user processes registered with this XRC receive QP drops to zero, the QP is destroyed.

Template: struct ibv_ah {}ibv_create_ah{*}(struct ibv_pd *pd, struct ibv_ah_attr *attr)

Input Parameters:

pd struct ibv_pd from ibv_alloc_pd

attr attributes of address

Output Parameters: none

Return Value: pointer to created address handle (AH) or NULL on failure.

Description: ibv_create_ah creates an AH. An AH contains all of the necessary data to reach a remote destination. In connected transport modes (RC, UC) the AH is associated with a queue pair (QP). In the datagram transport modes (UD), the AH is associated with a work request (WR).

struct ibv_ah_attr is defined as follows:

Copy
Copied!
            

struct ibv_ah_attr { struct ibv_global_route grh; uint16_t dlid; uint8_t sl; uint8_t src_path_bits; uint8_t static_rate; uint8_t is_global; uint8_t port_num; };   grh defined below dlid destination lid sl service level src_path_bits source path bits static_rate static rate is_global this is a global address, use grh. port_num physical port number to use to reach this destination struct ibv_global_route is defined as follows:   struct ibv_global_route { union ibv_gid dgid; uint32_t flow_label; uint8_t sgid_index; uint8_t hop_limit; uint8_t traffic_class; };   dgid destination GID (see ibv_query_gid for definition) flow_label flow label sgid_index index of source GID (see ibv_query_gid) hop_limit hop limit traffic_class traffic class

Template: int ibv_destroy_ah(struct ibv_ah *ah)

Input Parameters:

ah struct ibv_ah from ibv_create_ah

Output Parameters: none

Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.

Description: ibv_destroy_ah frees an address handle (AH). Once an AH is destroyed, it can't be used any- more in UD QPs

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.