Protection Domain Operations
Once you have established a protection domain (PD), you may create objects within that domain. This section describes operations available on a PD. These include registering memory regions (MR), creating queue pairs (QP) or shared receive queues (SRQ) and address handles (AH).
Template: struct ibv_mr {}ibv_reg_mr{*}(struct ibv_pd *pd, void *addr, size_t length, enum ibv_access_flags access)
Input Parameters:
pd protection domain, struct ibv_pd from ibv_alloc_pd
addr memory base address
length length of memory region in bytes
access access flags
Output Parameters: none
Return Value: pointer to created memory region (MR) or NULL on failure.
Description:
ibv_reg_mr registers a memory region (MR), associates it with a protection domain (PD), and assigns it local and remote keys (lkey, rkey). All VPI commands that use memory require the memory to be registered via this command. The same physical memory may be mapped to differ- ent MRs even allowing different permissions or PDs to be assigned to the same memory, depending on user requirements.
Access flags may be bitwise or one of the following enumerations:
IBV_ACCESS_LOCAL_WRITE Allow local host write access
IBV_ACCESS_REMOTE_WRITE Allow remote hosts write access
IBV_ACCESS_REMOTE_READ Allow remote hosts read access
IBV_ACCESS_REMOTE_ATOMIC Allow remote hosts atomic access
IBV_ACCESS_MW_BIND Allow memory windows on this MR
Local read access is implied and automatic.
Any VPI operation that violates the access permissions of the given memory operation will fail. Note that the queue pair (QP) attributes must also have the correct permissions or the operation will fail.
If IBV_ACCESS_REMOTE_WRITE or IBV_ACCESS_REMOTE_ATOMIC is set, then IBV_ACCESS_LOCAL_WRITE must be set as well.
struct ibv_mr is defined as follows:
struct ibv_mr
{
struct ibv_context*context;
struct ibv_pd *pd;
void
*addr;
size_tlength;
uint32_thandle;
uint32_tlkey;
uint32_trkey;
};
Template: int ibv_dereg_mr(struct ibv_mr *mr)
Input Parameters: mr struct ibv_mr from ibv_reg_mr
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_dereg_mr frees a memory region (MR). The operation will fail if any memory windows (MW) are still bound to the MR.
Template: struct ibv_qp {}ibv_create_qp{*}(struct ibv_pd *pd, struct ibv_qp_init_attr *qp_init_attr)
Input Parameters: pd struct ibv_pd from ibv_alloc_pd qp_init_attrinitial attributes of queue pair
Output Parameters: qp_init_attractual values are filled in
Return Value: pointer to created queue pair (QP) or NULL on failure.
Description: ibv_create_qp creates a QP. When a QP is created, it is put into the RESET state.
struct qp_init_attr is defined as follows:
struct ibv_qp_init_attr
{
void
*qp_context;
struct ibv_cq *send_cq;
struct ibv_cq *recv_cq;
struct ibv_srq *srq;
struct ibv_qp_cap cap;
enum
ibv_qp_type qp_type;
int
sq_sig_all;
struct ibv_xrc_domain *xrc_domain;
};
qp_context (optional) user defined value associated with QP.
send_cq send CQ. This must be created by the user prior to calling ibv_create_qp.
recv_cq receive CQ. This must be created by the user prior to calling ibv_create_qp. It may be the same as send_cq.
srq (optional) shared receive queue. Only used for
SRQ QP’s.
cap defined below.
qp_type must be one of the following:
IBV_QPT_RC = 2
,
IBV_QPT_UC,
IBV_QPT_UD,
IBV_QPT_XRC,
IBV_QPT_RAW_PACKET = 8
,
IBV_QPT_RAW_ETH = 8
sq_sig_all If this
value is set to 1
, all send requests (WR) will generate completion queue events (CQE). If this
value is set to 0
, only WRs that are flagged will generate CQE’s (see ibv_post_send).
xrc_domain (Optional) Only used for
XRC operations.
struct ibv_qp_cap is defined as follows:
struct ibv_qp_cap
{
uint32_t max_send_wr;
uint32_t max_recv_wr;
uint32_t max_send_sge;
uint32_t max_recv_sge;
uint32_t max_inline_data;
};
max_send_wr Maximum number of outstanding send requests in the send queue.
max_recv_wr Maximum number of outstanding receive requests (buffers) in the receive queue.
max_send_sge Maximum number of scatter/gather elements (SGE) in a WR on the send queue.
max_recv_sge Maximum number of SGEs in a WR on the receive queue.
max_inline_data Maximum size in bytes of inline data on the send queue.
Template: int ibv_destroy_qp(struct ibv_qp *qp)
Input Parameters: qp struct ibv_qp from ibv_create_qp
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_destroy_qp frees a queue pair (QP).
Template: struct ibv_srq *ibv_create_srq(struct ibv_pd *pd, struct ibv_srq_init_attr *srq_init_attr)
Input Parameters:
pd The protection domain associated with the shared receive queue (SRQ)
srq_init_attr A list of initial attributes required to create the SRQ
Output Parameters: ibv_srq attrActual values of the struct are set
Return Value: A pointer to the created SRQ or NULL on failure
Description: ibv_create_srq creates a shared receive queue (SRQ). srq_attr->max_wr and srq_attr->max_sge are read to determine the requested size of the SRQ, and set to the actual values allocated on return. If ibv_create_srq succeeds, then max_wr and max_sge will be at least as large as the requested values.
struct ibv_srq is defined as follows:
struct ibv_srq {
struct ibv_context *context; struct ibv_context from ibv_open_device
void
*srq_context;
struct ibv_pd *pd; Protection domain
uint32_t handle;
pthread_mutex_t mutex;
pthread_cond_t cond;
uint32_t events_completed;
}
struct ibv_srq_init_attr is defined as follows:
struct ibv_srq_init_attr
{
void
*srq_context;
struct ibv_srq_attr attr;
};
srq_context struct ibv_context from ibv_open_device
attr An ibv_srq_attr struct defined as follows:
struct ibv_srq_attr is defined as follows:
struct ibv_srq_attr
{
uint32_t max_wr;
uint32_t max_sge;
uint32_t srq_limit;
};
max_wr Requested maximum number of outstanding WRs in the SRQ
max_sge Requested number of scatter elements per WR
srq_limit; The limit value of the SRQ (irrelevant for
ibv_create_srq)
Template: int ibv_modify_srq (struct ibv_srq *srq, struct ibv_srq_attr *srq_attr, int srq_attr_mask)
Input Parameters:
srq The SRQ to modify
srq_attr Specifies the SRQ to modify (input)/the current values of the selected SRQ attributes are returned (output)
srq_attr_mask A bit-mask used to specify which SRQ attributes are being modified
Output Parameters: srq_attr The struct ibv_srq_attr is returned with the updated values
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_modify_srq modifies the attributes of the SRQ srq using the attribute values in srq_attr based on the mask srq_attr_mask. srq_attr is an ibv_srq_attr struct as defined above under the verb ibv_create_srq. The argument srq_attr_mask specifies the SRQ attributes to be modified. It is either 0 or the bitwise OR of one or more of the flags:
IBV_SRQ_MAX_WR Resize the SRQ
IBV_SRQ_LIMIT Set the SRQ limit
If any of the attributes to be modified is invalid, none of the attributes will be modified. Also, not all devices support resizing SRQs. To check if a device supports resizing, check if the IBV_DE- VICE_SRQ_RESIZE bit is set in the device capabilities flags.
Modifying the SRQ limit arms the SRQ to produce an IBV_EVENT_SRQ_LIMIT_REACHED 'low watermark' async event once the number of WRs in the SRQ drops below the SRQ limit.
Template: int ibv_destroy_srq(struct ibv_srq *srq)
Input Parameters:
srq The SRQ to destroy
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_destroy_srq destroys the specified SRQ. It will fail if any queue pair is still associated with this SRQ.
Template: struct ibv_xrc_domain *ibv_open_xrc_domain(struct ibv_context *context, int fd, int oflag)
Input Parameters:
context struct ibv_context from ibv_open_device
fd The file descriptor to be associated with the XRC domain
oflag The desired file creation attributes
Output Parameters: A file descriptor associated with the opened XRC domain
Return Value: A reference to an opened XRC domain or NULL
Description: ibv_open_xrc_domain opens an eXtended Reliable Connection (XRC) domain for the RDMA device context. The desired file creation attributes oflag can either be 0 or the bitwise OR of O_CREAT and O_EXCL. If a domain belonging to the device named by the context is already associated with the inode, then the O_CREAT flag has no effect. If both O_CREAT and O_XCL are set, open will fail if a domain associated with the inode already exists. Otherwise a new XRC domain will be created and associated with the inode specified by fd.
Please note that the check for the existence of the domain and creation of the domain if it does not exist is atomic with respect to other processes executing open with fd naming the same inode.
If fd equals -1, then no inode is associated with the domain, and the only valid value for oflag is O_CREAT.
Since each ibv_open_xrc_domain call increments the xrc_domain object's reference count, each such call must have a corresponding ibv_close_xrc_domain call to decrement the xrc_domain object's reference count.
Template:
struct ibv_srq {}ibv_create_xrc_srq{*}(struct ibv_pd *pd,
struct ibv_xrc_domain *xrc_domain,
struct ibv_cq *xrc_cq,
struct ibv_srq_init_attr *srq_init_attr)
Input Parameters:
pd The protection domain associated with the shared receive queue
xrc_domain The XRC domain
xrc_cq The CQ which will hold the XRC completion
srq_init_attr A list of initial attributes required to create the SRQ (described above)
Output Parameters: ibv_srq_attrActual values of the struct are set
Return Value: A pointer to the created SRQ or NULL on failure
Description: ibv_create_xrc_srq creates an XRC shared receive queue (SRQ) associated with the protection domain pd, the XRC domain domain_xrc and the CQ which will hold the completion xrc_cq
struct ibv_xrc_domain is defined as follows:
struct ibv_xrc_domain {
struct ibv_context *context; struct ibv_context from ibv_open_device
uint64_t handle;
}
Template: int ibv_close_xrc_domain(struct ibv_xrc_domain *d)
Input Parameters:
d A pointer to the XRC domain the user wishes to close
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_close_xrc_domain closes the XRC domain, d. If this happens to be the last reference, then the XRC domain will be destroyed. This function decrements a reference count and may fail if any QP or SRQ are still associated with the XRC domain being closed.
Template: int ibv_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, uint32_t *xrc_rcv_qpn)
Input Parameters:
init_attr The structure to be populated with QP information
xrc_rcv_qpn The QP number associated with the receive QP to be created
Output Parameters:
init_attr Populated with the XRC domain information the QP will be associated with
xrc_rcv_qpn The QP number associated with the receive QP being created
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_create_xrc_rcv_qp creates an XRC queue pair (QP) to serve as a receive side only QP and returns the QP number through xrc_rcv_qpn. This number must be passed to the remote (sender) node. The remote node will use xrc_rcv_qpn in ibv_post_send when it sends messages to an XRC SRQ on this host in the same xrc domain as the XRC receive QP.
The QP with number xrc_rcv_qpn is created in kernel space and persists until the last process registered for the QP called ibv_unreg_xrc_rcv_qp, at which point the QP is destroyed. The pro- cess which creates this QP is automatically registered for it and should also call ibv_unreg_x- rc_rcv_qp at some point to unregister.
Any process which wishes to receive on an XRC SRQ via this QP must call ibv_reg_xrc_rcv_qp for this QP to ensure that the QP will not be destroyed while they are still using it.
Please note that because the QP xrc_rcv_qpn is a receive only QP, the send queue in the init_attr struct is ignored.
Template: int ibv_modify_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num, struct ibv_qp_attr *attr, int attr_mask)
Input Parameters:
xrc_domain The XRC domain associated with this QP xrc_qp_num The queue pair number to identify this QP
attr The attributes to use to modify the XRC receive QP attr_mask The mask to use for modifying the QP attributes
Output Parameters: None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_modify_xrc_rcv_qp modifies the attributes of an XRC receive QP with the number xrc_qp_num which is associated with the attributes in the struct attr according to the mask attr_mask. It then moves the QP through the following transitions: Reset->Init->RTR
At least the following masks must be set (the user may add optional attributes as needed)
Next State | Next State Required attributes |
Init | IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS |
RTR | IBV_QP_STATE, IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN, IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER |
Please note that if any attribute to modify is invalid or if the mask as invalid values, then none of the attributes will be modified, including the QP state.
Template: int ibv_reg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num)
Input Parameters:
xrc_domain The XRC domain associated with the receive QP
xrc_qp_num The number associated with the created QP to which the user process is to be registered
Output Parameters: None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_reg_xrc_rcv_qp registers a user process with the XRC receive QP whose number is xrc_qp_num associated with the XRC domain xrc_domain.
This function may fail if the number xrc_qp_num is not the number of a valid XRC receive QP (for example if the QP is not allocated or it is the number of a non-XRC QP), or the XRC receive QP was created with an XRC domain other than xrc_domain.
Template: int ibv_unreg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num)
Input Parameters:
xrc_domain The XRC domain associated with the XRC receive QP from which the user wishes to unregister
xrc_qp_num TheQPnumberfromwhich the user process istobe unregistered
Output Parameters: None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_unreg_xrc_rcv_qp unregisters a user process from the XRC receive QP number xrc_qp_- num which is associated with the XRC domain xrc_domain. When the number of user processes registered with this XRC receive QP drops to zero, the QP is destroyed.
Template: struct ibv_ah {}ibv_create_ah{*}(struct ibv_pd *pd, struct ibv_ah_attr *attr)
Input Parameters:
pd struct ibv_pd from ibv_alloc_pd
attr attributes of address
Output Parameters: none
Return Value: pointer to created address handle (AH) or NULL on failure.
Description: ibv_create_ah creates an AH. An AH contains all of the necessary data to reach a remote destination. In connected transport modes (RC, UC) the AH is associated with a queue pair (QP). In the datagram transport modes (UD), the AH is associated with a work request (WR).
struct ibv_ah_attr is defined as follows:
struct ibv_ah_attr
{
struct ibv_global_route grh;
uint16_t dlid;
uint8_t sl;
uint8_t src_path_bits;
uint8_t static_rate;
uint8_t is_global;
uint8_t port_num;
};
grh defined below
dlid destination lid
sl service level
src_path_bits source path bits
static_rate static
rate
is_global this
is a global address, use grh.
port_num physical port number to use to reach this
destination
struct ibv_global_route is defined as follows:
struct ibv_global_route
{
union ibv_gid dgid;
uint32_t flow_label;
uint8_t sgid_index;
uint8_t hop_limit;
uint8_t traffic_class;
};
dgid destination GID (see ibv_query_gid for
definition)
flow_label flow label
sgid_index index of source GID (see ibv_query_gid)
hop_limit hop limit
traffic_class traffic class
Template: int ibv_destroy_ah(struct ibv_ah *ah)
Input Parameters:
ah struct ibv_ah from ibv_create_ah
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_destroy_ah frees an address handle (AH). Once an AH is destroyed, it can't be used any- more in UD QPs