Template: int rdma_post_recvv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge)
Input Parameters:
id A reference to the communication identifier where the message buffer(s) will be posted
context A user-defined context associated with the request
sgl A scatter-gather list of memory buffers posted as a single request
nsge The number of scatter-gather entries in the sgl array
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_recvv posts a single work request to the receive queue of the queue pair associated with the rdma_cm_id, id. The posted buffers will be queued to receive an incoming message sent by the remote peer.
Please note that this routine supports multiple scatter-gather entries. The user is responsible for ensuring that the receive is posted, and the total buffer space is large enough to contain all sent data before the peer posts the corresponding send message. The message buffers must have been registered before being posted, and the buffers must remain registered until the receive completes.
Messages may be posted to an rdma_cm_id only after a queue pair has been associated with it. A queue pair is bound to an rdma_cm_id after calling rdma_create_ep or rdma_create_qp, if the rdma_cm_id is allocated using rdma_create_id.
The user-defined context associated with the receive request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_post_sendv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge, int flags)
Input Parameters:
id A reference to the communication identifier where the message buffer will be posted
context A user-defined context associated with the request
sgl A scatter-gather list of memory buffers posted as a single request
nsge The number of scatter-gather entries in the sgl array
flags Optional flags used to control the send operation
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_sendv posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the posted buffers will be sent to the remote peer of the connection.
The user is responsible for ensuring that the remote peer has queued a receive request before issuing the send operations. Also, unless the send request is using inline data, the message buffers must already have been registered before being posted. The buffers must remain registered until the send completes.
This routine supports multiple scatter-gather entries.
Send operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
The user-defined context associated with the send request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_post_readv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
Input Parameters:
id A reference to the communication identifier where the request will be posted
context A user-defined context associated with the request
sgl A scatter-gather list of the destination buffers of the read
nsge The number of scatter-gather entries in the sgl array
flags Optional flags used to control the read operation
remote_addr The address of the remote registered memory to read from rkey The registered memory key associated with the remote address
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_readv posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the remote memory region at remote_addr will be read into the local data buffers given in the sgl array.
The user must ensure that both the remote and local data buffers have been registered before the read is issued. The buffers must remain registered until the read completes.
Read operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
The user-defined context associated with the read request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_post_writev(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl, int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
Input Parameters:
id A reference to the communication identifier where the request will be posted
context A user-defined context associated with the request
sgl A scatter-gather list of the source buffers of the write
nsge The number of scatter-gather entries in the sgl array
flags Optional flags used to control the write operation
remote_addr The address of the remote registered memory to write into rkey The registered memory key associated with the remote address
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_writev posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the local data buffers in the sgl array will be written to the remote memory region at remote_addr.
Unless inline data is specified, the local data buffers must have been registered before the write is issued, and the buffers must remain registered until the write completes. The remote buffers must always be registered.
Write operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
The user-defined context associated with the write request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_post_recv(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct ibv_mr *mr)
Input Parameters:
id A reference to the communication identifier where the message buffer will be posted
context A user-defined context associated with the request addr The address of the memory buffer to post
length The length of the memory buffer
mr A registered memory region associated with the posted buffer
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_recv posts a work request to the receive queue of the queue pair associated with the rdma_cm_id, id. The posted buffer will be queued to receive an incoming message sent by the remote peer.
The user is responsible for ensuring that receive buffer is posted and is large enough to contain all sent data before the peer posts the corresponding send message. The buffer must have already been registered before being posted, with the mr parameter referencing the registration. The buffer must remain registered until the receive completes.
Messages may be posted to an rdma_cm_id only after a queue pair has been associated with it. A queue pair is bound to an rdma_cm_id after calling rdma_create_ep or rdma_create_qp, if the rdma_cm_id is allocated using rdma_create_id.
The user-defined context associated with the receive request will be returned to the user through the work completion request identifier (wr_id) field.
Please note that this is a simple receive call. There are no scatter-gather lists involved here.
Template: int rdma_post_send(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct ibv_mr *mr, int flags)
Input Parameters:
id A reference to the communication identifier where the message buffer will be posted
context A user-defined context associated with the request addr The address of the memory buffer to post
length The length of the memory buffer
mr Optional registered memory region associated with the posted buffer
flags Optional flags used to control the send operation
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_send posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the posted buffer will be sent to the remote peer of the connection.
The user is responsible for ensuring that the remote peer has queued a receive request before issuing the send operations. Also, unless the send request is using inline data, the message buffer must already have been registered before being posted, with the mr parameter referencing the registration. The buffer must remain registered until the send completes.
Send operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
The user-defined context associated with the send request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_post_read(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct ibv_mr *mr, int flags, uint64_t remote_addr, uint32_t rkey)
Input Parameters:
id A reference to the communication identifier where the request will be posted
context A user-defined context associated with the request
addr The address of the local destination of the read request
length The length of the read operation
mr Registered memory region associated with the local buffer
flags Optional flags used to control the read operation
remote_addr The address of the remote registered memory to read from rkey The registered memory key associated with the remote address
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_read posts a work request to the send queue of the queue pair associated with the rdma_cm_id. The contents of the remote memory region will be read into the local data buffer.
For a list of supported flags, see ibv_post_send. The user must ensure that both the remote and local data buffers must have been registered before the read is issued, and the buffers must remain registered until the read completes.
Read operations may not be posted to an rdma_cm_id or the corresponding queue pair until it has been connected.
The user-defined context associated with the read request will be returned to the user through the work completion wr_id, work request identifier, field.
Template: int rdma_post_write(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct ibv_mr *mr, int flags, uint64_t remote_addr, uint32_t rkey)
Input Parameters:
id A reference to the communication identifier where the request will be posted
context A user-defined context associated with the request addr The local address of the source of the write request
length The length of the write operation
mr Optional registered memory region associated with the local buffer
flags Optional flags used to control the write operation
remote_addr The address of the remote registered memory to write into rkey The registered memory key associated with the remote address
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_write posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the local data buffer will be written into the remote memory region.
Unless inline data is specified, the local data buffer must have been registered before the write is issued, and the buffer must remain registered until the write completes. The remote buffer must always be registered.
Write operations may not be posted to an rdma_cm_id or the corresponding queue pair until a connection has been established.
The user-defined context associated with the write request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_post_ud_send(struct rdma_cm_id *id, void *context, void *addr, size_t length, struct ibv_mr *mr, int flags, struct ibv_ah *ah, uint32_t remote_qpn)
Input Parameters:
id A reference to the communication identifier where the request will be posted
context A user-defined context associated with the request addr The address of the memory buffer to post
length The length of the memory buffer
mr Optional registered memory region associated with the posted buffer
flags Optional flags used to control the send operation
ah An address handle describing the address of the remote node
remote_qpn The destination node's queue pair number
Output Parameters:
None
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_post_ud_send posts a work request to the send queue of the queue pair associated with the rdma_cm_id, id. The contents of the posted buffer will be sent to the specified destination queue pair, remote_qpn.
The user is responsible for ensuring that the destination queue pair has queued a receive request before issuing the send operations. Unless the send request is using inline data, the message buffer must have been registered before being posted, with the mr parameter referencing the registration. The buffer must remain registered until the send completes.
The user-defined context associated with the send request will be returned to the user through the work completion work request identifier (wr_id) field.
Template: int rdma_get_send_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
Input Parameters:
id A reference to the communication identifier to check for completions
wc A reference to a work completion structure to fill in
Output Parameters:
wc A reference to a work completion structure. The structure will contain information about the completed request when routine returns
Return Value: A non-negative value (0 or 1) equal to the number of completions found on success, or -1 on failure. If the call fails, errno will be set to indicate the reason for the failure.
Description: rdma_get_send_comp retrieves a completed work request for a send, RDMA read or RDMA write operation. Information about the completed request is returned through the ibv_wc, wc parameter, with the wr_id set to the context of the request. Please see ibv_poll_cq for details on the work completion structure, ibv_wc.
Please note that this call polls the send completion queue associated with the rdma_cm_id, id. If a completion is not found, the call blocks until a request completes. This means, therefore, that the call should only be used on rdma_cm_ids which do not share CQs with other rdma_cm_ids, and maintain separate CQs for sends and receive completions.
Template: int rdma_get_recv_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
Input Parameters:
id A reference to the communication identifier to check for completions
wc A reference to a work completion structure to fill in
Output Parameters:
wc A reference to a work completion structure. The structure will contain information about the completed request when routine returns
Return Value: A non-negative value equal to the number of completions found on success, or errno on failure
Description: rdma_get_recv_comp retrieves a completed work request a receive operation. Information about the completed request is returned through the ibv_wc, wc parameter, with the wr_id set to the context of the request. Please see ibv_poll_cq for details on the work completion structure, ibv_wc.
Please note that this call polls the receive completion queue associated with the rdma_cm_id, id. If a completion is not found, the call blocks until a request completes. This means, therefore, that the call should only be used on rdma_cm_ids which do not share CQs with other rdma_cm_ids, and maintain separate CQs for sends and receive completions.