Queue Pair Bringup (ibv_modify_qp)
Queue pairs (QP) must be transitioned through an incremental sequence of states prior to being able to be used for communication.
QP States:
RESET Newly created, queues empty.
INIT Basic information set. Ready for posting to receive queue.
RTR Ready to Receive. Remote address info set for connected QPs, QP may now receive packets.
RTS Ready to Send. Timeout and retry parameters set, QP may now send packets.
Template: int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, enum ibv_qp_attr_mask attr_mask)
Input Parameters:
qp struct ibv_qp from ibv_create_qp
attr QP attributes
attr_mask bit mask that defines which attributes within attr have been set for this call
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_modify_qp this verb changes QP attributes and one of those attributes may be the QP state. Its name is a bit of a misnomer, since you cannot use this command to modify qp attributes at will. There is a very strict set of attributes that may be modified during each transition, and transitions must occur in the proper order. The following subsections describe each transition in more detail.
struct ibv_qp_attr is defined as follows:
struct ibv_qp_attr
{
enum ibv_qp_state qp_state;
enum ibv_qp_state cur_qp_state;
enum ibv_mtu path_mtu;
enum ibv_mig_state path_mig_state;
uint32_t qkey;
uint32_t rq_psn;
uint32_t sq_psn;
uint32_t dest_qp_num;
int qp_access_flags;
struct ibv_qp_cap cap;
struct ibv_ah_attr ah_attr;
struct ibv_ah_attr alt_ah_attr;
uint16_t pkey_index;
uint16_t alt_pkey_index;
uint8_t en_sqd_async_notify;
uint8_t sq_draining;
uint8_t max_rd_atomic;
uint8_t max_dest_rd_atomic;
uint8_t min_rnr_timer;
uint8_t port_num;
uint8_t timeout;
uint8_t retry_cnt;
uint8_t rnr_retry;
uint8_t alt_port_num;
uint8_t alt_timeout;
};
The following values select one of the above attributes and should be OR’d into the attr_mask field:
IBV_QP_STATE
IBV_QP_CUR_STATE
IBV_QP_EN_SQD_ASYNC_NOTIFY
IBV_QP_ACCESS_FLAGS
IBV_QP_PKEY_INDEX
IBV_QP_PORT
IBV_QP_QKEY
IBV_QP_AV
IBV_QP_PATH_MTU
IBV_QP_TIMEOUT
IBV_QP_RETRY_CNT
IBV_QP_RNR_RETRY
IBV_QP_RQ_PSN
IBV_QP_MAX_QP_RD_ATOMIC
IBV_QP_ALT_PATH
IBV_QP_MIN_RNR_TIMER
IBV_QP_SQ_PSN
IBV_QP_MAX_DEST_RD_ATOMIC
IBV_QP_PATH_MIG_STATE
IBV_QP_CAP
IBV_QP_DEST_QPN
When a queue pair (QP) is newly created, it is in the RESET state. The first state transition that needs to happen is to bring the QP in the INIT state.
Required Attributes:
*** All QPs ***
qp_state / IBV_QP_STATE IBV_QPS_INIT
pkey_index / IBV_QP_PKEY_INDEX pkey index, normally 0
port_num / IBV_QP_PORT physical port number (1...n)
qp_access_flags /
IBV_QP_ACCESS_FLAGS access flags (see ibv_reg_mr)
*** Unconnected QPs only ***
qkey / IBV_QP_QKEY qkey (see ibv_post_send)
Optional Attributes: none
Effect of transition: Once the QP is transitioned into the INIT state, the user may begin to post receive buffers to the receive queue via the ibv_post_recv command. At least one receive buffer should be posted before the QP can be transitioned to the RTR state.
Once a queue pair (QP) has receive buffers posted to it, it is now possible to transition the QP into the ready to receive (RTR) state.
Required Attributes:
*** All QPs ***
qp_state / IBV_QP_STATE IBV_QPS_RTR
path_mtu / IBV_QP_PATH_MTU IB_MTU_256
IB_MTU_512 (recommended value)
IB_MTU_1024
IB_MTU_2048
IB_MTU_4096
*** Connected QPs only ***
ah_attr / IBV_QP_AV an address handle (AH) needs to be created and filled in as appropriate. Minimally, ah_attr.dlid needs to be filled in.
dest_qp_num / IBV_QP_DEST_QPN QP number of remote QP.
rq_psn / IBV_QP_RQ_PSN starting receive packet sequence number (should match remote QP’s sq_psn)
max_dest_rd_atomic /
IBV_MAX_DEST_RD_ATOMIC maximum number of resources for incoming RDMA requests
min_rnr_timer /
IBV_QP_MIN_RNR_TIMER minimum RNR NAK timer (recommended value: 12)
Optional Attributes:
*** All QPs ***
qp_access_flags /
IBV_QP_ACCESS_FLAGS access flags (see ibv_reg_mr)
pkey_index / IBV_QP_PKEY_INDEX pkey index, normally 0
*** Connected QPs only ***
alt_ah_attr / IBV_QP_ALT_PATH AH with alternate path info filled in
*** Unconnected QPs only ***
qkey / IBV_QP_QKEY qkey (see ibv_post_send)
Effect of transition: Once the QP is transitioned into the RTR state, the QP begins receive processing.
Once a queue pair (QP) has reached ready to receive (RTR) state, it may then be transitioned to the ready to send (RTS) state.
Required Attributes:
*** All QPs ***
qp_state / IBV_QP_STATE IBV_QPS_RTS
*** Connected QPs only ***
timeout / IBV_QP_TIMEOUT local ack timeout (recommended value: 14)
retry_cnt / IBV_QP_RETRY_CNT retry count (recommended value: 7)
rnr_retry / IBV_QP_RNR_RETRY RNR retry count (recommended value: 7)
sq_psn / IBV_SQ_PSN send queue starting packet sequence number (should match remote QP’s rq_psn)
max_rd_atomic
/ IBV_QP_MAX_QP_RD_ATOMIC number of outstanding RDMA reads and atomic operations allowed.
Optional Attributes:
*** All QPs ***
qp_access_flags /
IBV_QP_ACCESS_FLAGS access flags (see ibv_reg_mr)
*** Connected QPs only ***
alt_ah_attr / IBV_QP_ALT_PATH AH with alternate path info filled in
min_rnr_timer /
IBV_QP_MIN_RNR_TIMER minimum RNR NAK timer
*** Unconnected QPs only ***
qkey / IBV_QP_QKEY qkey (see ibv_post_send)
Effect of transition: Once the QP is transitioned into the RTS state, the QP begins send processing and is fully operational. The user may now post send requests with the ibv_post_send command.