The following commands are used once a device has been opened. These commands allow you to get more specific information about a device or one of its ports, create completion queues (CQ), completion channels (CC), and protection domains (PD) which can be used for further opera- tions.
Template: int ibv_query_device(struct ibv_context *context, struct ibv_device_attr *device_attr)
Input Parameters: context struct ibv_context from ibv_open_device
Output Parameters: device_attr struct ibv_device_attr containing device attributes
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_query_device retrieves the various attributes associated with a device. The user should malloc a struct ibv_device_attr, pass it to the command, and it will be filled in upon successful return. The user is responsible to free this struct.
struct ibv_device_attr is defined as follows:
struct ibv_device_attr
{
char fw_ver[64];
uint64_t node_guid;
uint64_t sys_image_guid;
uint64_t max_mr_size;
uint64_t page_size_cap;
uint32_t vendor_id;
uint32_t vendor_part_id;
uint32_t hw_ver;
int max_qp;
int max_qp_wr;
int device_cap_flags;
int max_sge;
int max_sge_rd;
int max_cq;
int max_cqe;
int max_mr;
int max_pd;
int max_qp_rd_atom;
int max_ee_rd_atom;
int max_res_rd_atom;
int max_qp_init_rd_atom;
int max_ee_init_rd_atom;
enum ibv_atomic_cap atomic_cap;
int max_ee;
int max_rdd;
int max_mw;
int max_raw_ipv6_qp;
int max_raw_ethy_qp;
int max_mcast_grp;
int max_mcast_qp_attach;
int max_total_mcast_qp_attach;
int max_ah;
int max_fmr;
int max_map_per_fmr;
int max_srq;
int max_srq_wr;
int max_srq_sge;
uint16_t max_pkeys;
uint8_t local_ca_ack_delay;
uint8_t phys_port_cnt;
}
fw_ver Firmware version
node_guid Node global unique identifier (GUID)
sys_image_guid System image GUID
max_mr_size Largest contiguous block that can be registered
page_size_cap Supported page sizes
vendor_id Vendor ID, per IEEE
vendor_part_id Vendor supplied part ID
hw_ver Hardware version
max_qp Maximum number of Queue Pairs (QP)
max_qp_wr Maximum outstanding work requests (WR) on any queue
device_cap_flags IBV_DEVICE_RESIZE_MAX_WR
IBV_DEVICE_BAD_PKEY_CNTR
IBV_DEVICE_BAD_QKEY_CNTR
IBV_DEVICE_RAW_MULTI
IBV_DEVICE_AUTO_PATH_MIG
IBV_DEVICE_CHANGE_PHY_PORT
IBV_DEVICE_UD_AV_PORT_ENFORCE
IBV_DEVICE_CURR_QP_STATE_MOD
IBV_DEVICE_SHUTDOWN_PORT
IBV_DEVICE_INIT_TYPE
IBV_DEVICE_PORT_ACTIVE_EVENT
IBV_DEVICE_SYS_IMAGE_GUID
IBV_DEVICE_RC_RNR_NAK_GEN
IBV_DEVICE_SRQ_RESIZE
IBV_DEVICE_N_NOTIFY_CQ
IBV_DEVICE_XRC
max_sge Maximum scatter/gather entries (SGE) per WR for non-RD QPs
max_sge_rd Maximum SGEs per WR for RD QPs
max_cq Maximum supported completion queues (CQ)
max_cqe Maximum completion queue entries (CQE) per CQ
max_mr Maximum supported memory regions (MR)
max_pd Maximum supported protection domains (PD)
max_qp_rd_atom Maximum outstanding RDMA read and atomic operations per QP
max_ee_rd_atom Maximum outstanding RDMA read and atomic operations per End to End (EE) context (RD connections)
max_res_rd_atom Maximum resources used for incoming RDMA read and atomic operations
max_qp_init_rd_atom Maximium RDMA read and atomic operations that may be initiated per QP
max_ee_init_atom Maximum RDMA read and atomic operations that may be initiated per EE
atomic_cap IBV_ATOMIC_NONE - no atomic guarantees
IBV_ATOMIC_HCA - atomic guarantees within this device
IBV_ATOMIC_GLOB - global atomic guarantees
max_ee Maximum supported EE contexts
max_rdd Maximum supported RD domains
max_mw Maximum supported memory windows (MW)
max_raw_ipv6_qp Maximum supported raw IPv6 datagram QPs
max_raw_ethy_qp Maximum supported ethertype datagram QPs
max_mcast_grp Maximum supported multicast groups
max_mcast_qp_attach Maximum QPs per multicast group that can be attached
max_total_mcast_qp_attach
Maximum total QPs that can be attached to multicast groups
max_ah Maximum supported address handles (AH)
max_fmr Maximum supported fast memory regions (FMR)
max_map_per_fmr Maximum number of remaps per FMR before an unmap operation is required
max_srq Maximum supported shared receive queues (SRCQ)
max_srq_wr Maximum work requests (WR) per SRQ
max_srq_sge Maximum SGEs per SRQ
max_pkeys Maximum number of partitions
local_ca_ack_delay Local CA ack delay
phys_port_cnt Number of physical ports
Template: int ibv_query_port(struct ibv_context *context, uint8_t port_num, struct ibv_port_attr *port_attr)
Input Parameters: context struct ibv_context from ibv_open_device port_numphysical port number (1 is first port)
Output Parameters: port_attrstruct ibv_port_attr containing port attributes
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_query_port retrieves the various attributes associated with a port. The user should allocate a struct ibv_port_attr, pass it to the command, and it will be filled in upon successful return. The user is responsible to free this struct.
struct ibv_port_attr is defined as follows:
struct ibv_port_attr
{
enum ibv_port_state state;
enum ibv_mtu max_mtu;
enum ibv_mtu active_mtu;
int gid_tbl_len;
uint32_t port_cap_flags;
uint32_t max_msg_sz;
uint32_t bad_pkey_cntr;
uint32_t qkey_viol_cntr;
uint16_t pkey_tbl_len;
uint16_t lid;
uint16_t sm_lid;
uint8_t lmc;
uint8_t max_vl_num;
uint8_t sm_sl;
uint8_t subnet_timeout;
uint8_t init_type_reply;
uint8_t active_width;
uint8_t active_speed;
uint8_t phys_state;
};
state IBV_PORT_NOP
IBV_PORT_DOWN
IBV_PORT_INIT
IBV_PORT_ARMED
IBV_PORT_ACTIVE
IBV_PORT_ACTIVE_DEFER
max_mtu Maximum Transmission Unit (MTU) supported by port. Can be:
IBV_MTU_256
IBV_MTU_512
IBV_MTU_1024
IBV_MTU_2048
IBV_MTU_4096
active_mtu Actual MTU in use
gid_tbl_len Length of source global ID (GID) table
port_cap_flags Supported capabilities of this port. There are currently no enumerations/defines declared in verbs.h
max_msg_sz Maximum message size
bad_pkey_cntr Bad P_Key counter
qkey_viol_cntr Q_Key violation counter
pkey_tbl_len Length of partition table
lid First local identifier (LID) assigned to this port
sm_lid LID of subnet manager (SM)
lmc LID Mask control (used when multiple LIDs are assigned to port)
max_vl_num Maximum virtual lanes (VL)
sm_sl SM service level (SL)
subnet_timeout Subnet propagation delay
init_type_reply Type of initialization performed by SM
active_width Currently active link width
active_speed Currently active link speed
phys_state Physical port state
Template: int ibv_query_gid(struct ibv_context *context, uint8_t port_num, int index, union ibv_gid *gid)
Input Parameters:
context struct ibv_context from ibv_open_device
port_num physical port number (1 is first port)
index which entry in the GID table to return (0 is first)
Output Parameters:
gid union ibv_gid containing gid information
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_query_gid retrieves an entry in the port’s global identifier (GID) table. Each port is assigned at least one GID by the subnet manager (SM). The GID is a valid IPv6 address composed of the globally unique identifier (GUID) and a prefix assigned by the SM. GID[0] is unique and contains the port's GUID.
The user should allocate a union ibv_gid, pass it to the command, and it will be filled in upon successful return. The user is responsible to free this union.
union ibv_gid is defined as follows:
union ibv_gid
{
uint8_t raw[16];
struct
{
uint64_t subnet_prefix;
uint64_t interface_id;
} global;
};
Template: int ibv_query_pkey(struct ibv_context *context, uint8_t port_num, int index, uint16_t *pkey)
Input Parameters:
context struct ibv_context from ibv_open_device
port_num physical port number (1 is first port)
index which entry in the pkey table to return (0 is first)
Output Parameters:
pkey desired pkey
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_query_pkey retrieves an entry in the port's partition key (pkey) table. Each port is assigned at least one pkey by the subnet manager (SM). The pkey identifies a partition that the port belongs to. A pkey is roughly analogous to a VLAN ID in Ethernet networking.
The user passes in a pointer to a uint16 that will be filled in with the requested pkey. The user is responsible to free this uint16.
Template: struct ibv_pd {}ibv_alloc_pd{*}(struct ibv_context *context)
Input Parameters: context struct ibv_context from ibv_open_device
Output Parameters: none
Return Value: Pointer to created protection domain or NULL on failure.
Description: ibv_alloc_pd creates a protection domain (PD). PDs limit which memory regions can be accessed by which queue pairs (QP) providing a degree of protection from unauthorized access. The user must create at least one PD to use VPI verbs.
Template: int ibv_dealloc_pd(struct ibv_pd *pd)
Input Parameters: pd struct ibv_pd from ibv_alloc_pd
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_dealloc_pd frees a protection domain (PD). This command will fail if any other objects are currently associated with the indicated PD.
Template: struct ibv_cq {}ibv_create_cq{*}(struct ibv_context *context, int cqe, void *cq_context, struct ibv_- comp_channel *channel, int comp_vector)
Input Parameters:
context struct ibv_context from ibv_open_device
cqe Minimum number of entries CQ will support
cq_context (Optional)User defined value returned with completion events
channel (Optional) Completion channel
comp_vector (Optional) Completion vector
Output Parameters: none
Return Value: pointer to created CQ or NULL on failure.
Description: ibv_create_cq creates a completion queue (CQ). A completion queue holds completion queue entries (CQE). Each Queue Pair (QP) has an associated send and receive CQ. A single CQ can be shared for sending and receiving as well as be shared across multiple QPs.
The parameter cqe defines the minimum size of the queue. The actual size of the queue may be larger than the specified value.
The parameter cq_context is a user defined value. If specified during CQ creation, this value will be returned as a parameter in ibv_get_cq_event when using a completion channel (CC).
The parameter channel is used to specify a CC. A CQ is merely a queue that does not have a built in notification mechanism. When using a polling paradigm for CQ processing, a CC is unnecessary. The user simply polls the CQ at regular intervals. If, however, you wish to use a pend para-
digm, a CC is required. The CC is the mechanism that allows the user to be notified that a new CQE is on the CQ.
The parameter comp_vector is used to specify the completion vector used to signal completion events. It must be >=0 and < context->num_comp_vectors.
Template: int ibv_resize_cq(struct ibv_cq *cq, int cqe)
Input Parameters:
cq CQ to resize
cqe Minimum number of entries CQ will support
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_resize_cq resizes a completion queue (CQ). The parameter cqe must be at least the number of outstanding entries on the queue. The actual size of the queue may be larger than the specified value. The CQ may (or may not) contain com- pletions when it is being resized thus, it can be resized during work with the CQ.
Template: int ibv_destroy_cq(struct ibv_cq *cq)
Input Parameters:
cq CQ to destroy
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_destroy_cq frees a completion queue (CQ). This command will fail if there is any queue pair (QP) that still has the specified CQ associated with it.
Template: struct ibv_comp_channel {}ibv_create_comp_channel{*}(struct ibv_context *context)
Input Parameters: context struct ibv_context from ibv_open_device
Output Parameters: none
Return Value: pointer to created CC or NULL on failure.
Description: ibv_create_comp_channel creates a completion channel. A completion channel is a mechanism for the user to receive notifications when new completion queue event (CQE) has been placed on a completion queue (CQ).
Template: int ibv_destroy_comp_channel(struct ibv_comp_channel *channel)
Input Parameters: channel struct ibv_comp_channel from ibv_create_comp_channel
Output Parameters: none
Return Value: 0 on success, -1 on error. If the call fails, errno will be set to indicate the reason for the failure.
Description: ibv_destroy_comp_channel frees a completion channel. This command will fail if there are any completion queues (CQ) still associated with this completion channel.