Resource Domain Experimental Verbs

Linux Kernel Upstream Release Notes v5.17

Resource domain is a verb object that may be associated with QP and CQ objects upon creation to enhance data-path performance.

Thread model - defines the threading module for the objects (QP/CQ) associated with this resource domain.

IBV_EXP_THREAD_SAFE

Access to the associated objects are thread safe. Such objects may be accessed by any thread without concern for thread safety issues.

IBV_EXP_THREAD_UNSAFE

Access to the associated objects are not thread safe. Access to such objects must be coordinated by the calling threads.

IBV_EXP_THREAD_SINGLE

Access to the associated objects are from only one thread over the lifetime of the object. In addition, different objects associated with the same resource domain must be called by the same thread.


Message model - defines whether associated objects connection is optimized for low latency or high bandwidth.

IBV_EXP_MSG_FORCE_LOW_LATENCY

Forces the provider to optimize for low latency

IBV_EXP_MSG_LOW_LATENCY

Suggests the provider to optimize for low latency.

IBV_EXP_MSG_HIGH_BW

Suggests the provider to optimize for high bandwidth.

IBV_EXP_MSG_DEFAULT

Uses the provider's default message model.

Procedure_Heading_Icon.PNG

To create a resource domain, use:

Copy
Copied!
            

static inline struct ibv_exp_res_domain *ibv_exp_create_res_domain( struct ibv_context *context, struct ibv_exp_res_domain_init_attr *attr)


Procedure_Heading_Icon.PNG
To destroy a resource domain, use:

Copy
Copied!
            

static inline int ibv_exp_destroy_res_domain( struct ibv_context *context, struct ibv_exp_res_domain *res_dom, struct ibv_exp_destroy_res_domain_attr *attr)

Use the res_domain field and relevant comp_mask in the ibv_exp_cq_init_attr and ibv_exp_qp_init_attr structs to pass resource domain to the CQ and QP upon their creation.

Example - creating a CQ and QP that may be called from one thread only:

Copy
Copied!
            

struct ibv_exp_res_domain_init_attr res_domain_attr; struct ibv_exp_res_domain *res_domain; struct ibv_exp_cq_init_attr cq_attr; struct ibv_exp_qp_init_attr qp_attr;   … res_domain_attr.comp_mask = IBV_EXP_RES_DOMAIN_THREAD_MODEL | IBV_EXP_RES_DOMAIN_MSG_MODEL; res_domain_attr.thread_model = IBV_EXP_THREAD_SINGLE; res_domain_attr.msg_model = IBV_EXP_MSG_HIGH_BW;   res_domain = ibv_exp_create_res_domain(ctx->context, &res_domain_attr); if (!_domain) { fprintf(stderr, "Can't create resource domain\n"); exit(1); } … cq_attr.res_domain = res_domain; cq_attr.comp_mask |= IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN; cq = ibv_exp_create_cq(context, num_ entries, NULL, NULL, 0, &cq_attr); … qp_attr.res_domain = res_domain; qp_attr.comp_mask |= IBV_EXP_QP_INIT_ATTR_RES_DOMAIN; qp = ibv_exp_create_qp(context, &qp_attr);

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.