5. Supporting Structures#

5.1. cuObjScatterGatherEntry#

typedef struct cuObjScatterGatherEntry {
    void* addr;
    size_t size;
} cuObjScatterGatherEntry_t;

Fields:

  • addr: Host memory address.

  • size: Segment size in bytes.

5.2. cuObjScatterGatherEntryWithOffset#

typedef struct cuObjScatterGatherEntryWithOffset {
    loff_t offset;
    size_t size;
} cuObjScatterGatherEntryWithOffset_t;

Fields:

  • offset: Offset from the base registered buffer.

  • size: Segment size in bytes.

5.3. cuObjAsyncEvent_t#

typedef struct cuObjAsyncEvent_s {
    void* async_handle;
    ssize_t status;
} cuObjAsyncEvent_t;

Fields:

  • async_handle: User handle provided during asynchronous submission.

  • status: Completion status. The field is declared as ssize_t, but poll() stores an enum ibv_wc_status value in it. IBV_WC_SUCCESS (0) means success. For nonzero values, interpret the field as ibv_wc_status_str((enum ibv_wc_status)event.status).

5.4. cuObjRDMATunableParam#

struct cuObjRDMATunableParam {
    int num_dcis;
    unsigned cq_depth;
    unsigned long dc_key;
    int ibv_poll_max_comp_event;
    int service_level;
    uint8_t timeout;
    unsigned hop_limit;
    int pkey_index;
    int max_wr;
    int max_sge;
    uint32_t delay_interval;
    cuObjDelayMode_t delay_mode;
    bool qp_reset_on_failure;
    uint8_t retry_cnt;
    unsigned traffic_class;
    int max_rd_atomic;
};

5.5. cuObjRDMATunable#

cuObjRDMATunable stores cuObjRDMATunableParam values and provides getters and setters.

Parameter

Setter

Getter

Default

Valid Values / Current Behavior

num_dcis

setNumDcis(int)

getNumDcis()

128

Number of channels/DCIs. Must be usable by the RDMA device. Values greater than the device QP limit cause session start failure.

cq_depth

setCqDepth(unsigned)

getCqDepth()

640

Completion queue depth.

dc_key

setDcKey(unsigned long)

getDcKey()

0xffeeddcc

DC key value.

ibv_poll_max_comp_event

setIbvPollMaxCompEv(int)

getIbvPollMaxCompEv()

0

Not used currently.

service_level

setServiceLevel(int)

getServiceLevel()

0

Service level value.

timeout

setTimeout(uint8_t)

getTimeout()

16

Timeout value used by the RDMA connection.

hop_limit

setHopLimit(unsigned)

getHopLimit()

4

Hop limit value.

pkey_index

setPkeyIndex(int)

getPkeyIndex()

0

Partition key index.

max_wr

setMaxWr(int)

getMaxWr()

0

Not used currently.

max_sge

setMaxSge(int)

getMaxSge()

10

Maximum scatter-gather entries per operation. handleGetObject() and handlePutObject() fail with -EIO if the registered buffer entry count exceeds this value.

delay_interval

setDelayInterval(uint32_t)

getDelayInterval()

5000

Polling delay interval in nanoseconds.

delay_mode

setDelayMode(cuObjDelayMode_t)

getDelayMode()

CUOBJ_DELAY_BATCH

One of the cuObjDelayMode_t values.

qp_reset_on_failure

setQPResetOnFailure(bool)

getQPResetOnFailure()

true

Enables queue-pair reset after completion failure.

retry_cnt

setRetryCount(uint8_t)

getRetryCount()

7

Retry count value. The header documents this as a 3-bit value, 0 through 7.

traffic_class

setTrafficClass(unsigned)

getTrafficClass()

96

Values below 256 are applied. Values greater than or equal to 256 are logged as invalid.

max_rd_atomic

setMaxRdAtomic(uint8_t)

getMaxRdAtomic()

0

0 means auto-detect from the RDMA device. Setting MaxRdAtomic to 16 is required for optimal performance on PUT operations using RDMA_READ. Positive values are capped to the device maximum.

The setters do not return status values and do not perform full range validation. Configure values that are valid for the RDMA device and workload.