4. cuObjServer Class API#

4.1. Class Declaration#

class cuObjServer : public RDMAConnection {
public:
    cuObjServer(const char* ip, unsigned short port, unsigned proto);
    cuObjServer(const char* ip, unsigned short port, unsigned proto,
                cuObjRDMATunable params);
    cuObjServer() = default;
    ~cuObjServer();

    void* allocHostBuffer(size_t size);
    struct rdma_buffer* registerBuffer(void* ptr, size_t size);
    struct rdma_buffer* registerBuffer(
        std::vector<cuObjScatterGatherEntry_t> sglist);
    struct rdma_buffer* getRDMABufferFromSgList(
        struct rdma_buffer* rdma_buffer,
        std::vector<cuObjScatterGatherEntryWithOffset_t> sglist);
    void putRDMABufferFromSgList(struct rdma_buffer* rdma_buffer);
    void deRegisterBuffer(struct rdma_buffer* rdma_buff);

    ssize_t handleGetObject(const std::string& key,
                            struct rdma_buffer* local_rdma_buff,
                            uint64_t remote_buf_start, size_t size,
                            const std::string& rdma_descr,
                            uint32_t poll_delay,
                            uint16_t channel = 0,
                            uint64_t local_offset = 0,
                            ibv_wc_status* status = nullptr,
                            void* async_handle = nullptr);
    ssize_t handleGetObject(const std::string& key,
                            struct rdma_buffer* local_rdma_buff,
                            uint64_t remote_buf_start, size_t size,
                            const std::string& rdma_descr,
                            uint16_t channel = 0,
                            uint64_t local_offset = 0,
                            ibv_wc_status* status = nullptr,
                            void* async_handle = nullptr);

    ssize_t handlePutObject(const std::string& key,
                            struct rdma_buffer* local_rdma_buff,
                            uint64_t remote_buf_start, size_t size,
                            const std::string& rdma_descr,
                            uint32_t poll_delay,
                            uint16_t channel = 0,
                            uint64_t local_offset = 0,
                            ibv_wc_status* status = nullptr,
                            void* async_handle = nullptr);
    ssize_t handlePutObject(const std::string& key,
                            struct rdma_buffer* local_rdma_buff,
                            uint64_t remote_buf_start, size_t size,
                            const std::string& rdma_descr,
                            uint16_t channel = 0,
                            uint64_t local_offset = 0,
                            ibv_wc_status* status = nullptr,
                            void* async_handle = nullptr);

    int poll(cuObjAsyncEvent_t* events,
             int max_events,
             uint16_t channel = 0);

    bool isConnected();
    uint16_t allocateChannelId();
    void freeChannelId(uint16_t channel_id);

    static void setupTelemetry(bool use_OTEL, std::ostream* os);
    static void shutdownTelemetry();
    static void setTelemFlags(unsigned flags);
};

4.2. Constructors#

4.2.1. Basic Constructor#

cuObjServer(const char* ip, unsigned short port, unsigned proto);

Creates a server object with default RDMA tunables and starts the RDMA session.

Parameters:

ip: Server endpoint address string.
port: Server port.
proto: RDMA descriptor protocol. Use CUOBJ_PROTO_RDMA_DC_V1.

Notes:

Check isConnected() after construction.
A failed session start does not prevent object construction. Use isConnected() to report readiness.
Current behavior does not validate unsupported proto values.

4.2.2. Constructor With Tunables#

cuObjServer(const char* ip, unsigned short port, unsigned proto,
            cuObjRDMATunable params);

Creates a server object with caller-provided RDMA tunables and starts the RDMA session.

Parameters:

ip: Server endpoint address string.
port: Server port.
proto: RDMA descriptor protocol. Use CUOBJ_PROTO_RDMA_DC_V1.
params: RDMA tunables copied into the server before session start.

4.2.3. Default Constructor#

cuObjServer() = default;

The endpoint constructors above are the application-facing construction path. Applications should construct cuObjServer with ip, port, and proto, then check isConnected().

4.3. Memory Management APIs#

4.3.1. `allocHostBuffer()`#

void* allocHostBuffer(size_t size);

Allocates host memory aligned to the system page size.

Parameters:

size: Allocation size in bytes.

Returns:

Pointer to the allocated buffer on success.
nullptr when size == 0 or allocation fails.

Notes:

The returned memory should be released with free().
This API is optional. Applications may register their own host memory.

4.3.2. `registerBuffer()` for a Single Buffer#

struct rdma_buffer* registerBuffer(void* ptr, size_t size);

Registers a contiguous host buffer.

Parameters:

ptr: Host memory start address.
size: Buffer size in bytes.

Returns:

Opaque RDMA buffer handle on success.
nullptr on failure.

Failure cases:

Server RDMA device is not available.
ptr == nullptr.
size == 0.
Buffer registration fails.

Notes:

The memory must remain valid until deRegisterBuffer() is called.
The returned struct rdma_buffer* is an opaque handle for cuObjServer APIs.
Server registration supports host memory.

4.3.3. `registerBuffer()` for a Scatter-Gather List#

struct rdma_buffer* registerBuffer(
    std::vector<cuObjScatterGatherEntry_t> sglist);

Registers a scatter-gather list of host buffers.

Parameters:

sglist: Vector of scatter-gather entries.

Returns:

Opaque RDMA buffer handle on success.
nullptr on failure.

Failure cases:

Server RDMA device is not available.
sglist is empty.
sglist.size() > 10.
Any entry has addr == nullptr.
Any entry has size == 0.
Allocation or registration fails.

Notes:

The maximum number of entries is 10.
All entries must remain valid until deRegisterBuffer() is called.
For a registered scatter-gather buffer with more than one entry, handleGetObject() and handlePutObject() treat nonzero local_offset as 0.

4.3.4. `getRDMABufferFromSgList()`#

struct rdma_buffer* getRDMABufferFromSgList(
    struct rdma_buffer* rdma_buffer,
    std::vector<cuObjScatterGatherEntryWithOffset_t> sglist);

Creates a scatter-gather view over a previously registered buffer.

Parameters:

rdma_buffer: Opaque handle returned by registerBuffer().
sglist: Offset and size entries relative to the registered base buffer.

Returns:

Opaque RDMA buffer view on success.
nullptr on failure.

Failure cases:

sglist is empty.
rdma_buffer == nullptr.
rdma_buffer was not created by registerBuffer().
rdma_buffer has more than one registered entry.
Allocation fails.
Sum of view entry sizes is greater than the base registered buffer size.

Notes:

The returned view must be released with putRDMABufferFromSgList().
The base registered buffer must remain registered while the view exists.
Current validation checks the total view size against the base registered buffer size. Callers must provide offsets that are within the registered base buffer.

4.3.5. `putRDMABufferFromSgList()`#

void putRDMABufferFromSgList(struct rdma_buffer* rdma_buffer);

Releases a view created by getRDMABufferFromSgList().

Parameters:

rdma_buffer: View returned by getRDMABufferFromSgList().

Notes:

Passing nullptr logs an error and returns.
Passing a buffer that is not a scatter-gather view logs an error and returns.
This API does not deregister the base registered buffer.

4.3.6. `deRegisterBuffer()`#

void deRegisterBuffer(struct rdma_buffer* rdma_buff);

Deregisters a buffer returned by registerBuffer().

Parameters:

rdma_buff: Opaque handle returned by registerBuffer().

Notes:

Passing nullptr is a no-op.
Complete all synchronous and asynchronous I/O using the buffer before deregistering it.
This API does not free the underlying application memory.

4.4. I/O Operations#

4.4.1. `handleGetObject()`#

ssize_t handleGetObject(const std::string& key,
                        struct rdma_buffer* local_rdma_buff,
                        uint64_t remote_buf_start, size_t size,
                        const std::string& rdma_descr,
                        uint16_t channel = 0,
                        uint64_t local_offset = 0,
                        ibv_wc_status* status = nullptr,
                        void* async_handle = nullptr);

ssize_t handleGetObject(const std::string& key,
                        struct rdma_buffer* local_rdma_buff,
                        uint64_t remote_buf_start, size_t size,
                        const std::string& rdma_descr,
                        uint32_t poll_delay,
                        uint16_t channel = 0,
                        uint64_t local_offset = 0,
                        ibv_wc_status* status = nullptr,
                        void* async_handle = nullptr);

Services a GET request using data already present in the registered server buffer. handleGetObject() performs an RDMA WRITE to the client buffer.

Parameters:

key: Request identifier used for telemetry.
local_rdma_buff: Registered server buffer containing the data to send.
remote_buf_start: Remote buffer start address for this request.
size: Transfer size in bytes.
rdma_descr: Opaque RDMA descriptor string received from the client.
poll_delay: Per-request polling delay in nanoseconds for synchronous calls. The overload without this argument uses params.getDelayInterval().
channel: Channel returned by allocateChannelId().
local_offset: Offset into local_rdma_buff.
status: Optional completion status output for synchronous calls.
async_handle: User handle for asynchronous completion. Use nullptr for synchronous operation.

Returns:

Number of bytes transferred on synchronous success.
0 when asynchronous submission succeeds.
Negative errno-style value on failure.

Synchronous success normally returns size. Asynchronous success, when async_handle != nullptr, returns 0 after successful submission. Completion status is reported later by poll().

Failure values:

-ENODEV: Server RDMA device is not available.
-EIO: Invalid input, descriptor parse failure, invalid or unallocated channel, requested size exceeds the registered local buffer range, requested remote range is outside the descriptor range, submission failure, or completion failure.
-ENOMEM: Allocation failure.
-EAFNOSUPPORT: Address family is not supported by the active path.
-EPROTO is not a documented return value for handleGetObject() or handlePutObject() in this release.

Notes:

local_rdma_buff must be non-null.
remote_buf_start and size must be nonzero.
size + local_offset must not exceed the registered local buffer size.
For a registered scatter-gather buffer with more than one entry, current behavior first validates size + local_offset against the total registered size. If that validation passes and local_offset is nonzero, the call logs an error, treats local_offset as 0, and continues.
channel must be allocated before the call. The default argument value 0 is not automatically allocated.
When async_handle == nullptr, the call returns after completion.
When async_handle != nullptr, the call returns after submission and the application must call poll() on the same channel.
For synchronous completion failures, status receives the completion status value and the function returns -EIO.
status is not populated for pre-submission validation errors.

4.4.2. `handlePutObject()`#

ssize_t handlePutObject(const std::string& key,
                        struct rdma_buffer* local_rdma_buff,
                        uint64_t remote_buf_start, size_t size,
                        const std::string& rdma_descr,
                        uint16_t channel = 0,
                        uint64_t local_offset = 0,
                        ibv_wc_status* status = nullptr,
                        void* async_handle = nullptr);

ssize_t handlePutObject(const std::string& key,
                        struct rdma_buffer* local_rdma_buff,
                        uint64_t remote_buf_start, size_t size,
                        const std::string& rdma_descr,
                        uint32_t poll_delay,
                        uint16_t channel = 0,
                        uint64_t local_offset = 0,
                        ibv_wc_status* status = nullptr,
                        void* async_handle = nullptr);

Services a PUT request by populating the registered server buffer. handlePutObject() performs an RDMA READ from the client buffer.

Parameters and return values match handleGetObject().

Notes:

local_rdma_buff is the destination buffer for the request data.
For synchronous success, the requested data is available in local_rdma_buff when the call returns.
For asynchronous success, wait for poll() to report completion before reading or reusing the destination buffer.

4.5. Asynchronous Completion#

4.5.1. `poll()`#

int poll(cuObjAsyncEvent_t* events, int max_events, uint16_t channel = 0);

Polls for completions submitted with non-null async_handle.

Parameters:

events: Output array for completion events.
max_events: Maximum number of events to return.
channel: Channel used for the asynchronous submissions.

Returns:

Number of completion events returned.
0 when no completions are available.
-ENODEV if the server RDMA device is not available.
-EINVAL if events == nullptr or channel is invalid or unallocated.
-EIO if any polled completion reports a failure.

Notes:

Successful completions return a count from 0 to max_events, after the max_events cap is applied.
Pass a non-negative max_events value.
max_events values greater than 16 are capped to 16.
channel must be allocated before poll() is called.
poll() must be called on the same channel used for the asynchronous handleGetObject() or handlePutObject() submission.
For a non-negative return value n, entries events[0] through events[n - 1] are valid.
events[i].async_handle is the user handle passed to handleGetObject() or handlePutObject().
events[i].status is the completion status. IBV_WC_SUCCESS (0) means success.
If poll() returns -EIO because a completion has non-success status, completion status is still filled for the completions processed before the error return. In this error case, the return value is not a completion count.
If a completion failure is reported and qp_reset_on_failure is true, the server attempts queue-pair reset before poll() returns -EIO.

4.6. Channel Management#

4.6.1. `allocateChannelId()`#

uint16_t allocateChannelId();

Allocates a channel for I/O submission and polling.

Returns:

Allocated channel ID in the range 0 to num_dcis - 1.
INVALID_CHANNEL_ID (UINT16_MAX) when no channel is available.

Notes:

The first successful allocation returns channel 0.
A channel must be allocated before it is passed to handleGetObject(), handlePutObject(), or poll().
Use one channel from only one thread at a time.

4.6.2. `freeChannelId()`#

void freeChannelId(uint16_t channel_id);

Releases a channel returned by allocateChannelId().

Parameters:

channel_id: Channel to release.

Notes:

Passing a value greater than or equal to num_dcis is a no-op.
Do not free a channel while operations submitted on that channel are still in progress.

4.7. Connection Management#

4.7.1. `isConnected()`#

bool isConnected();

Returns:

true if the server RDMA session was started successfully.
false otherwise.

4.8. Telemetry Management#

4.8.1. `setupTelemetry()`#

static void setupTelemetry(bool use_OTEL, std::ostream* os);

Configures telemetry output for subsequently created cuObjServer objects.

Parameters:

use_OTEL: Request OpenTelemetry telemetry when OpenTelemetry support is built in.
os: Output stream for stream telemetry.

Valid values:

Parameter	Value	Description
`use_OTEL`	`false`	Use stream telemetry through `os`.
`use_OTEL`	`true`	Request OpenTelemetry telemetry when OpenTelemetry support is built in.

Notes:

Call setupTelemetry() before creating the first cuObjServer object. Current behavior applies the setting when the telemetry object is created.
If OpenTelemetry support is not built in, stream telemetry is used.
os must be non-null when stream telemetry is used.
os must remain valid until all cuObjServer objects are destroyed.

4.8.2. `shutdownTelemetry()`#

static void shutdownTelemetry();

Resets telemetry configuration to the default output stream.

Notes:

Call shutdownTelemetry() before closing a custom stream passed to setupTelemetry().
Telemetry resources are fully released when the last cuObjServer object is destroyed.

4.8.3. `setTelemFlags()`#

static void setTelemFlags(unsigned flags);

Configures telemetry logging flags.

Parameters:

flags: Bitmask of telemetry logging flags.

Valid values:

Flag	Value	Description
`CUOBJ_LOG_PATH_INFO`	`0x0001`	Enables informational telemetry messages.
`CUOBJ_LOG_PATH_DEBUG`	`0x0002`	Enables debug telemetry messages.
`CUOBJ_LOG_PATH_ERROR`	`0x0004`	Enables error telemetry messages.

Flags may be combined with bitwise OR:

cuObjServer::setTelemFlags(CUOBJ_LOG_PATH_ERROR | CUOBJ_LOG_PATH_INFO);

Passing 0 disables info, debug, and error telemetry messages. Bits other than the valid values listed above have no defined effect.

Notes:

Call setTelemFlags() before creating the first cuObjServer object. Current behavior applies the flags when the telemetry object is created.
The API does not return a status value.

4. cuObjServer Class API#

4.1. Class Declaration#

4.2. Constructors#

4.2.1. Basic Constructor#

4.2.2. Constructor With Tunables#

4.2.3. Default Constructor#

4.3. Memory Management APIs#

4.3.1. allocHostBuffer()#

4.3.2. registerBuffer() for a Single Buffer#

4.3.3. registerBuffer() for a Scatter-Gather List#

4.3.4. getRDMABufferFromSgList()#

4.3.5. putRDMABufferFromSgList()#

4.3.6. deRegisterBuffer()#

4.4. I/O Operations#

4.4.1. handleGetObject()#

4.4.2. handlePutObject()#

4.5. Asynchronous Completion#

4.5.1. poll()#

4.6. Channel Management#

4.6.1. allocateChannelId()#

4.6.2. freeChannelId()#

4.7. Connection Management#

4.7.1. isConnected()#

4.8. Telemetry Management#

4.8.1. setupTelemetry()#

4.8.2. shutdownTelemetry()#

4.8.3. setTelemFlags()#

4.3.1. `allocHostBuffer()`#

4.3.2. `registerBuffer()` for a Single Buffer#

4.3.3. `registerBuffer()` for a Scatter-Gather List#

4.3.4. `getRDMABufferFromSgList()`#

4.3.5. `putRDMABufferFromSgList()`#

4.3.6. `deRegisterBuffer()`#

4.4.1. `handleGetObject()`#

4.4.2. `handlePutObject()`#

4.5.1. `poll()`#

4.6.1. `allocateChannelId()`#

4.6.2. `freeChannelId()`#

4.7.1. `isConnected()`#

4.8.1. `setupTelemetry()`#

4.8.2. `shutdownTelemetry()`#

4.8.3. `setTelemFlags()`#