Class SharedMemoryControlContext

Class Documentation

class SharedMemoryControlContext

A SharedMemoryControlContext object is used to control the registration / unregistration and get the status of shared memory regions on the inference server.

Once created, a SharedMemoryControlContext object can be used repeatedly.

A SharedMemoryControlContext object can use either HTTP protocol or GRPC protocol depending on the Create function (SharedMemoryControlHttpContext::Create or SharedMemoryControlGrpcContext::Create). For example:

std::unique_ptr<SharedMemoryControlContext> ctx;
SharedMemoryControlGrpcContext::Create(&ctx, "localhost:8000");
std::string name = "shared_memory";
std::string shm_key = "/input";
ctx->RegisterSharedMemory(name, shm_key, 0, 128);
...
cudaIpcMemHandle_t cuda_shm_handle;
ctx->RegisterCudaSharedMemory(name, cuda_shm_handle, 128);
...
ctx->UnregisterSharedMemory(name);
...
ctx->UnregisterAllSharedMemory();
...
SharedMemoryStatus status;
ctx->GetSharedMemoryStatus(&status);
...

Public Functions

virtual ~SharedMemoryControlContext() = 0
virtual Error RegisterSharedMemory(const std::string &name, const std::string &shm_key, size_t offset, size_t byte_size) = 0

Register a system shared memory region on the inference server.

If the shared memory region is already registered, it will return error ‘TRTSERVER_ERROR_ALEADY_EXISTS’.

Return

Error object indicating success or failure.

Parameters

  • name: The user-given name for the shared memory region to be registered.

  • shm_key: The unique name of the location in shared memory being registered.

  • offset: The offset into the shared memory region.

  • byte_size: The size, in bytes of the shared memory region.

virtual Error RegisterCudaSharedMemory(const std::string &name, const cudaIpcMemHandle_t &cuda_shm_handle, size_t byte_size, int device_id) = 0

Register a CUDA shared memory region on the inference server.

If the shared memory region is already registered, it will return error ‘TRTSERVER_ERROR_ALEADY_EXISTS’.

Return

Error object indicating success or failure.

Parameters

  • name: The user-given name for the shared memory region to be registered.

  • cuda_shm_handle: The CUDA memory handle representing the location in shared memory being registered.

  • byte_size: The size, in bytes of the shared memory region.

  • device: id The GPU device ID the shared memory region is in.

virtual Error UnregisterSharedMemory(const std::string &name) = 0

Unregister a registered shared memory region on the inference server.

If the shared memory region is not registered, do nothing and return success.

Return

Error object indicating success or failure.

Parameters

  • name: The user-given name for the shared memory region to be unregistered.

virtual Error UnregisterAllSharedMemory() = 0

Unregisters all registered shared memory regions on the inference server.

Return

Error object indicating success or failure.

virtual Error GetSharedMemoryStatus(SharedMemoryStatus *status) = 0

Get the list of all active/registered shared memory regions on the inference server.

Return

Error object indicating success or failure.

Parameters

  • status: Returns the active shared memory regions.