Function TRITONSERVER_ServerOptionsSetCudaMemoryPoolByteSize

Function Documentation

TRITONSERVER_Error *TRITONSERVER_ServerOptionsSetCudaMemoryPoolByteSize(TRITONSERVER_ServerOptions *options, int gpu_device, uint64_t size)

Set the total CUDA memory byte size that the server can allocate on given GPU device in a server options.

The pinned memory pool will be shared across Triton itself and the backends that use TRITONBACKEND_MemoryManager to allocate memory.

Return

a TRITONSERVER_Error indicating success or failure.

Parameters

  • options: The server options object.

  • gpu_device: The GPU device to allocate the memory pool.

  • size: The CUDA memory pool byte size.