Class CudaStreamHandler

This class handles usage of CUDA streams for operators.

When using CUDA operations the default stream ‘0’ synchronizes with all other streams in the same context, see https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html#stream-sync-behavior. This can reduce performance. The CudaStreamHandler class manages streams across operators and makes sure that CUDA operations are properly chained.

Usage:

• add an instance of CudaStreamHandler to your operator

• call CudaStreamHandler::register_interface(spec) from the operator setup() function

• in the compute() function call CudaStreamHandler::from_message(), this will get the CUDA stream from the message of the previous operator. When the operator receives multiple messages, then call CudaStreamHandler::from_messages(). This will synchronize with multiple streams.

• when executing CUDA functions CudaStreamHandler::get() to get the CUDA stream which should be used by your CUDA function

• before publishing the output message(s) of your operator call CudaStreamHandler::to_message() on each message. This will add the CUDA stream used by the CUDA functions in your operator to the output message.

Public Functions

~CudaStreamHandler()

Destroy the CudaStreamHandler object.

void define_params(OperatorSpec &spec, bool required = false)

Define the parameters used by this class.

Parameters

• spec – OperatorSpec to define the cuda_stream_pool parameter

• required – if set then it’s required that the CUDA stream pool is specified

void defineParams(OperatorSpec &spec, bool required = false)

Define the parameters used by this class.

This method is deprecated in favor of define_params.

Deprecated:

since 1.0

Parameters

• spec – OperatorSpec to define the cuda_stream_pool parameter

• required – if set then it’s required that the CUDA stream pool is specified

gxf_result_t from_message(gxf_context_t context, const nvidia::gxf::Expected<nvidia::gxf::Entity> &message)

Get the CUDA stream for the operation from the incoming message

Parameters

• context –

• message –

Returns

gxf_result_t

gxf_result_t fromMessage(gxf_context_t context, const nvidia::gxf::Expected<nvidia::gxf::Entity> &message)

Get the CUDA stream for the operation from the incoming message

This method is deprecated in favor of from_message.

Deprecated:

since 1.0

Parameters

• context –

• message –

Returns

gxf_result_t

gxf_result_t from_messages(gxf_context_t context, const std::vector<nvidia::gxf::Entity> &messages)

Get the CUDA stream for the operation from the incoming messages

Parameters

• context –

• messages –

Returns

gxf_result_t

gxf_result_t fromMessages(gxf_context_t context, const std::vector<nvidia::gxf::Entity> &messages)

Get the CUDA stream for the operation from the incoming messages

This method is deprecated in favor of from_messages.

Deprecated:

since 1.0

Parameters

• context –

• messages –

Returns

gxf_result_t

gxf_result_t to_message(nvidia::gxf::Expected<nvidia::gxf::Entity> &message)

Add the used CUDA stream to the outgoing message

Parameters

message –

Returns

gxf_result_t

gxf_result_t toMessage(nvidia::gxf::Expected<nvidia::gxf::Entity> &message)

Add the used CUDA stream to the outgoing message

This method is deprecated in favor of to_message.

Deprecated:

since 1.0

Parameters

message –

Returns

gxf_result_t

nvidia::gxf::Handle<nvidia::gxf::CudaStream> get_stream_handle(gxf_context_t context)

Get the CUDA stream handle which should be used for CUDA commands

Parameters

context –

Returns

nvidia::gxf::Handle<nvidia::gxf::CudaStream>

nvidia::gxf::Handle<nvidia::gxf::CudaStream> getStreamHandle(gxf_context_t context)

Get the CUDA stream handle which should be used for CUDA commands

This method is deprecated in favor of get_stream_handle.

Deprecated:

since 1.0

Parameters

context –

Returns

nvidia::gxf::Handle<nvidia::gxf::CudaStream>

cudaStream_t get_cuda_stream(gxf_context_t context)

Get the CUDA stream which should be used for CUDA commands.

If no message stream is set and no stream can be allocated, return the default stream.

Parameters

context –

Returns

cudaStream_t

cudaStream_t getCudaStream(gxf_context_t context)

Get the CUDA stream which should be used for CUDA commands.

If no message stream is set and no stream can be allocated, return the default stream.

This method is deprecated in favor of get_cuda_stream.

Deprecated:

since 1.0

Parameters

context –

Returns

cudaStream_t