holoscan::CudaStreamHandler

This class handles usage of CUDA streams for operators.

When using CUDA operations the default stream ‘0’ synchronizes with all other streams in the same context, see https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html#stream-sync-behavior. This can reduce performance. The CudaStreamHandler class manages streams across operators and makes sure that CUDA operations are properly chained.

Usage:

• add an instance of CudaStreamHandler to your operator
• call CudaStreamHandler::register_interface(spec) from the operator setup() function
• in the compute() function call CudaStreamHandler::from_message(), this will get the CUDA stream from the message of the previous operator. When the operator receives multiple messages, then call CudaStreamHandler::from_messages(). This will synchronize with multiple streams.
• when executing CUDA functions CudaStreamHandler::get() to get the CUDA stream which should be used by your CUDA function
• before publishing the output message(s) of your operator call CudaStreamHandler::to_message() on each message. This will add the CUDA stream used by the CUDA functions in your operator to the output message.

#include <holoscan/cuda_stream_handler.hpp>

Constructors

Destructor

~CudaStreamHandler

holoscan::CudaStreamHandler::~CudaStreamHandler()

Destroy the CudaStreamHandler object.

Methods

define_params

void holoscan::CudaStreamHandler::define_params(
    OperatorSpec &spec,
    bool required = false
)

Define the parameters used by this class.

Parameters

defineParams

void holoscan::CudaStreamHandler::defineParams(
    OperatorSpec &spec,
    bool required = false
)

Define the parameters used by this class.

This method is deprecated in favor of define_params.

Parameters

from_message

gxf_result_t holoscan::CudaStreamHandler::from_message(
    gxf_context_t context,
    const nvidia::gxf::Expected<nvidia::gxf::Entity> &message
)

Get the CUDA stream for the operation from the incoming message.

Returns: gxf_result_t

fromMessage

gxf_result_t holoscan::CudaStreamHandler::fromMessage(
    gxf_context_t context,
    const nvidia::gxf::Expected<nvidia::gxf::Entity> &message
)

Get the CUDA stream for the operation from the incoming message.

This method is deprecated in favor of from_message.

Returns: gxf_result_t

from_messages

gxf_result_t holoscan::CudaStreamHandler::from_messages(
    gxf_context_t context,
    const std::vector<holoscan::gxf::Entity> &messages
)

fromMessages

gxf_result_t holoscan::CudaStreamHandler::fromMessages(
    gxf_context_t context,
    const std::vector<nvidia::gxf::Entity> &messages
)

Get the CUDA stream for the operation from the incoming messages.

This method is deprecated in favor of from_messages.

Returns: gxf_result_t

to_message

gxf_result_t holoscan::CudaStreamHandler::to_message(
    nvidia::gxf::Expected<nvidia::gxf::Entity> &message
)

Add the used CUDA stream to the outgoing message.

Returns: gxf_result_t

toMessage

gxf_result_t holoscan::CudaStreamHandler::toMessage(
    nvidia::gxf::Expected<nvidia::gxf::Entity> &message
)

Add the used CUDA stream to the outgoing message.

This method is deprecated in favor of to_message.

Returns: gxf_result_t

get_stream_handle

nvidia::gxf::Handle<nvidia::gxf::CudaStream> holoscan::CudaStreamHandler::get_stream_handle(
    gxf_context_t context
)

Get the CUDA stream handle which should be used for CUDA commands.

Returns: nvidia::gxf::Handle<nvidia::gxf::CudaStream>

getStreamHandle

nvidia::gxf::Handle<nvidia::gxf::CudaStream> holoscan::CudaStreamHandler::getStreamHandle(
    gxf_context_t context
)

Get the CUDA stream handle which should be used for CUDA commands.

This method is deprecated in favor of get_stream_handle.

Returns: nvidia::gxf::Handle<nvidia::gxf::CudaStream>

get_cuda_stream

cudaStream_t holoscan::CudaStreamHandler::get_cuda_stream(
    gxf_context_t context
)

Get the CUDA stream which should be used for CUDA commands.

If no message stream is set and no stream can be allocated, return the default stream.

Returns: cudaStream_t

getCudaStream

cudaStream_t holoscan::CudaStreamHandler::getCudaStream(
    gxf_context_t context
)

Get the CUDA stream which should be used for CUDA commands.

If no message stream is set and no stream can be allocated, return the default stream.

This method is deprecated in favor of get_cuda_stream.

Returns: cudaStream_t

allocate_internal_stream

gxf_result_t holoscan::CudaStreamHandler::allocate_internal_stream(
    gxf_context_t context
)

Allocate the internal CUDA stream.

Returns: gxf_result_t

Member variables