holoscan::gxf::CudaObjectHandler

This class handles usage of CUDA streams for operators.

When using CUDA operations the default stream ‘0’ synchronizes with all other streams in the same context, see https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html#stream-sync-behavior. This can reduce performance. The CudaObjectHandler class manages CUDA streams and events across operators and makes sure that CUDA operations are properly chained.

Usage:

• This class is automatically added as an internal data member of each operators ExecutionContext. It will be automatically configured by ExecutionContext::init_cuda_object_handler(op) by GXFWrapper::start(), just before Operator::start is called.
• A stream pool for use by CudaObjectHandler can be added to the operator either by explicitly adding a parameter with type std::shared_ptr<CudaStreamPool> and name cuda_stream_pool or by passing an Arg<std::shared_ptr<CudaStreamPool>> to Fragment::make_operator when creating the operator. It is not required to provide a stream pool, but allocation of an internal stream or allocation of additional streams via allocate_cuda_stream is only possible if a stream pool is present.
• This class is not intended for direct use by Application authors, but instead to support the public methods available on InputContext, OutputContext and ExecutionContext as described below.
• When the InputContext::receive method is called for a given port, the operator’s CudaObjectHandler class will update its internal mapping of the streams available on the input ports.
• When InputContext::receive_cuda_stream is called, any received streams found by the prior receive call for the specified port will be synchronized to the operator’s internal stream and then that internal stream will returned as a standard CUDA Runtime API cudaStream_t. If no CudaStreamPool was configured, it will not be possible to create the internal stream, so in that case, the first CUDA stream found on the input will be returned and any remaining streams on the input are synchronized to it. If there are no streams on the input port and there is no internal CudaStreamPool, then cudaStreamDefault is returned. When a non-default stream is returned, this method calls cudaSetDevice to set the active device to match the stream that is returned. When a non-default stream is returned, this method also will have automatically configured the output ports of the operator to emit that stream, so manually calling OutputContext::set_cuda_stream is not necessary when using this method.
• The InputContext::receive_cuda_streams method is intended for advanced use cases where the user wants to handle all streams found and their synchronization manually. It just returns a vector<std::optional<``cudaStream_t``>> where the size of the vector is equal to the number of messages found on the input port. Any messages without a stream will have a std::nullopt entry in the vector.
• The ExecutionContext::allocate_cuda_stream method can be used if it is necessary to allocate an additional stream for use by the operator. In most cases, this will not be necessary and the stream that is returned by InputContext::receive_cuda_stream can be used.
• The ExecutionContext::device_from_stream method can be used to determined which CUDA device id a given cudaStream_t returned by InputContext::receive_cuda_stream or InputContext::receive_cuda_streams belongs to.
• The OutputContext::set_cuda_stream method can be used to emit specific streams on specific output ports. Any non-default stream received by InputContext::receive_cuda_stream would already automatically be output, so this method is mainly useful if doing manual management of the streams received via InputContext::receive_cuda_streams or if additional internal streams were allocated via ExecutionContext::allocate_cuda_stream.

#include <holoscan/gxf/gxf_cuda.hpp>

Inherits from: holoscan::CudaObjectHandler (public)

Constructors

Destructor

~CudaObjectHandler

holoscan::gxf::CudaObjectHandler::~CudaObjectHandler() overrideholoscan::gxf::CudaObjectHandler::~CudaObjectHandler() override

Destroy the CudaObjectHandler object.

Methods

init_from_operator

void holoscan::gxf::CudaObjectHandler::init_from_operator(
    Operator *op
) override

Use a CudaStreamPool from the specified Operator if one is present.

Parameters

is_gpu_available

bool holoscan::gxf::CudaObjectHandler::is_gpu_available() const override

Check if GPU capability is present on the system.

Returns: true if GPU(s) are available, false if no GPU is present

add_stream

gxf_result_t holoscan::gxf::CudaObjectHandler::add_stream(
    const CudaStreamHandle &stream_handle,
    const std::string &output_port_name
)

get_cuda_stream_handle

expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::get_cuda_stream_handle(expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::get_cuda_stream_handle(
    gxf_context_t context,
    const std::string &input_port_name,
    bool allocate = true,
    bool sync_to_default = false
)

Get the CUDA stream handle which should be used for CUDA commands involving data from the specified input port.

For multi-receivers or input ports with queue size > 1, the first stream found is returned after any remaining streams are synchronized to it.

See get_cuda_stream_handles() instead to receive a vector of (optional) CUDA stream handles (one for each message).

If no message stream is set and the allocate flag is true, a stream will be allocated from the internal CudaStreamPool. Only if this allocation fails, would an unexpected be returned.

Returns: CudaStreamHandle

Parameters

get_cuda_stream_handles

expected<std::vector<std::optional<CudaStreamHandle>>, RuntimeError> holoscan::gxf::CudaObjectHandler::get_cuda_stream_handles(expected<std::vector<std::optional<CudaStreamHandle>>, RuntimeError> holoscan::gxf::CudaObjectHandler::get_cuda_stream_handles(
    gxf_context_t context,
    const std::string &input_port_name
)

Get the CUDA stream handles which should be used for CUDA commands involving data from the specified input port.

The size of the vector returned will be equal to the number of messages received on the input port. Any messages which did not contain a stream will result in a std::nullopt in the vector.

Returns: vector<std::optional<CudaStreamHandle>>

Parameters

get_cuda_stream

cudaStream_t holoscan::gxf::CudaObjectHandler::get_cuda_stream(
    void *context,
    const std::string &input_port_name,
    bool allocate = false,
    bool sync_to_default = true
) override

Get the CUDA stream which should be used for CUDA commands involving data from the specified input port.

For multi-receivers or input ports with queue size > 1, see get_cuda_streams() instead to receive a vector of CUDA streams (one for each message).

If no message stream is set and no stream can be allocated from the internal CudaStreamPool, returns CudaStreamDefault.

Returns: cudaStream_t

Parameters

get_cuda_streams

std::vector<std::optional<cudaStream_t>> holoscan::gxf::CudaObjectHandler::get_cuda_streams(
    void *context,
    const std::string &input_port_name
) override

Get the CUDA stream which should be used for CUDA commands involving data from the specified input port.

The size of the vector returned will be equal to the number of messages received on the input port. Any messages which did not contain a stream will result in a cudaStreamDefault in the vector.

Returns: vector<std::optional<cudaStream_t>>

Parameters

synchronize_streams

gxf_result_t holoscan::gxf::CudaObjectHandler::synchronize_streams(
    std::vector<std::optional<CudaStreamHandle>> stream_handles,
    CudaStreamHandle target_stream_handle,
    bool sync_to_default_stream = true
)

stream_from_stream_handle

cudaStream_t holoscan::gxf::CudaObjectHandler::stream_from_stream_handle(
    CudaStreamHandle stream_handle
)

Get the cudaStream_t value corresponding to a CudaStreamHandle.

Returns: The CUDA stream contained within the CudaStream object

Parameters

stream_handle_from_stream

expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::stream_handle_from_stream(expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::stream_handle_from_stream(
    cudaStream_t stream
)

Get the CudaStreamHandle corresponding to a cudaStream_t.

Returns: GXF Handle to the CudaStream object if found, otherwise an unexpected is returned.

Parameters

get_output_stream_cid

expected<gxf_uid_t, ErrorCode> holoscan::gxf::CudaObjectHandler::get_output_stream_cid(
    const std::string &output_port_name
)

Get the GXF component ID for any stream to be emitted on the specified output port.

Returns: expected<gxf_uid_t>

Parameters

streams_from_message

gxf_result_t holoscan::gxf::CudaObjectHandler::streams_from_message(
    gxf_context_t context,
    const nvidia::gxf::Entity &message,
    const std::string &input_name
)

Get the GXF component IDs for any events to be emitted on the specified output port.

Returns: expected<std::vector<gxf_uid_t>>

Parameters

allocate_internal_stream

expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::allocate_internal_stream(expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::allocate_internal_stream(
    gxf_context_t context,
    const std::string &stream_name
)

Allocate an internal CUDA stream and store it in the mapping for the given input port.

Returns: GXF Handle to the allocated CudaStream component

Parameters

release_internal_streams

int holoscan::gxf::CudaObjectHandler::release_internal_streams(
    void *context
) override

Release all internally allocated CUDA streams.

clear_received_streams

void holoscan::gxf::CudaObjectHandler::clear_received_streams() override

Retain the existing unordered_maps and vectors of received streams, but clear the contents.

This is used to refresh the state of the received streams before each Operator::compute call.

allocate_cuda_stream

expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::allocate_cuda_stream(expected<CudaStreamHandle, RuntimeError> holoscan::gxf::CudaObjectHandler::allocate_cuda_stream(
    gxf_context_t context
)

allocate a new stream from the internal stream pool

from_messages

gxf_result_t holoscan::gxf::CudaObjectHandler::from_messages(
    gxf_context_t context,
    size_t message_count,
    const nvidia::gxf::Entity *messages
)

cuda_stream_pool_handle

expected<nvidia::gxf::Handle<nvidia::gxf::CudaStreamPool>, RuntimeError> holoscan::gxf::CudaObjectHandler::cuda_stream_pool_handle(expected<nvidia::gxf::Handle<nvidia::gxf::CudaStreamPool>, RuntimeError> holoscan::gxf::CudaObjectHandler::cuda_stream_pool_handle(expected<nvidia::gxf::Handle<nvidia::gxf::CudaStreamPool>, RuntimeError> holoscan::gxf::CudaObjectHandler::cuda_stream_pool_handle(
    gxf_context_t context
)

Member variables