holoscan::StreamOrderedAllocator

Beta

CUDA device memory allocator using stream-ordered allocation.

StreamOrderedAllocator uses CUDA’s stream-ordered memory allocator (cudaMallocAsync/cudaFreeAsync) to dynamically allocate device memory. Stream-ordered allocation enables memory operations to be tied to specific CUDA streams, allowing allocation and deallocation without blocking the host or other streams.

See the CUDA Programming Guide section on Stream-Ordered Memory Allocator for details on the underlying CUDA feature.

This allocator only supports CUDA device memory. If host memory is also needed, see RMMAllocator which provides both device and pinned host memory pools.

Because it is a CudaAllocator it supports both synchronous (allocate, free) and asynchronous (allocate_async, free_async) APIs for memory allocation.

The values for the memory parameters, such as device_memory_initial_size must be specified in the form of a string containing a non-negative integer value followed by a suffix representing the units. Supported units are B, KB, MB, GB and TB where the values are powers of 1024 bytes (e.g. MB = 1024 * 1024 bytes). Examples of valid units are “512MB”, “256 KB”, “1 GB”. If a floating point number is specified that decimal portion will be truncated (i.e. the value is rounded down to the nearest integer).

==Parameters==

device_memory_initial_size (std::string, optional): The initial size of the device memory pool. See above for the format accepted. Defaults to “8MB” on aarch64 and “16MB” on x86_64.
device_memory_max_size (std::string, optional): The maximum size of the device memory pool. See above for the format accepted. The default is to use twice the value set for device_memory_initial_size.
release_threshold (std::string, optional): The amount of reserved memory to hold onto before trying to release memory back to the OS. See above for the format accepted. The default value is “4MB”.
dev_id (int32_t, optional): The CUDA device id specifying which device the memory pool will use. (Default: 0)

#include <holoscan/stream_ordered_allocator.hpp>

Inherits from: holoscan::CudaAllocator (public)

Constructors

StreamOrderedAllocator

Move

Default

Copy

inlineexplicit

template <typename ArgT,
          typename... ArgsT,
          typename = std::enable_if_t<!std::is_base_of_v<::holoscan::Resource, std::decay_t<ArgT>> &&   (std::is_same_v<::holoscan::Arg, std::decay_t<ArgT>> ||   std::is_same_v<::holoscan::ArgList, std::decay_t<ArgT>>)>>
holoscan::StreamOrderedAllocator::StreamOrderedAllocator(holoscan::StreamOrderedAllocator::StreamOrderedAllocator(
    ArgT &&arg,
    ArgsT &&... args
)

Methods

gxf_typename

const char * holoscan::StreamOrderedAllocator::gxf_typename() const override

setup

void holoscan::StreamOrderedAllocator::setup(
    ComponentSpec &spec
) override

Define the resource specification.

Parameters

spec

ComponentSpec &

The reference to the component specification.

get

nvidia::gxf::StreamOrderedAllocator * holoscan::StreamOrderedAllocator::get() constnvidia::gxf::StreamOrderedAllocator * holoscan::StreamOrderedAllocator::get() const

allocate_async

nvidia::byte * holoscan::StreamOrderedAllocator::allocate_async(
    uint64_t size,
    cudaStream_t stream
)

free_async

void holoscan::StreamOrderedAllocator::free_async(
    byte *pointer,
    cudaStream_t stream
)

pool_size

size_t holoscan::StreamOrderedAllocator::pool_size(
    MemoryStorageType type
) const

is_available

virtual bool holoscan::Allocator::is_available(
    uint64_t size
)

allocate

virtual nvidia::byte * holoscan::Allocator::allocate(
    uint64_t size,
    MemoryStorageType type
)

free

virtual void holoscan::Allocator::free(
    nvidia::byte *pointer
)

block_size

uint64_t holoscan::Allocator::block_size()

initialize

void holoscan::gxf::GXFResource::initialize() override

Initialize the component.

This method is called only once when the component is created for the first time, and use of light-weight initialization.

resource_type

ResourceType holoscan::Resource::resource_type() const

Get the resource type.

Returns: The resource type.

name

Set the name of the resource (1)

Set the name of the resource (2)

Const

Resource & holoscan::Resource::name(Resource & holoscan::Resource::name(
    const std::string &name
) &

Set the name of the resource.

Returns: The reference to the resource.

Parameters

name

const std::string &

The name of the resource.

fragment

Set the fragment of the resource

Get a pointer to Fragment object

Const

Resource & holoscan::Resource::fragment(Resource & holoscan::Resource::fragment(
    Fragment *fragment
)

Set the fragment of the resource.

Returns: The reference to the resource.

Parameters

fragment

Fragment *

The pointer to the fragment of the resource.

spec

Set the component specification to the resource

Get the component specification of the resource

Resource & holoscan::Resource::spec(Resource & holoscan::Resource::spec(
    const std::shared_ptr<ComponentSpec> &spec
)

Set the component specification to the resource.

Returns: The reference to the resource.

Parameters

spec

const std::shared_ptr<ComponentSpec> &

The component specification.

spec_shared

std::shared_ptr<ComponentSpec> holoscan::Resource::spec_shared()

Get the shared pointer to the component spec.

Returns: The shared pointer to the component spec.

id

int64_t holoscan::ComponentBase::id() const

Get the identifier of the component.

By default, the identifier is set to -1. It is set to a valid value when the component is initialized.

With the default executor (GXFExecutor), the identifier is set to the GXF component ID.

Returns: The identifier of the component.

add_arg

Add an argument to the component (1)

Add an argument to the component (2)

Add a list of arguments to the component (1)

Add a list of arguments to the component (2)

void holoscan::ComponentBase::add_arg(
    const Arg &arg
)

Add an argument to the component.

Parameters

arg

const Arg &

The argument to add.

args

std::vector<Arg> & holoscan::ComponentBase::args()

Get the list of arguments.

Returns: The vector of arguments.

description

std::string holoscan::ComponentBase::description() const

Get a description of the component.

Returns: YAML string.

See also: to_yaml_node()

service

template <typename ServiceT = DefaultFragmentService>
std::shared_ptr<ServiceT> holoscan::ComponentBase::service(
    std::string_view id = ""
) const

Retrieve a registered fragment service or resource.

Retrieves a previously registered fragment service or resource by its type and optional identifier. Returns nullptr if no service/resource is found with the specified type and identifier.

Note that any changes to the service retrieval logic in this method should be synchronized with the implementation in Fragment::service() method to maintain consistency.

Returns: The shared pointer to the service/resource, or nullptr if not found or if type casting fails.

Template parameters

ServiceT

typename

The type of the service/resource to retrieve. Must inherit from either Resource or FragmentService. Defaults to DefaultFragmentService if not specified.

Parameters

std::string_viewDefaults to ""

The identifier of the service/resource. If empty, retrieves by type only.

get_service_by_type_info

std::shared_ptr<FragmentService> holoscan::ComponentBase::get_service_by_type_info(
    const std::type_info &service_type,
    std::string_view id = ""
) const

Retrieve a registered fragment service or resource for Python bindings.

This is a helper method for Python bindings to retrieve a service by its C++ type info.

Returns: The shared pointer to the base service, or nullptr if not found.

Parameters

service_type

const std::type_info &

The type info of the service/resource to retrieve.

std::string_viewDefaults to ""

The identifier of the service/resource. If empty, retrieves by type only.

gxf_context

Mutable

Const

void holoscan::gxf::GXFComponent::gxf_context(
    gxf_context_t gxf_context
)

gxf_eid

Mutable

Const

void holoscan::gxf::GXFComponent::gxf_eid(
    gxf_uid_t gxf_eid
)

gxf_tid

Mutable

Const

void holoscan::gxf::GXFComponent::gxf_tid(
    gxf_tid_t gxf_tid
)

gxf_cid

Mutable

Const

void holoscan::gxf::GXFComponent::gxf_cid(
    gxf_uid_t gxf_cid
)

gxf_cname

Overload 1

Overload 2

std::string & holoscan::gxf::GXFComponent::gxf_cname()

gxf_graph_entity

Overload 1

Overload 2

std::shared_ptr<nvidia::gxf::GraphEntity> holoscan::gxf::GXFComponent::gxf_graph_entity()

gxf_entity_group_name

std::string holoscan::gxf::GXFComponent::gxf_entity_group_name()

The name of the entity group this component belongs to.

gxf_entity_group_id

gxf_uid_t holoscan::gxf::GXFComponent::gxf_entity_group_id()

The group id of the entity group this component belongs to.

gxf_cptr

void * holoscan::gxf::GXFComponent::gxf_cptr()

gxf_component

nvidia::gxf::Handle<nvidia::gxf::Component> holoscan::gxf::GXFComponent::gxf_component()

gxf_initialize

void holoscan::gxf::GXFComponent::gxf_initialize()

set_gxf_parameter

void holoscan::gxf::GXFComponent::set_gxf_parameter(
    const std::string &component_name,
    const std::string &key,
    ParameterWrapper &param_wrap
)

Set a given parameter on the underlying GXF component.

reset_gxf_graph_entity

void holoscan::gxf::GXFComponent::reset_gxf_graph_entity()

add_to_graph_entity

Overload 1

Overload 2

virtual

virtual void holoscan::gxf::GXFResource::add_to_graph_entity(
    Operator *op
)

to_yaml_node

YAML::Node holoscan::gxf::GXFResource::to_yaml_node() const override

Get a YAML representation of the resource.

Returns: YAML node including type and specs of the resource in addition to the base component properties.

set_parameters

void holoscan::gxf::GXFResource::set_parameters() override

This method is invoked by GXFResource::initialize().

By overriding this method, we can modify how GXF Codelet’s parameters are set from the arguments.

handle_dev_id

bool holoscan::gxf::GXFResource::handle_dev_id(
    std::optional<int32_t> &dev_id_value
)

reset_backend_objects

void holoscan::gxf::GXFResource::reset_backend_objects() override

Reset any backend-specific state.

update_params_from_args

Update parameters based on the specified arguments

Update parameters based on the specified arguments (with params)

void holoscan::Resource::update_params_from_args()

Update parameters based on the specified arguments.

service_provider

void holoscan::ComponentBase::service_provider(
    FragmentServiceProvider *provider
)

Set the service provider that owns this component.

Static methods

register_converter

template <typename typeT>
static void holoscan::ComponentBase::register_converter()

If an operator or resource has an argument with a custom type, the argument setter must be registered using this method.

The argument setter is used to set the value of the argument from the YAML configuration.

This method can be called in the initialization phase of the operator/resource (e.g., initialize()). The example below shows how to register the argument setter for the custom type (Vec3):

It is assumed that YAML::convert<T>::encode and YAML::convert<T>::decode are implemented for the given type. You need to specialize the YAML::convert<> template class.

For example, suppose that you had a Vec3 class with the following members:

You can define the YAML::convert<Vec3> as follows in a ‘.cpp’ file:

Please refer to the yaml-cpp documentation for more details.

Template parameters

typeT

typename

The type of the argument to register.

Example

void MyOp::initialize() {
  register_converter<Vec3>();
}

Example

struct Vec3 {
  // make sure you have overloaded operator==() for the comparison
  double x, y, z;
};

Example

namespace YAML {
template<>
struct convert<Vec3> {
  static Node encode(const Vec3& rhs) {
    Node node;
    node.push_back(rhs.x);
    node.push_back(rhs.y);
    node.push_back(rhs.z);
    return node;
  }
  static bool decode(const Node& node, Vec3& rhs) {
    if(!node.IsSequence() || node.size() != 3) {
      return false;
    }
    rhs.x = node[0].as<double>();
    rhs.y = node[1].as<double>();
    rhs.z = node[2].as<double>();
    return true;
  }
};
}

register_argument_setter

template <typename typeT>
void holoscan::ComponentBase::register_argument_setter()

Please refer to the documentation of register_converter() for more details.

Template parameters

typeT

typename

The type of the argument to register.

Types

ResourceType

Resource type used for the initialization of the resource.

Name	Value	Description
`kNative`		Native resource.
`kGXF`		GXF resource.

Member variables

Name	Type	Description
`release_threshold_`	`Parameter< std::string >`
`device_memory_initial_size_`	`Parameter< std::string >`
`device_memory_max_size_`	`Parameter< std::string >`
`dev_id_`	`Parameter< int32_t >`
`gxf_typename_`	`std::string`	The GXF type name (used for GXFComponentResource).
`resource_type_`	`ResourceType`	The type of the resource.
`is_initialized_`	`bool`	Whether the resource is initialized.
`spec_`	`std::shared_ptr< ComponentSpec >`	The component specification.
`id_`	`int64_t`	The ID of the component.
`name_`	`std::string`	Name of the component.
`fragment_`	`Fragment *`	Pointer to the fragment that owns this component.
`args_`	`std::vector< Arg >`	List of arguments.
`service_provider_`	`FragmentServiceProvider *`	Pointer to the service provider.
`gxf_context_`	`gxf_context_t`
`gxf_eid_`	`gxf_uid_t`
`gxf_tid_`	`gxf_tid_t`
`gxf_cid_`	`gxf_uid_t`
`gxf_graph_entity_`	`std::shared_ptr< nvidia::gxf::GraphEntity >`
`gxf_cname_`	`std::string`
`gxf_component_`	`nvidia::gxf::Handle< nvidia::gxf::Component >`
`gxf_cptr_`	`void *`