holoscan::AsyncDataLoggerResource

Asynchronous data logger.

Maintains a queue of items to be logged that are processed by a background thread.

The log_data method is used to send data entries to the primary data queue and is intended to be used to log most data types (e.g. strings, numeric types, small structs, etc.).

This logger can be operated in single queue or dual queue modes.

When the enable_large_data_queue parameter is true, a separate queue will be available for “large” data (e.g. Tensor and TensorMap data). This large data queue is processed by a separate worker thread. If log_tensor_data_contents is true, it is expected that AsyncDataLoggerBackend::process_large_entry would handle logging the actual tensor contents. The AsyncDataLoggerBackend::process_entry method corresponding to the primary queue would typically be designed to log only generic tensor attributes such as shape and dtype.

The dual queue design allows for prioritized processing and selective dropping of large data contents while preserving important metadata if the large data queue becomes full. It is the responsibility of the backend (AsyncDataLoggerBackend) to determine which data types to log to which queue.

When enable_large_data_queue is false, “large” data is sent to the primary queue instead.

Both queues should handle logging the MetadataDictionary when Holoscan’s metadata feature is enabled.

The shutdown_wait_period_ms parameter controls how long the logger waits for remaining messages in the queue(s) to be processed during shutdown. A negative value (default) means wait indefinitely, 0 means don’t wait at all, and a positive value specifies the timeout in milliseconds. The HOLOSCAN_ASYNC_LOGGER_SHUTDOWN_WAIT_MS environment variable can be used to override this value. During shutdown, this resource emits INFO logs that include the component name, approximate queue depths, entries processed so far, and a final summary when worker threads have finished; WARN is used if a drain timeout expires and entries are discarded.

Inherited parameters from DataLoggerResource:

• log_inputs: bool (optional, default: true)
• log_outputs: bool (optional, default: true)
• log_metadata: bool (optional, default: true)
• log_tensor_data_content: bool (optional, default: false)
• use_scheduler_clock: bool (optional, default: false)
• allowlist_patterns: std::vector<std::string> (optional, default: empty)
• denylist_patterns: std::vector<std::string> (optional, default: empty)

See the DataLoggerResource documentation for details on these inherited parameters.

#include <holoscan/async_data_logger.hpp>

Inherits from: holoscan::DataLoggerResource (public)

Constructors

AsyncDataLoggerResource

holoscan::AsyncDataLoggerResource::AsyncDataLoggerResource() = defaultholoscan::AsyncDataLoggerResource::AsyncDataLoggerResource() = default

Destructor

~AsyncDataLoggerResource

holoscan::AsyncDataLoggerResource::~AsyncDataLoggerResource() overrideholoscan::AsyncDataLoggerResource::~AsyncDataLoggerResource() override

Assignment operators

operator=

AsyncDataLoggerResource & holoscan::AsyncDataLoggerResource::operator=(const AsyncDataLoggerResource &) = delete;AsyncDataLoggerResource & holoscan::AsyncDataLoggerResource::operator=(const AsyncDataLoggerResource &) = delete;AsyncDataLoggerResource & holoscan::AsyncDataLoggerResource::operator=(const AsyncDataLoggerResource &) = delete;
AsyncDataLoggerResource & holoscan::AsyncDataLoggerResource::operator=(AsyncDataLoggerResource &&) = delete;AsyncDataLoggerResource & holoscan::AsyncDataLoggerResource::operator=(AsyncDataLoggerResource &&) = delete;AsyncDataLoggerResource & holoscan::AsyncDataLoggerResource::operator=(AsyncDataLoggerResource &&) = delete;

Methods

setup

void holoscan::AsyncDataLoggerResource::setup(
    ComponentSpec &spec
) override

Define the resource specification.

Parameters

initialize

void holoscan::AsyncDataLoggerResource::initialize() override

Initialize the component.

This method is called only once when the component is created for the first time, and use of light-weight initialization.

log_data

bool holoscan::AsyncDataLoggerResource::log_data(
    const std::any &data,
    const std::string &unique_id,
    int64_t acquisition_timestamp = -1,
    const std::shared_ptr<MetadataDictionary> &metadata = nullptr,
    IOSpec::IOType io_type = IOSpec::IOType::kOutput,
    std::optional<cudaStream_t> stream = std::nullopt
) override

Logs a message.

The unique_id for the message will have the form:

• operator_name.port_name
• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name
• fragment_name.operator_name.port_name:index

Returns: true if logging (including serialization and sending) was successful, false otherwise.

Parameters

log_tensor_data

bool holoscan::AsyncDataLoggerResource::log_tensor_data(
    const std::shared_ptr<Tensor> &tensor,
    const std::string &unique_id,
    int64_t acquisition_timestamp = -1,
    const std::shared_ptr<MetadataDictionary> &metadata = nullptr,
    IOSpec::IOType io_type = IOSpec::IOType::kOutput,
    std::optional<cudaStream_t> stream = std::nullopt
) override

Logs a Tensor with optional data content logging.

This specialized method allows efficient logging of tensor metadata without the overhead of logging large tensor data arrays when only header information is needed.

The unique_id for the message will have the form:

• operator_name.port_name
• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name
• fragment_name.operator_name.port_name:index

Returns: true if logging was successful, false otherwise.

Parameters

log_tensormap_data

bool holoscan::AsyncDataLoggerResource::log_tensormap_data(
    const TensorMap &tensor_map,
    const std::string &unique_id,
    int64_t acquisition_timestamp = -1,
    const std::shared_ptr<MetadataDictionary> &metadata = nullptr,
    IOSpec::IOType io_type = IOSpec::IOType::kOutput,
    std::optional<cudaStream_t> stream = std::nullopt
) override

Logs a TensorMap with optional data content logging.

This specialized method allows efficient logging of tensor map metadata without the overhead of logging large tensor data arrays when only header information is needed.

The unique_id for the message will have the form:

• operator_name.port_name
• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name
• fragment_name.operator_name.port_name:index

Returns: true if logging was successful, false otherwise.

Parameters

log_backend_specific

bool holoscan::AsyncDataLoggerResource::log_backend_specific(
    const std::any &data,
    const std::string &unique_id,
    int64_t acquisition_timestamp = -1,
    const std::shared_ptr<MetadataDictionary> &metadata = nullptr,
    IOSpec::IOType io_type = IOSpec::IOType::kOutput,
    std::optional<cudaStream_t> stream = std::nullopt
) override

Logs backend-specific data types.

This method is called for logging backend-specific data types (intended for use with backends that have separate emit/receive codepaths for backend-specific types). The data parameter is kept as std::any here to avoid making the base interface specific to a particular backend, but a backend-specific concrete implementation should be provided as needed via run-time type checking.

A concrete example of a backend-specific type is the GXF Entity type which is a heterogeneous collection of components. An implementation of this method for GXF entities is provided in the concrete implementation of the GXFConsoleLogger.

The unique_id for the message will have the form:

• operator_name.port_name
• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name
• fragment_name.operator_name.port_name:index

Returns: true if logging was successful, false if backend-specific logging is not supported.

Parameters

shutdown

void holoscan::AsyncDataLoggerResource::shutdown() override

Shutdown the data logger.

This method should be called to properly shutdown the data logger, including stopping any background threads and releasing resources. The default implementation does nothing. Data loggers that use background threads or other resources should override this method to perform proper cleanup.

set_backend

void holoscan::AsyncDataLoggerResource::set_backend(
    std::shared_ptr<AsyncDataLoggerBackend> backend
)

get_statistics

std::string holoscan::AsyncDataLoggerResource::get_statistics() const

get_data_dropped_count

size_t holoscan::AsyncDataLoggerResource::get_data_dropped_count() const

get_large_data_dropped_count

size_t holoscan::AsyncDataLoggerResource::get_large_data_dropped_count() const

get_data_queue_size

size_t holoscan::AsyncDataLoggerResource::get_data_queue_size() const

get_large_data_queue_size

size_t holoscan::AsyncDataLoggerResource::get_large_data_queue_size() const

should_log_message

bool holoscan::AsyncDataLoggerResource::should_log_message(
    const std::string &unique_id
) const

Checks if a message with the given unique_id should be logged based on allowlist/denylist patterns.

This utility function implements the filtering logic:

• First check if denylist patterns are specified and if there is a match, do not log it.
• Next check if allowlist_patterns were specified: If no, return true (allow everything) If yes, return true only if there is a match to the specified patterns.
  • If no, return true (allow everything)
  • If yes, return true only if there is a match to the specified patterns.

Returns: true if the message should be logged, false otherwise.

Parameters

should_log_output

bool holoscan::AsyncDataLoggerResource::should_log_output() const override

Checks if the logger should log output ports.

If False, the data logger will not be applied during op_input.emit() calls from Operator::compute.

Returns: true if the logger should log output ports, false otherwise.

should_log_input

bool holoscan::AsyncDataLoggerResource::should_log_input() const override

Checks if the logger should log input ports.

If False, the data logger will not be applied during op_input.receive() calls from Operator::compute.

Returns: true if the logger should log input ports, false otherwise.

should_log_metadata

bool holoscan::AsyncDataLoggerResource::should_log_metadata() const

Checks if the logger should log metadata.

If False, the data logger will not log metadata for each operator.

Returns: true if the logger should log metadata, false otherwise.

should_log_tensor_data_content

bool holoscan::AsyncDataLoggerResource::should_log_tensor_data_content() const

Checks if the logger should log tensor data content.

If False, only tensor header information will be logged, not the actual data arrays. When true, the full tensor data is also logged.

Returns: true if the logger should log tensor data content, false otherwise.

get_timestamp

virtual int64_t holoscan::AsyncDataLoggerResource::get_timestamp() const

Get the current timestamp for logging operations.

This method is called internally by the logging functions to obtain timestamps for emit_timestamp (when io_type==IOSpec::IOType::kOutput) or receive_timestamp (when io_type==IOSpec::IOType::kInput). The default implementation provides high-resolution timestamps in microseconds since epoch. Implementations can override this to provide custom timing mechanisms as appropriate.

Returns: Current timestamp in microseconds since epoch, or -1 if not available.

resource_type

ResourceType holoscan::Resource::resource_type() const

Get the resource type.

Returns: The resource type.

name

Resource & holoscan::Resource::name(Resource & holoscan::Resource::name(
    const std::string &name
) &

fragment

Resource & holoscan::Resource::fragment(Resource & holoscan::Resource::fragment(
    Fragment *fragment
)

spec

Resource & holoscan::Resource::spec(Resource & holoscan::Resource::spec(
    const std::shared_ptr<ComponentSpec> &spec
)

spec_shared

std::shared_ptr<ComponentSpec> holoscan::Resource::spec_shared()

Get the shared pointer to the component spec.

Returns: The shared pointer to the component spec.

to_yaml_node

YAML::Node holoscan::Resource::to_yaml_node() const override

Get a YAML representation of the resource.

Returns: YAML node including spec of the resource in addition to the base component properties.

set_parameters

void holoscan::Resource::set_parameters() override

Set the parameters based on defaults (sets GXF parameters for GXF components).

id

int64_t holoscan::ComponentBase::id() const

Get the identifier of the component.

By default, the identifier is set to -1. It is set to a valid value when the component is initialized.

With the default executor (GXFExecutor), the identifier is set to the GXF component ID.

Returns: The identifier of the component.

add_arg

void holoscan::ComponentBase::add_arg(
    const Arg &arg
)

args

std::vector<Arg> & holoscan::ComponentBase::args()

Get the list of arguments.

Returns: The vector of arguments.

description

std::string holoscan::ComponentBase::description() const

Get a description of the component.

Returns: YAML string.

See also: to_yaml_node()

service

template <typename ServiceT = DefaultFragmentService>
std::shared_ptr<ServiceT> holoscan::ComponentBase::service(
    std::string_view id = ""
) const

Retrieve a registered fragment service or resource.

Retrieves a previously registered fragment service or resource by its type and optional identifier. Returns nullptr if no service/resource is found with the specified type and identifier.

Note that any changes to the service retrieval logic in this method should be synchronized with the implementation in Fragment::service() method to maintain consistency.

Returns: The shared pointer to the service/resource, or nullptr if not found or if type casting fails.

Template parameters

Parameters

get_service_by_type_info

std::shared_ptr<FragmentService> holoscan::ComponentBase::get_service_by_type_info(
    const std::type_info &service_type,
    std::string_view id = ""
) const

Retrieve a registered fragment service or resource for Python bindings.

This is a helper method for Python bindings to retrieve a service by its C++ type info.

Returns: The shared pointer to the base service, or nullptr if not found.

Parameters

reset_backend_objects

virtual void holoscan::ComponentBase::reset_backend_objects()

Reset any backend-specific objects (e.g. GXF GraphEntity).

start_worker_threads

bool holoscan::AsyncDataLoggerResource::start_worker_threads()

stop_worker_threads

void holoscan::AsyncDataLoggerResource::stop_worker_threads()

data_worker_function

void holoscan::AsyncDataLoggerResource::data_worker_function()

large_data_worker_function

void holoscan::AsyncDataLoggerResource::large_data_worker_function()

enqueue_data_entry

bool holoscan::AsyncDataLoggerResource::enqueue_data_entry(
    DataEntry &&entry
)

enqueue_large_data_entry

bool holoscan::AsyncDataLoggerResource::enqueue_large_data_entry(
    DataEntry &&entry
)

copy_value_from_args

template <typename ArgT>
ArgT holoscan::AsyncDataLoggerResource::copy_value_from_args(
    const std::string &arg_name,
    ArgT default_value
)

Helper function to extract typed values from component arguments.

This function handles both direct values and YAML node values, providing robust parameter extraction with fallback to default values.

Returns: The extracted value or the default value

Template parameters

Parameters

update_params_from_args

void holoscan::Resource::update_params_from_args()

service_provider

void holoscan::ComponentBase::service_provider(
    FragmentServiceProvider *provider
)

Set the service provider that owns this component.

Static methods

register_converter

template <typename typeT>
static void holoscan::ComponentBase::register_converter()

Register the argument setter for the given type.

If an operator or resource has an argument with a custom type, the argument setter must be registered using this method.

The argument setter is used to set the value of the argument from the YAML configuration.

This method can be called in the initialization phase of the operator/resource (e.g., initialize()). The example below shows how to register the argument setter for the custom type (Vec3):

It is assumed that YAML::convert<T>::encode and YAML::convert<T>::decode are implemented for the given type. You need to specialize the YAML::convert<> template class.

For example, suppose that you had a Vec3 class with the following members:

You can define the YAML::convert<Vec3> as follows in a ‘.cpp’ file:

Please refer to the yaml-cpp documentation for more details.

Template parameters

Example

void MyOp::initialize() {
  register_converter<Vec3>();
}

Example

struct Vec3 {
  // make sure you have overloaded operator==() for the comparison
  double x, y, z;
};

Example

namespace YAML {
template<>
struct convert<Vec3> {
  static Node encode(const Vec3& rhs) {
    Node node;
    node.push_back(rhs.x);
    node.push_back(rhs.y);
    node.push_back(rhs.z);
    return node;
  }
  static bool decode(const Node& node, Vec3& rhs) {
    if(!node.IsSequence() || node.size() != 3) {
      return false;
    }
    rhs.x = node[0].as<double>();
    rhs.y = node[1].as<double>();
    rhs.z = node[2].as<double>();
    return true;
  }
};
}

register_argument_setter

template <typename typeT>
void holoscan::ComponentBase::register_argument_setter()

Register the argument setter for the given type.

Please refer to the documentation of register_converter() for more details.

Template parameters

Types

ResourceType

Resource type used for the initialization of the resource.

Member variables