Class AsyncDataLoggerResource

• Defined in File async_data_logger.hpp

Base Type

• public holoscan::DataLoggerResource (Class DataLoggerResource)

Derived Type

• public holoscan::data_loggers::AsyncConsoleLogger (Class AsyncConsoleLogger)

class AsyncDataLoggerResource : public holoscan::DataLoggerResource

Asynchronous data logger.

Maintains a queue of items to be logged that are processed by a background thread.

The log_data method is used to send data entries to the primary data queue and is intended to be used to log most data types (e.g. strings, numeric types, small structs, etc.).

This logger can be operated in single queue or dual queue modes.

When the enable_large_data_queue parameter is true, a separate queue will be available for “large” data (e.g. Tensor and TensorMap data). This large data queue is processed by a separate worker thread. If log_tensor_data_contents is true, it is expected that AsyncDataLoggerBackend::process_large_entry would handle logging the actual tensor contents. The AsyncDataLoggerBackend::process_entry method corresponding to the primary queue would typically be designed to log only generic tensor attributes such as shape and dtype.

The dual queue design allows for prioritized processing and selective dropping of large data contents while preserving important metadata if the large data queue becomes full. It is the responsibility of the backend (AsyncDataLoggerBackend) to determine which data types to log to which queue.

When enable_large_data_queue is false, “large” data is sent to the primary queue instead.

Both queues should handle logging the MetadataDictionary when Holoscan’s metadata feature is enabled.

Subclassed by holoscan::data_loggers::AsyncConsoleLogger

Public Functions

HOLOSCAN_RESOURCE_FORWARD_ARGS_SUPER (AsyncDataLoggerResource, DataLoggerResource) AsyncDataLoggerResource()=default

~AsyncDataLoggerResource() override

AsyncDataLoggerResource(const AsyncDataLoggerResource&) = delete

AsyncDataLoggerResource &operator=(const AsyncDataLoggerResource&) = delete

AsyncDataLoggerResource(AsyncDataLoggerResource&&) = delete

AsyncDataLoggerResource &operator=(AsyncDataLoggerResource&&) = delete

virtual void setup(ComponentSpec &spec) override

Define the resource specification.

Parameters

spec – The reference to the component specification.

virtual void initialize() override

Initialize the component.

This method is called only once when the component is created for the first time, and use of light-weight initialization.

virtual bool log_data(const std::any &data, const std::string &unique_id, int64_t acquisition_timestamp = -1, const std::shared_ptr<MetadataDictionary> &metadata = nullptr, IOSpec::IOType io_type = IOSpec::IOType::kOutput) override

Logs a message.

The unique_id for the message will have the form:

• operator_name.port_name

• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name

• fragment_name.operator_name.port_name:index

Parameters

• data – The data to log, passed as std::any.

• unique_id – A unique identifier for the message.

• acquisition_timestamp – Timestamp when the data was acquired (-1 if unknown).

• metadata – Associated metadata dictionary for the message.

• io_type – The type of I/O port (kInput or kOutput).

Returns

true if logging (including serialization and sending) was successful, false otherwise.

virtual bool log_tensor_data(const std::shared_ptr<Tensor> &tensor, const std::string &unique_id, int64_t acquisition_timestamp = -1, const std::shared_ptr<MetadataDictionary> &metadata = nullptr, IOSpec::IOType io_type = IOSpec::IOType::kOutput) override

Logs a Tensor with optional data content logging.

This specialized method allows efficient logging of tensor metadata without the overhead of logging large tensor data arrays when only header information is needed.

The unique_id for the message will have the form:

• operator_name.port_name

• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name

• fragment_name.operator_name.port_name:index

Parameters

• tensor – The Tensor to log.

• unique_id – A unique identifier for the message.

• acquisition_timestamp – Timestamp when the data was acquired (-1 if unknown).

• metadata – Associated metadata dictionary for the message.

• io_type – The type of I/O port (kInput or kOutput).

Returns

true if logging was successful, false otherwise.

virtual bool log_tensormap_data(const TensorMap &tensor_map, const std::string &unique_id, int64_t acquisition_timestamp = -1, const std::shared_ptr<MetadataDictionary> &metadata = nullptr, IOSpec::IOType io_type = IOSpec::IOType::kOutput) override

Logs a TensorMap with optional data content logging.

This specialized method allows efficient logging of tensor map metadata without the overhead of logging large tensor data arrays when only header information is needed.

The unique_id for the message will have the form:

• operator_name.port_name

• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name

• fragment_name.operator_name.port_name:index

Parameters

• tensor_map – The TensorMap to log.

• unique_id – A unique identifier for the message.

• acquisition_timestamp – Timestamp when the data was acquired (-1 if unknown).

• metadata – Associated metadata dictionary for the message.

• io_type – The type of I/O port (kInput or kOutput).

Returns

true if logging was successful, false otherwise.

inline virtual bool log_backend_specific(const std::any &data, const std::string &unique_id, int64_t acquisition_timestamp = -1, const std::shared_ptr<MetadataDictionary> &metadata = nullptr, IOSpec::IOType io_type = IOSpec::IOType::kOutput) override

Logs backend-specific data types.

This method is called for logging backend-specific data types (intended for use with backends that have separate emit/receive codepaths for backend-specific types). The data parameter is kept as std::any here to avoid making the base interface specific to a particular backend, but a backend-specific concrete implementation should be provided as needed via run-time type checking.

A concrete example of a backend-specific type is the GXF Entity type which is a heterogeneous collection of components. An implementation of this method for GXF entities is provided in the concrete implementation of the GXFConsoleLogger.

The unique_id for the message will have the form:

• operator_name.port_name

• operator_name.port_name:index (for multi-receivers with N:1 connection)

For distributed applications, the fragment name will also appear in the unique id:

• fragment_name.operator_name.port_name

• fragment_name.operator_name.port_name:index

Parameters

• data – The backend-specific data to log, passed as std::any.

• unique_id – A unique identifier for the message.

• acquisition_timestamp – Timestamp when the data was acquired (-1 if unknown).

• metadata – Associated metadata dictionary for the message.

• io_type – The type of I/O port (kInput or kOutput).

Returns

true if logging was successful, false if backend-specific logging is not supported.

virtual void shutdown() override

Shutdown the data logger.

This method should be called to properly shutdown the data logger, including stopping any background threads and releasing resources. The default implementation does nothing. Data loggers that use background threads or other resources should override this method to perform proper cleanup.

void set_backend(std::shared_ptr<AsyncDataLoggerBackend> backend)

std::string get_statistics() const

inline size_t get_data_dropped_count() const

inline size_t get_large_data_dropped_count() const

size_t get_data_queue_size() const

size_t get_large_data_queue_size() const

Protected Functions

bool start_worker_threads()

void stop_worker_threads()

void data_worker_function()

void large_data_worker_function()

bool enqueue_data_entry(DataEntry &&entry)

bool enqueue_large_data_entry(DataEntry &&entry)

template<typename ArgT>
inline ArgT copy_value_from_args(const std::string &arg_name, ArgT default_value)

Helper function to extract typed values from component arguments.

This function handles both direct values and YAML node values, providing robust parameter extraction with fallback to default values.

Note

For a concrete example, see how this function is used in AsyncConsoleLogger.

Template Parameters

ArgT – The type of the argument to extract

Parameters

• arg_name – The name of the argument to look for

• default_value – The default value to return if the argument is not found or cannot be parsed

Returns

The extracted value or the default value