Data Logging

Holoscan provides a flexible data logging system that allows applications to capture and log data flowing through operator workflows. This system is designed to help with debugging, monitoring, and analyzing the behavior of Holoscan applications.

Overview

The data logging system consists of several key components:

• DataLogger (holoscan::DataLogger) - The core interface that defines methods for logging different data types
• DataLoggerResource (holoscan::DataLoggerResource) - A resource-based implementation with common configuration parameters
• BasicConsoleLogger (holoscan::data_loggers::BasicConsoleLogger) - A concrete implementation that logs data to the console
• GXFConsoleLogger (holoscan::data_loggers::GXFConsoleLogger) - A version of BasicConsoleLogger which also logs Tensors and/or nvidia::gxf::VideoBuffer objects within messages that are emitted or received as a holoscan::gxf::Entity or nvidia::gxf::Entity. It is recommended to use this version over BasicConsoleLogger as it will also log the tensors emitted or received by a number of the built-in operators of the SDK which directly use GXF::Entity type to support both Tensors as well as GXF VideoBuffers. VideoBuffer components are not currently logged by this implementation, but this could be added by overriding the log_backend_specific method.
• AsyncConsoleLogger (holoscan::data_loggers::AsyncConsoleLogger) - A version of GXFConsoleLogger which performs logging asynchronously by a queue that is processed by a worker thread owned by the logger. This version has the option of using separate logging queues for “large” data (e.g. Tensor/TensorMap) vs. other types so that data contents can be dropped if the queue size reaches a user-specfied limit.
• SimpleTextSerializer (holoscan::data_loggers::SimpleTextSerializer) - A serializer for converting data to human-readable text

When Logging Occurs

Data logging occurs automatically during the execution of operator workflows:

• Input logging: When operators call receive() methods on their input ports
• Output logging: When operators call emit() methods on their output ports

The timing of logging corresponds exactly to these receive and emit calls within each operator’s compute() method.

Data Types Supported

The data logging system provides specialized methods for different data types:

• log_data() - For general data types (passed as std::any)
• log_tensor_data() - For Tensor objects with optional data content logging
• log_tensormap_data() - For TensorMap objects with optional data content logging
• log_backend_specific() - An optional method that can be used to log backend-specific data types. Currently GXF is the only supported backend for Holscan SDK and this method is called by GXFInputContext::receive<T>() or GXFOutputContext::emit(data) when the data type, T, is holoscan::gxf::Entity or nvidia::gxf::Entity.

DataLogger Interface

The DataLogger interface defines the contract that all data loggers must implement:

1
  class DataLogger {
2
   public:
3
    // Log general data types
4
    virtual bool log_data(std::any data, const std::string& unique_id,
5
                          int64_t acquisition_timestamp = -1,
6
                          std::shared_ptr<MetadataDictionary> metadata = nullptr,
7
                          IOSpec::IOType io_type = IOSpec::IOType::kOutput) = 0;
8
   
9
    // Log Tensor data
10
    virtual bool log_tensor_data(const std::shared_ptr<Tensor>& tensor,
11
                                 const std::string& unique_id,
12
                                 int64_t acquisition_timestamp = -1,
13
                                 const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
14
                                 IOSpec::IOType io_type = IOSpec::IOType::kOutput) = 0;
15
   
16
    // Log TensorMap data
17
    virtual bool log_tensormap_data(const TensorMap& tensor_map,
18
                                    const std::string& unique_id,
19
                                    int64_t acquisition_timestamp = -1,
20
                                    const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
21
                                    IOSpec::IOType io_type = IOSpec::IOType::kOutput) = 0;
22
   
23
    // Configuration methods
24
    virtual bool should_log_output() const = 0;
25
    virtual bool should_log_input() const = 0;
26
  };

DataLoggerResource Base Class

The DataLoggerResource class provides a convenient base implementation with common configuration parameters:

Configuration Parameters

• log_inputs (bool, default: true) - Whether to log data received on input ports
• log_outputs (bool, default: true) - Whether to log data emitted on output ports
• log_metadata (bool, default: true) - Whether to include metadata in logs
• log_tensor_data_content (bool, default: false) - Whether to log actual tensor data arrays (if false, only header info is logged)
• allowlist_patterns (vector<string>, default: empty) - Regex patterns for message IDs to always log
• denylist_patterns (vector<string>, default: empty) - Regex patterns for message IDs to never log

Also note that Holoscan resources can have other Resource classes as a parameter as demonstrated by having a separate SimpleTextSerializer resource in the concrete BasicConsoleLogger class that inherits from DataLoggerResource. Note that when implementing a class that inherits from this one, it is mandatory to call the DataLoggerResource::setup method within the derived class’s setup method as in this example from BasicConsoleLogger. If the initialize method is overridden the parent class’s initialize method should also be called. This is an example from BasicConsoleLogger:

1
 
2
void BasicConsoleLogger::setup(ComponentSpec& spec) {
3
  spec.param(serializer_, "serializer", "Serializer", "Serializer to use for logging data");
4
  // setup the parameters present on the base DataLoggerResource
5
  DataLoggerResource::setup(spec);
6
}
7
 
8
void BasicConsoleLogger::initialize() {
9
  // Find if there is an argument for 'serializer'
10
  auto has_serializer = std::find_if(
11
      args().begin(), args().end(), [](const auto& arg) { return (arg.name() == "serializer"); });
12
 
13
  // Create appropriate serializer if none was provided
14
  if (has_serializer == args().end()) {
15
    add_arg(Arg("serializer", fragment()->make_resource<SimpleTextSerializer>("serializer")));
16
  }
17
 
18
  // call parent initialize after adding missing serializer arg above
19
  DataLoggerResource::initialize();
20
}
21
 

BasicConsoleLogger Example

The BasicConsoleLogger and GXFConsoleLogger are concrete implementations that output logs to the console. For existing Holoscan apps which always use the GXF-based backend, GXFConsoleLogger should be preferred as it also implements logging of Tensor objects present within data emitted or received as a holoscan::gxf::Entity or nvidia::gxf::Entity.

1
  #include <holoscan/data_loggers/basic_console_logger/basic_console_logger.hpp>
2
  #include <holoscan/data_loggers/basic_console_logger/simple_text_serializer.hpp>
3
   
4
  class MyApp : public holoscan::Application {
5
   public:
6
    void compose() override {
7
      // Create operators (example)
8
      auto source = make_operator<SourceOp>("source");
9
      auto processor = make_operator<ProcessorOp>("processor");
10
      auto sink = make_operator<SinkOp>("sink");
11
   
12
      // Create and configure data logger
13
      auto logger = make_resource&lt;holoscan::data_loggers::GXFConsoleLogger&gt;(
14
          "console_logger",
15
          Arg("log_inputs", true),
16
          Arg("log_outputs", true),
17
          Arg("log_metadata", false),
18
          Arg("log_tensor_data_content", false),
19
          Arg("denylist_patterns", std::vector&lt;std::string&gt;{".*debug.*"})
20
      );
21
   
22
      // Add logger to application
23
      add_data_logger(logger);
24
   
25
      // Define workflow
26
      add_flow(source, processor);
27
      add_flow(processor, sink);
28
    }
29
  };

The example above shows example code adding the logger within the compose method, but it can also be added from the main application file via as done in the following example applications:

1. Tensor Interop (C++, Python)
2. Multithread Scheduling (C++, Python)
3. Video Replayer (C++, Python)

As with any other resource or operator in the SDK, parameters can be passed in directly via arguments or indirectly via reading from the YAML config.

YAML Configuration

Data loggers can be configured using YAML configuration files, making it easy to adjust logging behavior without recompiling:

YAML Configuration Example

1
# Data logging configuration
2
data_logging: true
3
 
4
basic_console_logger:
5
  log_inputs: true
6
  log_outputs: true
7
  log_metadata: true
8
  log_tensor_data_content: false
9
  allowlist_patterns: []
10
  denylist_patterns:
11
    - ".*debug.*"
12
    - ".*internal.*"
13
 
14
# Optional: Configure the text serializer
15
simple_text_serializer:
16
  max_elements: 10
17
  max_metadata_items: 5
18
  log_python_object_contents: true  # Python only

Loading Configuration from YAML

1
  class MyApp : public holoscan::Application {
2
   public:
3
    void compose() override {
4
      // Create operators
5
      auto source = make_operator<SourceOp>("source");
6
      auto processor = make_operator<ProcessorOp>("processor");
7
      auto sink = make_operator<SinkOp>("sink");
8
   
9
      // Check if data logging is enabled
10
      auto enable_data_logging = from_config("data_logging").as<bool>();
11
      if (enable_data_logging) {
12
        auto text_serializer = make_resource&lt;holoscan::data_loggers::SimpleTextSerializer&gt;(
13
          "text-serializer",
14
          from_config("simple_text_serializer")
15
        )
16
   
17
        // Create logger with YAML configuration
18
        auto logger = make_resource&lt;holoscan::data_loggers::GXFConsoleLogger&gt;(
19
            "console_logger",
20
            holoscan::Arg("serializer", text_serializer),
21
            from_config("basic_console_logger")
22
        );
23
        add_data_logger(logger);
24
      }
25
   
26
      // Define workflow
27
      add_flow(source, processor);
28
      add_flow(processor, sink);
29
    }
30
  };
31
   
32
  int main(int argc, char** argv) {
33
    auto app = holoscan::make_application<MyApp>();
34
   
35
    // Load configuration
36
    app->config("path/to/config.yaml");
37
   
38
    app->run();
39
    return 0;
40
  }

Custom Data Logger Implementation

You can create custom data loggers by implementing the DataLogger interface. To be able to use Holoscan Parameters and configure them via YAML it may be useful to inherit from the provided DataLoggerResource (as done for GXFConsoleLogger). Note that it is not required to inherit from DataLoggerResource, though, only the DataLogger interface.

1
  #include <holoscan/core/resources/data_logger.hpp>
2
   
3
  class MyCustomLogger : public holoscan::DataLoggerResource {
4
   public:
5
    HOLOSCAN_RESOURCE_FORWARD_ARGS_SUPER(MyCustomLogger, DataLoggerResource)
6
   
7
    void setup(ComponentSpec& spec) override {
8
      // Add custom parameters
9
      spec.param(output_file_, "output_file", "Output File",
10
                 "Path to output log file", std::string(""));
11
   
12
      // Call parent setup for common parameters
13
      DataLoggerResource::setup(spec);
14
    }
15
   
16
    bool log_data(std::any data, const std::string& unique_id,
17
                  int64_t acquisition_timestamp = -1,
18
                  std::shared_ptr<MetadataDictionary> metadata = nullptr,
19
                  IOSpec::IOType io_type = IOSpec::IOType::kOutput) override {
20
      // Implement custom logging logic
21
      return true;
22
    }
23
   
24
    bool log_tensor_data(const std::shared_ptr<Tensor>& tensor,
25
                         const std::string& unique_id,
26
                         int64_t acquisition_timestamp = -1,
27
                         const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
28
                         IOSpec::IOType io_type = IOSpec::IOType::kOutput) override {
29
      // Implement custom tensor logging
30
      return true;
31
    }
32
   
33
    bool log_tensormap_data(const TensorMap& tensor_map,
34
                            const std::string& unique_id,
35
                            int64_t acquisition_timestamp = -1,
36
                            const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
37
                            IOSpec::IOType io_type = IOSpec::IOType::kOutput) override {
38
      // Implement custom tensor map logging
39
      return true;
40
    }
41
   
42
   private:
43
    Parameter&lt;std::string&gt; output_file_;
44
  };

Filtering and Pattern Matching

Data loggers inheriting from DataLoggerResource will automatically support filtering of messages using regex patterns:

1
basic_console_logger:
2
  # Only log messages from specific operators with "source" or "processor" appearing in the operator or port name
3
  allowlist_patterns:
4
    - ".*source.*"
5
    - ".*processor.*"
6
 
7
  # Exclude messages from ports with "debug" appearing in their name
8
  denylist_patterns:
9
    - ".*debug.*"

For a distributed application, the fragments will be named and the unique_id format used for each port will be:

• {fragment_name}.{operator_name}.{port_name}
• {fragment_name}.{operator_name}.{port_name}:index (for N:1 receiver ports (IOSpec::kAnySize))

For non-distributed applications, the single fragment is typically not named and the following simpler unique_id format is used:

• {operator_name}.{port_name}
• {operator_name}.{port_name}:index (for N:1 receiver ports (IOSpec::kAnySize))

The allowlist_patterns and denylist_patterns provide a way to include or exclude messages based on operator and/or port names. If there are no allowlist patterns, any messages not matching one of the denylist patterns are logged. If both types of patterns are specified, any messages that match at least one of the allowlist patterns but are not excluded by any of the denylist patterns are logged.