Data Logging - NVIDIA Docs

Holoscan provides a flexible data logging system that allows applications to capture and log data flowing through operator workflows. This system is designed to help with debugging, monitoring, and analyzing the behavior of Holoscan applications.

Overview

The data logging system consists of several key components:

DataLogger - The core interface that defines methods for logging different data types
DataLoggerResource - A resource-based implementation with common configuration parameters
BasicConsoleLogger - A concrete implementation that logs data to the console
GXFConsoleLogger - A version of BasicConsoleLogger which also logs Tensor within messages that are emitted or received as a holoscan::gxf::Entity or nvidia::gxf::Entity. It is recommended to use this version over BasicConsoleLogger as it will also log the tensors emitted or received by a number of the built-in operators of the SDK which directly use GXF::Entity type to support both Tensors as well as GXF VideoBuffers. VideoBuffer components are not currently logged by this implementation, but this could be added by overriding the log_backend_specific method.
AsyncConsoleLogger - A version of GXFConsoleLogger which performs logging asynchronously by a queue that is processed by a worker thread owned by the logger. This version has the option of using separate logging queues for “large” data (e.g. Tensor/TensorMap) vs. other types so that data contents can be dropped if the queue size reaches a user-specfied limit.
SimpleTextSerializer - A serializer for converting data to human-readable text

When Logging Occurs

Data logging occurs automatically during the execution of operator workflows:

Input logging: When operators call receive() methods on their input ports
Output logging: When operators call emit() methods on their output ports

The timing of logging corresponds exactly to these receive and emit calls within each operator’s compute() method.

Data Types Supported

The data logging system provides specialized methods for different data types:

log_data() - For general data types (passed as std::any)
log_tensor_data() - For Tensor objects with optional data content logging
log_tensormap_data() - For TensorMap objects with optional data content logging
log_backend_specific() - An optional method that can be used to log backend-specific data types. Currently GXF is the only supported backend for Holscan SDK and this method is called by GXFInputContext::receive<T>() or GXFOutputContext::emit(data) when the data type, T, is holoscan::gxf::Entity or nvidia::gxf::Entity.

DataLogger Interface

The DataLogger interface defines the contract that all data loggers must implement:

C++
Python

Copy
Copied!

            
            class DataLogger {
 public:
  // Log general data types
  virtual bool log_data(std::any data, const std::string& unique_id,
                        int64_t acquisition_timestamp = -1,
                        std::shared_ptr<MetadataDictionary> metadata = nullptr,
                        IOSpec::IOType io_type = IOSpec::IOType::kOutput) = 0;

  // Log Tensor data
  virtual bool log_tensor_data(const std::shared_ptr<Tensor>& tensor,
                               const std::string& unique_id,
                               int64_t acquisition_timestamp = -1,
                               const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
                               IOSpec::IOType io_type = IOSpec::IOType::kOutput) = 0;

  // Log TensorMap data
  virtual bool log_tensormap_data(const TensorMap& tensor_map,
                                  const std::string& unique_id,
                                  int64_t acquisition_timestamp = -1,
                                  const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
                                  IOSpec::IOType io_type = IOSpec::IOType::kOutput) = 0;

  // Configuration methods
  virtual bool should_log_output() const = 0;
  virtual bool should_log_input() const = 0;
};

Copy
Copied!

            
            class DataLogger:
    def log_data(self, data, unique_id: str, acquisition_timestamp: int = -1,
                 metadata=None, io_type=IOSpec.IOType.OUTPUT) -> bool:
        """Log general data types"""
        pass

    def log_tensor_data(self, tensor, unique_id: str, acquisition_timestamp: int = -1,
                        metadata=None, io_type=IOSpec.IOType.OUTPUT) -> bool:
        """Log Tensor data"""
        pass

    def log_tensormap_data(self, tensor_map, unique_id: str, acquisition_timestamp: int = -1,
                           metadata=None, io_type=IOSpec.IOType.OUTPUT) -> bool:
        """Log TensorMap data"""
        pass

    def should_log_output(self) -> bool:
        """Check if output ports should be logged"""
        pass

    def should_log_input(self) -> bool:
        """Check if input ports should be logged"""
        pass

DataLoggerResource Base Class

The DataLoggerResource class provides a convenient base implementation with common configuration parameters:

Configuration Parameters

log_inputs (bool, default: true) - Whether to log data received on input ports
log_outputs (bool, default: true) - Whether to log data emitted on output ports
log_metadata (bool, default: true) - Whether to include metadata in logs
log_tensor_data_content (bool, default: false) - Whether to log actual tensor data arrays (if false, only header info is logged)
allowlist_patterns (vector, default: empty) - Regex patterns for message IDs to always log
denylist_patterns (vector, default: empty) - Regex patterns for message IDs to never log

Also note that Holoscan resources can have other Resource classes as a parameter as demonstrated by having a separate SimpleTextSerializer resource in the concrete BasicConsoleLogger class that inherits from DataLoggerResource. Note that when implementing a class that inherits from this one, it is mandatory to call the DataLoggerResource::setup method within the derived class’s setup method as in this example from BasicConsoleLogger. If the initialize method is overridden the parent class’s initialize method should also be called. This is an example from BasicConsoleLogger:

Copy
Copied!

            
            void BasicConsoleLogger::setup(ComponentSpec& spec) {
  spec.param(serializer_, "serializer", "Serializer", "Serializer to use for logging data");
  // setup the parameters present on the base DataLoggerResource
  DataLoggerResource::setup(spec);
}

void BasicConsoleLogger::initialize() {
  // Find if there is an argument for 'serializer'
  auto has_serializer = std::find_if(
      args().begin(), args().end(), [](const auto& arg) { return (arg.name() == "serializer"); });

  // Create appropriate serializer if none was provided
  if (has_serializer == args().end()) {
    add_arg(Arg("serializer", fragment()->make_resource<SimpleTextSerializer>("serializer")));
  }

  // call parent initialize after adding missing serializer arg above
  DataLoggerResource::initialize();
}

Note

If allowlist_patterns is specified, only messages matching those patterns will be logged. If no allowlist is specified, all messages will be logged except those matching denylist_patterns.

Note

Currently this simple DataLoggerResource performs logging synchronously on the same thread that is executing the Operator::compute call.

In cases where logging overhead may be non-negligible (e.g. logging tensor contents to disk), the AsyncDataLoggerResource which maintains its own queue and corresponding worker thread for data logging is likely to be advantageous. For the AsyncDataLoggerResource, the thread running Operator::compute is only responsible for pushing the item to be logged onto the queue. The actual logging is handled by the logger’s own worker thread(s).

BasicConsoleLogger Example

The BasicConsoleLogger and GXFConsoleLogger are concrete implementations that output logs to the console. For existing Holoscan apps which always use the GXF-based backend, GXFConsoleLogger should be preferred as it also implements logging of Tensor objects present within data emitted or received as a holoscan::gxf::Entity or nvidia::gxf::Entity.

C++
Python

Copy
Copied!

            
            #include "holoscan/data_loggers/basic_console_logger/basic_console_logger.hpp"
#include "holoscan/data_loggers/basic_console_logger/simple_text_serializer.hpp"

class MyApp : public holoscan::Application {
 public:
  void compose() override {
    // Create operators (example)
    auto source = make_operator<SourceOp>("source");
    auto processor = make_operator<ProcessorOp>("processor");
    auto sink = make_operator<SinkOp>("sink");

    // Create and configure data logger
    auto logger = make_resource<holoscan::data_loggers::GXFConsoleLogger>(
        "console_logger",
        Arg("log_inputs", true),
        Arg("log_outputs", true),
        Arg("log_metadata", false),
        Arg("log_tensor_data_content", false),
        Arg("denylist_patterns", std::vector<std::string>{".*debug.*"})
    );

    // Add logger to application
    add_data_logger(logger);

    // Define workflow
    add_flow(source, processor);
    add_flow(processor, sink);
  }
};

Copy
Copied!

            
            from holoscan.core import Application
from holoscan.data_loggers import GXFConsoleLogger, SimpleTextSerializer

class MyApp(Application):
    def compose(self):
        # Create operators (example)
        source = SourceOp(self, name="source")
        processor = ProcessorOp(self, name="processor")
        sink = SinkOp(self, name="sink")

        # Create and configure data logger
        logger = GXFConsoleLogger(
            self,
            name="console_logger",
            log_inputs=True,
            log_outputs=True,
            log_metadata=False,
            log_tensor_data_content=False,
            denylist_patterns=[".*debug.*"]
        )

        # Add logger to application
        self.add_data_logger(logger)

        # Define workflow
        self.add_flow(source, processor)
        self.add_flow(processor, sink)

The example above shows example code adding the logger within the compose method, but it can also be added from the main application file via as done in the following example applications:

Tensor Interop (C++, Python)
Multithread Scheduling (C++, Python)
Video Replayer (C++, Python)

As with any other resource or operator in the SDK, parameters can be passed in directly via arguments or indirectly via reading from the YAML config.

YAML Configuration

Data loggers can be configured using YAML configuration files, making it easy to adjust logging behavior without recompiling:

YAML Configuration Example

Copy
Copied!

            
            # Data logging configuration
data_logging: true

basic_console_logger:
  log_inputs: true
  log_outputs: true
  log_metadata: true
  log_tensor_data_content: false
  allowlist_patterns: []
  denylist_patterns:
    - ".*debug.*"
    - ".*internal.*"

# Optional: Configure the text serializer
simple_text_serializer:
  max_elements: 10
  max_metadata_items: 5
  log_python_object_contents: true  # Python only

Copy
Copied!

            
            class MyApp : public holoscan::Application {
 public:
  void compose() override {
    // Create operators
    auto source = make_operator<SourceOp>("source");
    auto processor = make_operator<ProcessorOp>("processor");
    auto sink = make_operator<SinkOp>("sink");

    // Check if data logging is enabled
    auto enable_data_logging = from_config("data_logging").as<bool>();
    if (enable_data_logging) {
      auto text_serializer = make_resource<holoscan::data_loggers::SimpleTextSerializer>(
        "text-serializer",
        from_config("simple_text_serializer")
      )

      // Create logger with YAML configuration
      auto logger = make_resource<holoscan::data_loggers::GXFConsoleLogger>(
          "console_logger",
          holoscan::Arg("serializer", text_serializer),
          from_config("basic_console_logger")
      );
      add_data_logger(logger);
    }

    // Define workflow
    add_flow(source, processor);
    add_flow(processor, sink);
  }
};

int main(int argc, char** argv) {
  auto app = holoscan::make_application<MyApp>();

  // Load configuration
  app->config("path/to/config.yaml");

  app->run();
  return 0;
}

Copy
Copied!

            
            from holoscan.core import Application
from holoscan.data_loggers import GXFConsoleLogger, SimpleTextSerializer

class MyApp(Application):
    def compose(self):
        # Create operators
        source = SourceOp(self, name="source")
        processor = ProcessorOp(self, name="processor")
        sink = SinkOp(self, name="sink")

        # Check if data logging is enabled
        config = self.kwargs("data_logging")
        if config.get("data_logging", False):
            # Create logger with YAML configuration
            logger_config = self.kwargs("basic_console_logger")
            serializer_config = self.kwargs("simple_text_serializer")

            logger = GXFConsoleLogger(
                self,
                name="console_logger",
                serializer=SimpleTextSerializer(
                    self,
                    name="text_serializer",
                    **serializer_config
                ),
                **logger_config
            )
            self.add_data_logger(logger)

        # Define workflow
        self.add_flow(source, processor)
        self.add_flow(processor, sink)

def main():
    app = MyApp()

    # Load configuration
    app.config("path/to/config.yaml")

    app.run()

if __name__ == "__main__":
    main()

Custom Data Logger Implementation

You can create custom data loggers by implementing the DataLogger interface. To be able to use Holoscan Parameters and configure them via YAML it may be useful to inherit from the provided DataLoggerResource (as done for GXFConsoleLogger). Note that it is not required to inherit from DataLoggerResource, though, only the DataLogger interface.

C++
Python

Copy
Copied!

            
            #include "holoscan/core/resources/data_logger.hpp"

class MyCustomLogger : public holoscan::DataLoggerResource {
 public:
  HOLOSCAN_RESOURCE_FORWARD_ARGS_SUPER(MyCustomLogger, DataLoggerResource)

  void setup(ComponentSpec& spec) override {
    // Add custom parameters
    spec.param(output_file_, "output_file", "Output File",
               "Path to output log file", std::string(""));

    // Call parent setup for common parameters
    DataLoggerResource::setup(spec);
  }

  bool log_data(std::any data, const std::string& unique_id,
                int64_t acquisition_timestamp = -1,
                std::shared_ptr<MetadataDictionary> metadata = nullptr,
                IOSpec::IOType io_type = IOSpec::IOType::kOutput) override {
    // Implement custom logging logic
    return true;
  }

  bool log_tensor_data(const std::shared_ptr<Tensor>& tensor,
                       const std::string& unique_id,
                       int64_t acquisition_timestamp = -1,
                       const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
                       IOSpec::IOType io_type = IOSpec::IOType::kOutput) override {
    // Implement custom tensor logging
    return true;
  }

  bool log_tensormap_data(const TensorMap& tensor_map,
                          const std::string& unique_id,
                          int64_t acquisition_timestamp = -1,
                          const std::shared_ptr<MetadataDictionary>& metadata = nullptr,
                          IOSpec::IOType io_type = IOSpec::IOType::kOutput) override {
    // Implement custom tensor map logging
    return true;
  }

 private:
  Parameter<std::string> output_file_;
};

Copy
Copied!

            
            from holoscan.core import DataLoggerResource, ComponentSpec

class MyCustomLogger(DataLoggerResource):
    def __init__(self, fragment, output_file="", *args, **kwargs):
        self.output_file = output_file
        super().__init__(fragment, *args, **kwargs)

    def setup(self, spec: ComponentSpec):
        spec.param("output_file", "")
        # Call parent setup for common parameters
        super().setup(spec)

    def log_data(self, data, unique_id: str, acquisition_timestamp: int = -1,
                 metadata=None, io_type=None) -> bool:
        # Implement custom logging logic
        return True

    def log_tensor_data(self, tensor, unique_id: str, acquisition_timestamp: int = -1,
                        metadata=None, io_type=None) -> bool:
        # Implement custom tensor logging
        return True

    def log_tensormap_data(self, tensor_map, unique_id: str, acquisition_timestamp: int = -1,
                           metadata=None, io_type=None) -> bool:
        # Implement custom tensor map logging
        return True

Filtering and Pattern Matching

Data loggers inheriting from DataLoggerResource will automatically support filtering of messages using regex patterns:

Copy
Copied!

            
            basic_console_logger:
  # Only log messages from specific operators with "source" or "processor" appearing in the operator or port name
  allowlist_patterns:
    - ".*source.*"
    - ".*processor.*"

  # Exclude messages from ports with "debug" appearing in their name
  denylist_patterns:
    - ".*debug.*"

For a distributed application, the fragments will be named and the unique_id format used for each port will be:

{fragment_name}.{operator_name}.{port_name}
{fragment_name}.{operator_name}.{port_name}:index (for N:1 receiver ports (IOSpec::kAnySize))

For non-distributed applications, the single fragment is typically not named and the following simpler unique_id format is used:

{operator_name}.{port_name}
{operator_name}.{port_name}:index (for N:1 receiver ports (IOSpec::kAnySize))

The allowlist_patterns and denylist_patterns provide a way to include or exclude messages based on operator and/or port names. If there are no allowlist patterns, any messages not matching one of the denylist patterns are logged. If both types of patterns are specified, any messages that match at least one of the allowlist patterns but are not excluded by any of the denylist patterns are logged.

Tip

Use the multithread example as a reference for a complete working implementation with data logging enabled.

Note

Data logging can impact performance, especially when log_tensor_data_content is enabled for large tensors. Use filtering patterns to log only the data you need for debugging or monitoring.