Clara Holoscan Development Guide

Welcome to the Holoscan SDK development guide!

Here you will learn Holoscan core concepts and get started with the simple endoscopy application.

Then, we will walk you through the Graph eXecution Framework (GXF) extensions, and how to use Holoscan C++ API to wrap them as Holoscan operators, compose/build and run MyRecoder application as an example.

Since Holoscan Embedded SDK version 0.3.0, we are introducing a new framework with C++ API for the creation of applications.

The Holoscan API provides an easier and more flexible way to create applications using GXF’s features.

It is designed to be used as a drop-in replacement for the GXF’s API and provides a common interface for GXF components.

Fig. 1 Core concepts: Application

Fig. 2 Core concepts: Port

The core concepts of the Holoscan API are:

• Application: An application acquires and processes streaming data. An application is a collection of fragments where each fragment can be allocated to execute on a physical node of a Holoscan cluster.

• Fragment: A fragment is a building block of the Application. It is a Directed Acyclic Graph (DAG) of operators. A fragment can be assigned to a physical node of a Holoscan cluster during execution. The run-time execution manages communication across fragments. In a Fragment, Operators (Graph Nodes) are connected to each other by flows (Graph Edges).

• Operator: An operator is the most basic unit of work in this framework. An Operator receives streaming data at an input port, processes it, and publishes it to one of its output ports. A Codelet in GXF would be replaced with an Operator in the Framework. An Operator encapsulates Receivers and Transmitters of a GXF Entity as Input/Output Ports of the Operator.

• (Operator) Resource: Resources such as system memory or a GPU memory pool that an Operator needs to perform its job. Resources are allocated during the initialization phase of the application. This matches the semantics of GXF’s Memory Allocator or any other components derived from the Component class in GXF.

• Condition: A condition is a predicate that can be evaluated at runtime to determine if an operator should execute. This matches the semantics of GXF’s Scheduling Term.

• Port: An interaction point between two operators. Operators ingest data at Input ports and publish data at Output ports. Receiver, Transmitter, and MessageRouter in GXF would be replaced with the concept of Input/Output Port of the Operator and the Flow (Edge) of the Application Workflow (DAG) in the Framework.

• Message: A generic data object used by operators to communicate information.

• Executor: An Executor that manages the execution of a Fragment on a physical node. The framework provides a default Executor that uses a GXF Scheduler to execute an Application.

As of version 0.3.0, the Holoscan API provides a new convenient way to compose GXF Codelets as GXF Operators into Application workflows, without the need to write YAML files. The Holoscan API enables a more flexible/scalable approach to create applications.

Let’s get started with the Holoscan SDK.

The following figure shows a workflow graph of the simple endoscopy tool tracking application that consists of a single fragment as an application.

Fig. 3 Simple Endoscopy Workflow

The fragment consists of six operators that we provide as part of the Holoscan SDK. The operators are:

• Video Stream Replayer: This operator replays a video stream from a file. It is a GXF Operator (VideoStreamReplayerOp) that wraps a GXF Codelet ().

• Visualizer Format Converter: This operator converts the image format from RGB888 (24-bit pixel) to RGBA8888(32-bit pixel) for visualization for the Tool Tracking Visualizer. It is a GXF Operator (FormatConverterOp) that wraps a GXF Codelet ().

• Tool Tracking Visualizer: This operator visualizes the tool tracking results. It is a GXF Operator (ToolTrackingVizOp) that wraps a GXF Codelet ().

• Format Converter: This operator converts the data type of the image from uint8 to float32 for feeding into the tool tracking model. It is a GXF Operator (FormatConverterOp) that wraps a GXF Codelet ().

• LSTM TensorRT Inference: This operator performs the inference of the tool tracking model. It is a GXF Operator (LSTMTensorRTInferenceOp) that wraps a GXF Codelet ().

• Recorder: This operator records the video stream to a file. It is a GXF Operator (VideoStreamRecorderOp) that wraps a GXF Codelet ().

Code Example

Let’s see how we can create the application by composing the operators.

The following code snippet shows how to create the application with Holoscan SDK’s C++ API.

Code Snippet: apps/experiments/simple/simple.cpp

Listing 1 apps/experiments/simple/simple.cpp

            

            
#include <holoscan/holoscan.hpp>
#include <holoscan/std_ops.hpp>

class App : public holoscan::Application {
 public:
  void compose() override {
    using namespace holoscan;

    auto replayer = make_operator<ops::VideoStreamReplayerOp>("replayer", from_config("replayer"));

    auto recorder = make_operator<ops::VideoStreamRecorderOp>("recorder", from_config("recorder"));

    auto format_converter = make_operator<ops::FormatConverterOp>(
        "format_converter",
        from_config("format_converter_replayer"),
        Arg("pool") = make_resource<BlockMemoryPool>("pool", 1, 4919041, 2));

    auto lstm_inferer = make_operator<ops::LSTMTensorRTInferenceOp>(
        "lstm_inferer",
        from_config("lstm_inference"),
        Arg("pool") = make_resource<UnboundedAllocator>("pool"),
        Arg("cuda_stream_pool") = make_resource<CudaStreamPool>("cuda_stream", 0, 0, 0, 1, 5));

    auto visualizer_format_converter = make_operator<ops::FormatConverterOp>(
        "visualizer_format_converter",
        from_config("visualizer_format_converter_replayer"),
        Arg("pool") = make_resource<BlockMemoryPool>("pool", 1, 6558720, 2));

    auto visualizer = make_operator<ops::ToolTrackingVizOp>(
        "visualizer",
        from_config("visualizer"),
        Arg("pool") = make_resource<UnboundedAllocator>("pool"));

    // Flow definition
    add_flow(replayer, visualizer_format_converter);
    add_flow(visualizer_format_converter, visualizer, {{"tensor", "source_video"}});

    add_flow(replayer, format_converter);
    add_flow(format_converter, lstm_inferer);
    add_flow(lstm_inferer, visualizer, {{"tensor", "tensor"}});

    add_flow(replayer, recorder);
  }
};

int main() {
  App app;
  app.config("apps/endoscopy_tool_tracking/app_config.yaml");
  app.run();

  return 0;
}
        

In main() method, we create an instance of the App class that inherits from holoscan::Application. The App class overrides the compose() function to define the application’s flow graph. The compose() function is called by the run() function of the holoscan::Application class.

Before we call run(), we need to set the application configuration by calling the config() function.

The configuration file is a YAML file that contains the configuration of the operators and the application. The path to the configuration file is passed to the config() function as a string.

The configuration file for the simple application is located at apps/endoscopy_tool_tracking/app_config.yaml. Let’s take a look at the configuration file.

Code Snippet: apps/endoscopy_tool_tracking/app_config.yaml

Listing 2 apps/endoscopy_tool_tracking/app_config.yaml

            

            
...

replayer:
  directory: "/workspace/test_data/endoscopy/video"
  basename: "surgical_video"
  frame_rate: 0 # as specified in timestamps
  repeat: true # default: false
  realtime: true # default: true
  count: 0 # default: 0 (no frame count restriction)

...
        

In compose(), we create the operators and add them to the application flow graph. The operators are created using the make_operator() function. The make_operator() function takes the operator name and the operator configuration as arguments. The operator name is used to identify the operator in the flow graph. The operator configuration is holoscan::ArgList object(s) that contains the operator’s parameter values, or holoscan::Arg object(s).

The operator configuration (holoscan::ArgList object) is created using the from_config() function with a string argument that contains the name of the key in the configuration file. For example, from_config("replayer") creates an holoscan::ArgList object that contains the arguments of the replayer operator (such as values for ‘directory’, ‘basename’, ‘frame_rate’, ‘repeat’, ‘realtime’, and ‘count’ parameters).

For the Operator parameters that are not defined in the configuration file, we can pass them as holoscan::Arg objects to the make_operator() function. For example, the format_converter operator has a pool parameter that is not defined in the configuration file. We pass the pool parameter as an holoscan::Arg object to the make_operator() function, using make_resource() function to create the holoscan::Arg object. This section shows the available resources that can be used to create an operator resource.

After creating the operators, we add the operators to the application flow graph using the add_flow() function.

The add_flow() function takes the source operator, the destination operator, and the optional port pairs. The port pairs are used to connect the ports of the source operator to the ports of the destination operator. The first element of the pair is the port of the upstream operator and the second element is the port of the downstream operator. An empty port name (“”) can be used for specifying a port name if the operator has only one input/output port. If there is only one output port in the upstream operator and only one input port in the downstream operator, the port pairs can be omitted.

The following code snippet creates edges between the operators in the flow graph as shown in Fig. 3.

            

            
    add_flow(replayer, visualizer_format_converter);
    add_flow(visualizer_format_converter, visualizer, {{"tensor", "source_video"}});

    add_flow(replayer, format_converter);
    add_flow(format_converter, lstm_inferer);
    add_flow(lstm_inferer, visualizer, {{"tensor", "tensor"}});

    add_flow(replayer, recorder);
        

Build and Run the Application

Let’s build and run the application.

The code shown in the previous section is available in Clara Holoscan Embedded SDK repository(https://github.com/NVIDIA/clara-holoscan-embedded-sdk).

Please make sure that you have NVIDIA Container Toolkit installed on your system and that NVIDIA Container Toolkit is configured to use the NVIDIA driver installed on your system.

First, we need to clone the Clara Holoscan Embedded SDK repository.

            

            
git clone https://github.com/NVIDIA/clara-holoscan-embedded-sdk.git
cd clara-holoscan-embedded-sdk
        

Next, we need to install GXF package that contains GXF libraries and headers required to build the Holoscan application.

            

            
./run install_gxf
        

Now, we can build the application.

            

            
./run build
        

It will take some time to create the Docker image and build the application.

After the build is complete, the sample applications (including the simple endoscopy application whose code is located at apps/experiment/simple) are available under the build directory.

The simple endoscopy application binary is located at build/apps/experiment/simple/endoscopy_tool_tracking_simple.

Let’s run the application.

            

            
# Launch the docker container, mounting the current directory as /workspace/holoscan-sdk
./run launch
        

Inside the docker container (the current directory would be /workspace/holoscan-sdk/build), we can run the application.

            

            
export LD_LIBRARY_PATH=$(pwd):$(pwd)/lib:$LD_LIBRARY_PATH
./apps/experiments/simple/endoscopy_tool_tracking_simple
        

When the application is first launched, it will create a TensorRT engine file (.engine) under test_data/endoscopy/model/tool_loc_convlstm_engines/ directory and it will take some time to create the engine file.

After the engine file is created, the application will start running.

Fig. 4 Endoscopy application with tool tracking

Congratulations! You have successfully built and run the simple endoscopy application.

In the next sections, we will see how to create your application (MyRecorder) that records the video frames from the video file and save them to the disk.

GXF components in Holoscan can perform a multitude of sub-tasks ranging from data transformations, to memory management, to entity scheduling. In this section, we will explore an nvidia::gxf::Codelet component which in Holoscan is known as a “GXF extension”. Holoscan (GXF) extensions are typically concerned with application-specific sub-tasks such as data transformations, AI model inference, and the like.

Extension Lifecycle

The lifecycle of a Codelet is composed of the following five stages.

1. initialize - called only once when the codelet is created for the first time, and use of light-weight initialization.

2. deinitialize - called only once before the codelet is destroyed, and used for light-weight deinitialization.

3. start - called multiple times over the lifecycle of the codelet according to the order defined in the lifecycle, and used for heavy initialization tasks such as allocating memory resources.

4. stop - called multiple times over the lifecycle of the codelet according to the order defined in the lifecycle, and used for heavy deinitialization tasks such as deallocation of all resources previously assigned in start.

5. tick - called when the codelet is triggered, and is called multiple times over the codelet lifecycle; even multiple times between start and stop.

The flow between these stages is detailed in Fig. 5.

Fig. 5 Sequence of method calls in the lifecycle of a Holoscan extension

Implementing an Extension

In this section, we will implement a simple recorder that will highlight the actions we would perform in the lifecycle methods. The recorder receives data in the input queue and records the data to a configured location on the disk. The output format of the recorder files is the GXF-formatted index/binary replayer files (the format is also used for the data in the sample applications), where the gxf_index file contains timing and sequence metadata that refer to the binary/tensor data held in the gxf_entities file.

Declare the Class That Will Implement the Extension Functionality

The developer can create their Holoscan extension by extending the Codelet class, implementing the extension functionality by overriding the lifecycle methods, and defining the parameters the extension exposes at the application level via the registerInterface method. To define our recorder component we would need to implement some of the methods in the Codelet.

First, clone the Holoscan project from here and create a folder to develop our extension such as under gxf_extensions/my_recorder.

            

            
git clone https://github.com/NVIDIA/clara-holoscan-embedded-sdk.git
cd clara-holoscan-embedded-sdk
mkdir -p gxf_extensions/my_recorder
        

In our extension folder, we create a header file my_recorder.hpp with a declaration of our Holoscan component.

Listing 3 gxf_extensions/my_recorder/my_recorder.hpp

            

            
#include <string>

#include "gxf/core/handle.hpp"
#include "gxf/std/codelet.hpp"
#include "gxf/std/receiver.hpp"
#include "gxf/std/transmitter.hpp"
#include "gxf/serialization/file_stream.hpp"
#include "gxf/serialization/entity_serializer.hpp"


class MyRecorder : public nvidia::gxf::Codelet {
 public:
  gxf_result_t registerInterface(nvidia::gxf::Registrar* registrar) override;
  gxf_result_t initialize() override;
  gxf_result_t deinitialize() override;

  gxf_result_t start() override;
  gxf_result_t tick() override;
  gxf_result_t stop() override;

 private:
  nvidia::gxf::Parameter<nvidia::gxf::Handle<nvidia::gxf::Receiver>> receiver_;
  nvidia::gxf::Parameter<nvidia::gxf::Handle<nvidia::gxf::EntitySerializer>> my_serializer_;
  nvidia::gxf::Parameter<std::string> directory_;
  nvidia::gxf::Parameter<std::string> basename_;
  nvidia::gxf::Parameter<bool> flush_on_tick_;

  // File stream for data index
  nvidia::gxf::FileStream index_file_stream_;
  // File stream for binary data
  nvidia::gxf::FileStream binary_file_stream_;
  // Offset into binary file
  size_t binary_file_offset_;
};
        

Declare the Parameters to Expose at the Application Level

Next, we can start implementing our lifecycle methods in the my_recorder.cpp file, which we also create in gxf_extensions/my_recorder path.

Our recorder will need to expose the nvidia::gxf::Parameter variables to the application so the parameters can be modified by configuration.

Listing 4 registerInterface in gxf_extensions/my_recorder/my_recorder.cpp

            

            
#include "my_recorder.hpp"

gxf_result_t MyRecorder::registerInterface(nvidia::gxf::Registrar* registrar) {
  nvidia::gxf::Expected<void> result;
  result &= registrar->parameter(
      receiver_, "receiver", "Entity receiver",
      "Receiver channel to log");
  result &= registrar->parameter(
      my_serializer_, "serializer", "Entity serializer",
      "Serializer for serializing input data");
  result &= registrar->parameter(
      directory_, "out_directory", "Output directory path",
      "Directory path to store received output");
  result &= registrar->parameter(
      basename_, "basename", "File base name",
      "User specified file name without extension",
      nvidia::gxf::Registrar::NoDefaultParameter(), GXF_PARAMETER_FLAGS_OPTIONAL);
  result &= registrar->parameter(
      flush_on_tick_, "flush_on_tick", "Boolean to flush on tick",
      "Flushes output buffer on every `tick` when true", false);  // default value `false`
  return nvidia::gxf::ToResultCode(result);
}
        

If we are creating a pure GXF application (see Creating the GXF Application Definition section in the GXF documentation), in the application YAML, our component’s parameters can be specified in the following format. Don’t worry about what it means for now. With Holoscan API, the application is defined in C++ code instead of the YAML file, and the parameter values are set in the application code or via the configuration file (YAML).

Listing 5 Example parameters for MyRecorder component

            

            
name: my_recorder_entity
components:
  - name: my_recorder_component
    type: MyRecorder
    parameters:
      receiver: receiver
      serializer: my_serializer
      out_directory: /home/user/out_path
      basename: my_output_file  # optional
      # flush_on_tick: false # optional
        

Note that all the parameters exposed at the application level are mandatory except for flush_on_tick, which defaults to false, and basename, whose default is handled at initialize() below.

Implement the Lifecycle Methods

This extension does not need to perform any heavy-weight initialization tasks, so we will concentrate on initialize(), tick(), and deinitialize() methods which define the core functionality of our component. At initialization, we will create a file stream and keep track of the bytes we write on tick() via binary_file_offset.

Listing 6 initialize in gxf_extensions/my_recorder/my_recorder.cpp

            

            
gxf_result_t MyRecorder::initialize() {
  // Create path by appending receiver name to directory path if basename is not provided
  std::string path = directory_.get() + '/';
  if (const auto& basename = basename_.try_get()) {
    path += basename.value();
  } else {
    path += receiver_->name();
  }

  // Initialize index file stream as write-only
  index_file_stream_ = nvidia::gxf::FileStream("", path + nvidia::gxf::FileStream::kIndexFileExtension);

  // Initialize binary file stream as write-only
  binary_file_stream_ = nvidia::gxf::FileStream("", path + nvidia::gxf::FileStream::kBinaryFileExtension);

  // Open index file stream
  nvidia::gxf::Expected<void> result = index_file_stream_.open();
  if (!result) {
    return nvidia::gxf::ToResultCode(result);
  }

  // Open binary file stream
  result = binary_file_stream_.open();
  if (!result) {
    return nvidia::gxf::ToResultCode(result);
  }
  binary_file_offset_ = 0;

  return GXF_SUCCESS;
}
        

When de-initializing, our component will take care of closing the file streams that were created at initialization.

Listing 7 deinitialize in gxf_extensions/my_recorder/my_recorder.cpp

            

            
gxf_result_t MyRecorder::deinitialize() {
  // Close binary file stream
  nvidia::gxf::Expected<void> result = binary_file_stream_.close();
  if (!result) {
    return nvidia::gxf::ToResultCode(result);
  }

  // Close index file stream
  result = index_file_stream_.close();
  if (!result) {
    return nvidia::gxf::ToResultCode(result);
  }

  return GXF_SUCCESS;
}
        

In our recorder, no heavy-weight initialization tasks are required so we implement the following, however, we would use start() and stop() methods for heavy-weight tasks such as memory allocation and deallocation.

Listing 8 start/stop in gxf_extensions/my_recorder/my_recorder.cpp

            

            
gxf_result_t MyRecorder::start() {
  return GXF_SUCCESS;
}

gxf_result_t MyRecorder::stop() {
  return GXF_SUCCESS;
}
        

Finally, we write the component-specific functionality of our extension by implementing tick().

Listing 9 tick in gxf_extensions/my_recorder/my_recorder.cpp

            

            
gxf_result_t MyRecorder::tick() {
  // Receive entity
  nvidia::gxf::Expected<nvidia::gxf::Entity> entity = receiver_->receive();
  if (!entity) {
    return nvidia::gxf::ToResultCode(entity);
  }

  // Write entity to binary file
  nvidia::gxf::Expected<size_t> size = my_serializer_->serializeEntity(entity.value(), &binary_file_stream_);
  if (!size) {
    return nvidia::gxf::ToResultCode(size);
  }

  // Create entity index
  nvidia::gxf::EntityIndex index;
  index.log_time = std::chrono::system_clock::now().time_since_epoch().count();
  index.data_size = size.value();
  index.data_offset = binary_file_offset_;

  // Write entity index to index file
  nvidia::gxf::Expected<size_t> result = index_file_stream_.writeTrivialType(&index);
  if (!result) {
    return nvidia::gxf::ToResultCode(result);
  }
  binary_file_offset_ += size.value();

  if (flush_on_tick_) {
    // Flush binary file output stream
    nvidia::gxf::Expected<void> result = binary_file_stream_.flush();
    if (!result) {
      return nvidia::gxf::ToResultCode(result);
    }

    // Flush index file output stream
    result = index_file_stream_.flush();
    if (!result) {
      return nvidia::gxf::ToResultCode(result);
    }
  }

  return GXF_SUCCESS;
}
        

Register the Extension as a Holoscan Component

As a final step, we must register our extension so it is recognized as a component and loaded by the application executor. For this we create a simple declaration in my_recorder_ext.cpp as follows.

Listing 10 gxf_extensions/my_recorder/my_recorder_ext.cpp

            

            
#include "gxf/std/extension_factory_helper.hpp"

#include "my_recorder.hpp"

GXF_EXT_FACTORY_BEGIN()
GXF_EXT_FACTORY_SET_INFO(0xb891cef3ce754825, 0x9dd3dcac9bbd8483, "MyRecorderExtension",
                         "My example recorder extension", "NVIDIA", "0.1.0", "LICENSE");
GXF_EXT_FACTORY_ADD(0x2464fabf91b34ccf, 0xb554977fa22096bd, MyRecorder,
                    nvidia::gxf::Codelet, "My example recorder codelet.");
GXF_EXT_FACTORY_END()
        

GXF_EXT_FACTORY_SET_INFO configures the extension with the following information in order:

• UUID which can be generated using scripts/generate_extension_uuids.py which defines the extension id

• extension name

• extension description

• author

• extension version

• license text

GXF_EXT_FACTORY_ADD registers the newly built extension as a valid Codelet component with the following information in order:

• UUID which can be generated using scripts/generate_extension_uuids.py which defines the component id (this must be different from the extension id),

• fully qualified extension class,

• fully qualifies base class,

• component description

To build a shared library for our new extension which can be loaded by a Holoscan application at runtime we use a CMake file under gxf_extensions/my_recorder/CMakeLists.txt with the following content.

Listing 11 gxf_extensions/my_recorder/CMakeLists.txt

            

            
# Create library
add_library(my_recorder_lib SHARED
  my_recorder.cpp
  my_recorder.hpp
)
target_link_libraries(my_recorder_lib
  PUBLIC
    GXF::std
    GXF::serialization
    yaml-cpp
)

# Create extension
add_library(my_recorder SHARED
  my_recorder_ext.cpp
)
target_link_libraries(my_recorder
  PUBLIC my_recorder_lib
)

# Install GXF extension as a component 'holoscan-embedded-gxf_extensions'
install_gxf_extension(my_recorder) # this will also install my_recorder_lib
# install_gxf_extension(my_recorder_lib) # this statement is not necessary because this library follows `<extension library name>_lib` convention.
        

Here, we create a library my_recorder_lib with the implementation of the lifecycle methods, and the extension my_recorder which exposes the C API necessary for the application runtime to interact with our component.

To make our extension discoverable from the project root we add the line

            

            
add_subdirectory(my_recorder)
        

to the CMake file gxf_extensions/CMakeLists.txt.

At this point, we have a complete extension that records data coming into its receiver queue to the specified location on the disk using the GXF-formatted binary/index files.

Now that we know how to write a GXF extension, we can create a simple “identity” application consisting of a replayer, which reads contents from a file on disk, and our recorder from the last section, which will store the output of the replayer exactly in the same format. This allows us to see whether the output of the recorder matches the original input files.

With Holoscan C++ API, we can wrap GXF Components (including Codelets) in an Entity as a Holoscan Operator. Then, we can compose Operators programmatically, to create a Holoscan application.

For our C++ API-based application, we create the directory apps/my_recorder_app with our MyRecorderOp Operator implementation.

Listing 12 apps/my_recorder_app/my_recorder_op.hpp

            

            
#ifndef APPS_MY_RECORDER_APP_MY_RECORDER_OP_HPP
#define APPS_MY_RECORDER_APP_MY_RECORDER_OP_HPP

#include "holoscan/core/gxf/gxf_operator.hpp"

namespace holoscan::ops {

class MyRecorderOp : public holoscan::ops::GXFOperator {
 public:
  HOLOSCAN_OPERATOR_FORWARD_ARGS_SUPER(MyRecorderOp, holoscan::ops::GXFOperator)

  MyRecorderOp() = default;

  const char* gxf_typename() const override { return "MyRecorder"; }

  void setup(OperatorSpec& spec) override;

  void initialize() override;

 private:
  Parameter<holoscan::IOSpec*> receiver_;
  Parameter<std::shared_ptr<holoscan::Resource>> my_serializer_;
  Parameter<std::string> directory_;
  Parameter<std::string> basename_;
  Parameter<bool> flush_on_tick_;
};

}  // namespace holoscan::ops

#endif/* APPS_MY_RECORDER_APP_MY_RECORDER_OP_HPP */
        

holoscan::ops::MyRecorderOp class wraps a MyRecorder GXF Codelet by inheriting holoscan::ops::GXFOperator.

To wrap a GXF Codelet as a Holoscan Operator, we need to implement the following functions:

• const char* gxf_typename() const override: return the GXF type name of the Codelet. The fully-qualified class name (MyRecorder) for the GXF Codelet is specified.

• void setup(OperatorSpec& spec) override: setup the OperatorSpec with the inputs/outputs and parameters of the Operator.

• void initialize() override: initialize the Operator.

HOLOSCAN_OPERATOR_FORWARD_ARGS_SUPER(MyRecorderOp, holoscan::ops::GXFOperator)) macro is used to forward the arguments of the constructor to the base class.

Then, we need to define the fields of the MyRecorderOp class.

Let’s see the fields of the MyRecorderOp class. You can see that the same fields with the same names (but different types) are defined in both the MyRecorderOp class and the MyRecorder GXF Codelet.

Listing 13 Parameter declarations in apps/my_recorder_app/my_recorder_op.hpp

            

            
  nvidia::gxf::Parameter<nvidia::gxf::Handle<nvidia::gxf::Receiver>> receiver_;
  nvidia::gxf::Parameter<nvidia::gxf::Handle<nvidia::gxf::EntitySerializer>> my_serializer_;
  nvidia::gxf::Parameter<std::string> directory_;
  nvidia::gxf::Parameter<std::string> basename_;
  nvidia::gxf::Parameter<bool> flush_on_tick_;
        

In the MyRecorderOp class, the followings are changed:

• holoscan::Parameter type is used instead of nvidia::gxf::Parameter type

• holoscan::IOSpec* type is used instead of nvidia::gxf::Handle<nvidia::gxf::Receiver>> or nvidia::gxf::Handle<nvidia::gxf::Transmitter>> type

• std::shared_ptr<holoscan::Resource>> type is used instead of nvidia::gxf::Handle<T>> type (such as nvidia::gxf::Handle<nvidia::gxf::EntitySerializer>>)

The implementation of the setup(OperatorSpec& spec) function and the initialize() function are as follows:

Listing 14 apps/my_recorder_app/my_recorder_op.cpp

            

            
#include "./my_recorder_op.hpp"

#include "holoscan/core/fragment.hpp"
#include "holoscan/core/gxf/entity.hpp"
#include "holoscan/core/operator_spec.hpp"

#include "holoscan/core/resources/gxf/video_stream_serializer.hpp"

namespace holoscan::ops {

void MyRecorderOp::setup(OperatorSpec& spec) {
  auto& input = spec.input<::gxf::Entity>("input");
  // Above is same with the following two lines (a default condition is assigned to the input port if not specified):
  //
  // auto& input = spec.input<::gxf::Entity>("input")
  // .condition(ConditionType::kMessageAvailable, Arg("min_size") = 1);

  spec.param(receiver_, "receiver", "Entity receiver", "Receiver channel to log", &input);
  spec.param(my_serializer_,
             "serializer",
             "Entity serializer",
             "Serializer for serializing input data");
  spec.param(directory_, "out_directory", "Output directory path", "Directory path to store received output");
  spec.param(basename_, "basename", "File base name", "User specified file name without extension");
  spec.param(flush_on_tick_,
             "flush_on_tick",
             "Boolean to flush on tick",
             "Flushes output buffer on every `tick` when true",
             false);
}

void MyRecorderOp::initialize() {
  // Set up prerequisite parameters before calling GXFOperator::initialize()
  auto frag = fragment();
  auto serializer =
      frag->make_resource<holoscan::VideoStreamSerializer>("serializer");
  add_arg(Arg("serializer") = serializer);

  GXFOperator::initialize();
}

}  // namespace holoscan::ops
        

In the setup(OperatorSpec& spec) function, we set up the inputs/outputs and parameters of the Operator.

Please compare the content of the function with MyRecorder class’s registerInterface function. You can see that setup(OperatorSpec& spec) function is very similar to the registerInterface(OperatorSpec& spec) function in the MyRecorder class.

In C++ API, GXF Receiver and Transmitter components (such as DoubleBufferReceiver and DoubleBufferTransmitter) are considered as input and output ports of the Operator so we register the inputs/outputs of the Operator with input<T> and output<T> functions (where T is the data type of the port).

Compared to the pure GXF application that does the same job, the SchedulingTerm) definitions of an Entity in GXF Application YAML are specified as Conditions (e.g., holoscan::MessageAvailableCondition and holoscan::DownstreamMessageAffordableCondition) on the input/output ports.

The following statements

            

            
  auto& input = spec.input<::gxf::Entity>("input");
  // Above is same with the following two lines (a default condition is assigned to the input port if not specified):
  //
  // auto& input = spec.input<::gxf::Entity>("input")
  // .condition(ConditionType::kMessageAvailable, Arg("min_size") = 1);
  spec.param(receiver_, "receiver", "Entity receiver", "Receiver channel to log", &input);
        

represent the following highlighted statements of GXF Application YAML:

Listing 15 A part of apps/my_recorder_app_gxf/my_recorder_gxf.yaml

            

            
name: recorder
components:
 - name: input
   type: nvidia::gxf::DoubleBufferReceiver
 - name: allocator
   type: nvidia::gxf::UnboundedAllocator
 - name: component_serializer
   type: nvidia::gxf::StdComponentSerializer
   parameters:
     allocator: allocator
 - name: entity_serializer
   type: nvidia::holoscan::stream_playback::VideoStreamSerializer   # inheriting from nvidia::gxf::EntitySerializer
   parameters:
     component_serializers: [component_serializer]
 - type: MyRecorder
   parameters:
     receiver: input
     serializer: entity_serializer
     out_directory: "/tmp"
     basename: "tensor_out"
 - type: nvidia::gxf::MessageAvailableSchedulingTerm
   parameters:
     receiver: input
     min_size: 1

        

In the same way, if we had a Transmitter GXF component, we would have the following statements (Please see available constants for holoscan::ConditionType):

            

            
  auto& output = spec.output<::gxf::Entity>("output");
  // Above is same with the following two lines (a default condition is assigned to the output port if not specified):
  //
  // auto& output = spec.output<::gxf::Entity>("output")
  // .condition(ConditionType::kDownstreamMessageAffordable, Arg("min_size") = 1);
        

In the initialize() function, we set up the pre-defined parameters such as serializer.

            

            
  auto frag = fragment();
  auto serializer =
      frag->make_resource<holoscan::VideoStreamSerializer>("serializer");
  add_arg(Arg("serializer") = serializer); // set 'serializer' parameter with 'serializer' resource.
        

Holoscan C++ API provides holoscan::VideoStreamSerializer class (include/holoscan/core/resources/gxf/video_stream_serializer.hpp and src/core/resources/gxf/video_stream_serializer.cpp) for nvidia::holoscan::stream_playback::VideoStreamSerializer GXF component and above statements covers the highlighed statements of GXF Application YAML:

Listing 16 Another part of apps/my_recorder_app_gxf/my_recorder_gxf.yaml

            

            
name: recorder
components:
 - name: input
   type: nvidia::gxf::DoubleBufferReceiver
 - name: allocator
   type: nvidia::gxf::UnboundedAllocator
 - name: component_serializer
   type: nvidia::gxf::StdComponentSerializer
   parameters:
     allocator: allocator
 - name: entity_serializer
   type: nvidia::holoscan::stream_playback::VideoStreamSerializer   # inheriting from nvidia::gxf::EntitySerializer
   parameters:
     component_serializers: [component_serializer]
 - type: MyRecorder
   parameters:
     receiver: input
     serializer: entity_serializer
     out_directory: "/tmp"
     basename: "tensor_out"
 - type: nvidia::gxf::MessageAvailableSchedulingTerm
   parameters:
     receiver: input
     min_size: 1
        

The following code snippet shows how to create the Holoscan Application using the C++ API.

Listing 17 apps/my_recorder_app/main.cpp

            

            
#include <holoscan/holoscan.hpp>
#include <holoscan/std_ops.hpp>

#include "./my_recorder_op.hpp"

class App : public holoscan::Application {
 public:

  void compose() override {
    using namespace holoscan;

    HOLOSCAN_LOG_DEBUG("In App::compose() method");

    auto replayer = make_operator<ops::VideoStreamReplayerOp>("replayer", from_config("replayer"));
    // auto replayer = make_operator
 
  ("replayer", from_config("replayer"),
 
    // Arg("frame_rate") = 30.f, // same with Arg("frame_rate", 30.f)
    // Arg("repeat") = true);
    auto recorder = make_operator<ops::MyRecorderOp>("recorder", from_config("recorder"));

    HOLOSCAN_LOG_INFO("replayer.directory: {}", from_config("replayer.directory").as<std::string>());

    // Flow definition
    add_flow(replayer, recorder);
    // Above is same with:
    //
    // // replayer's output port named 'output' is connected to recorder's input port named 'input'
    // add_flow(replayer, recorder, {{"output", "input"}});
    // // or,
    // // replayer has only one output port and recorder has only one input port so names can be omitted
    // add_flow(replayer, recorder, {{"", ""}});
  }
};

int main(int argc, char** argv) {
  holoscan::load_env_log_level();

  auto app = holoscan::make_application<App>();
  app->config("apps/my_recorder_app/app_config.yaml");
  app->run();

  return 0;
}
        

The App class is the main class of the Holoscan Application. It inherits from holoscan::Application class.

In the compose() function, we create the operators and flows of the Holoscan Application.

We can call make_operator() to create an operator of type <OperatorT> and pass parameter values to the operator.

from_config() function is used to get parameter values from the configuration file (apps/my_recorder_app/app_config.yaml) that is passed to the config() function in the main function.

Let’s create a configuration file for the Holoscan Application (C++ API) in the apps/my_recorder_app/app_config.yaml file.

Listing 18 apps/my_recorder_app/app_config.yaml

            

            
# 'extensions' has the same content with 'build/apps/my_recorder_app_gxf/my_recorder_gxf_manifest.yaml' file.
extensions:
  - libgxf_std.so
  - libgxf_cuda.so
  - libgxf_multimedia.so
  - libgxf_serialization.so
  - ./gxf_extensions/my_recorder/libmy_recorder.so
  - ./gxf_extensions/stream_playback/libstream_playback.so

# Configururation for 'holoscan::ops::VideoStreamReplayerOp' Operator
replayer:
  directory: "/workspace/test_data/endoscopy/video"
  basename: "surgical_video"
  frame_rate: 0   # as specified in timestamps
  repeat: false   # default: false
  realtime: true  # default: true
  count: 0        # default: 0 (no frame count restriction)

# Configururation for 'holoscan::ops::MyRecorderOp' Operator
recorder:
  out_directory: "/tmp"
  basename: "tensor_out"
        

In app_config.yaml, it has the information of GXF extension paths (that is same with the content of build/apps/my_recorder_app_gxf/my_recorder_gxf_manifest.yaml file if you have followed the Creating the GXF Application Definition section) and some parameter values for VideoStreamReplayerOp and MyRecorderOp operators.

The extensions field in this YAML configuration file is a list of GXF extension paths. The first four paths are the paths of GXF core extensions that are required for the VideoStreamReplayerOp and MyRecorderOp operators (you can find the paths under the build/lib directory once you have built the Holoscan SDK). The last two paths are the paths of GXF extensions that are required for the MyRecorderOp and VideoStreamReplayerOp operator (you can find the paths under the build directory once you have built the Holoscan SDK).

Compared with my_recorder_gxf.yaml file, app_config.yaml file is concise and easier to read. It is also easier to modify the parameter values of operators (codelets). In GXF Application YAML, we had to specify graph edges between graph nodes through nvidia::gxf::Connection component. With C++ API, we can connect Operator nodes programmatically with add_flow() function.

On top of the parameter values from the configuration file, if you want to override the parameter values, you can pass them to the operator as an argument of the make_operator() function. For example, if you want to override the parameter values of replayer, you can do it as follows:

            

            
auto replayer = make_operator<ops::VideoStreamReplayerOp>("replayer", from_config("replayer"),
                    Arg("frame_rate") = 30.f, // same with Arg("frame_rate", 30.f)
                    Arg("repeat") = true);
        

In main() function, we create the Holoscan Application and pass the configuration file to the config() function.

holoscan::load_env_log_level() function is used to set the log level of the Holoscan Application from the environment variable HOLOSCAN_LOG_LEVEL:

HOLOSCAN_LOG_LEVEL can be set to one of the following values:

• TRACE

• DEBUG

• INFO

• WARN

• ERROR

• CRITICAL

• OFF

            

            
export HOLOSCAN_LOG_LEVEL=TRACE
        

We can call HOLOSCAN_LOG_XXX() macros to log messages. The format string follows the fmtlib format string syntax.

An Application object is created in the main() function through make_application() function and it is launched by calling run() function.

Once main.cpp file is available, we need to declare a CMake file apps/my_recorder_app/CMakeLists.txt as follows.

Listing 19 apps/my_recorder_app/CMakeLists.txt

            

            
add_executable(my_recorder_app
    my_recorder_op.hpp
    my_recorder_op.cpp
    main.cpp
)

target_link_libraries(my_recorder_app
   PRIVATE
   holoscan-embedded
)

# Download the associated dataset if needed
if(HOLOSCAN_DOWNLOAD_DATASETS)
  add_dependencies(my_recorder_app endoscopy_data)
endif()

# Copy config file
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/app_config.yaml" DESTINATION "${CMAKE_CURRENT_BINARY_DIR}")

# Get relative folder path for the app
file(RELATIVE_PATH app_relative_dest_path ${CMAKE_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR})

# Install the app
install(TARGETS "my_recorder_app"
    DESTINATION "${app_relative_dest_path}"
    COMPONENT "holoscan-embedded-apps"
)
install(FILES "${CMAKE_CURRENT_SOURCE_DIR}/app_config.yaml"
    DESTINATION ${app_relative_dest_path}
    COMPONENT "holoscan-embedded-apps"
)
        

In the apps/my_recorder_app/CMakeLists.txt file, we declare the executable my_recorder_app and link the holoscan-embedded library. We also copy the configuration file apps/my_recorder_app/app_config.yaml to the same folder under ${CMAKE_CURRENT_BINARY_DIR} to make it available to the application.

Finally, we install the application to the holoscan-embedded-apps component by calling install() function.

To make this Holoscan application discoverable by the build, in the root of the repository, we add the following line

            

            
add_subdirectory(my_recorder_app)
        

to apps/CMakeLists.txt.

To run our application in a local development container:

1. Follow the instructions under the Using a Development Container section steps 1-5 (try clearing the CMake cache by removing the build folder before compiling).

   You can execute the following commands to build

   Copy

   Copied!

               
   
               
   ./run install_gxf
   # ./run clear_cache # if you want to clear build/install/cache folders
   ./run build
           

2. Our application can now be run in the development container using the command, where $(pwd) is the path to the build folder:

   You can execute ./run launch to run the development container.

   Copy

   Copied!

               
   
               
   ./run launch
           

   Then, you can execute the following commands to run the application

   Copy

   Copied!

               
   
               
   LD_LIBRARY_PATH=$(pwd):$(pwd)/lib:$LD_LIBRARY_PATH ./apps/my_recorder_app/my_recorder_app
           

   inside the development container.

   Copy

   Copied!

               
   
               
   @LINUX:/workspace/holoscan-sdk/build$ LD_LIBRARY_PATH=$(pwd):$(pwd)/lib:$LD_LIBRARY_PATH ./apps/my_recorder_app/my_recorder_app
   2022-08-24 12:36:48.685 INFO  /workspace/holoscan-sdk/src/core/executors/gxf/gxf_executor.cpp@39: Creating context
   [2022-08-24 12:36:48.685] [holoscan] [info] [gxf_executor.cpp:64] Loading extensions...
   [2022-08-24 12:36:48.692] [holoscan] [info] [main.cpp:20] replayer.directory: /workspace/test_data/endoscopy/video
   [2022-08-24 12:36:48.692] [holoscan] [info] [gxf_executor.cpp:222] Activating Graph...
   [2022-08-24 12:36:48.693] [holoscan] [info] [gxf_executor.cpp:224] Running Graph...
   [2022-08-24 12:36:48.693] [holoscan] [info] [gxf_executor.cpp:226] Waiting for completion...
   2022-08-24 12:36:48.693 INFO  gxf/std/greedy_scheduler.cpp@170: Scheduling 2 entities
   2022-08-24 12:37:16.084 INFO  /workspace/holoscan-sdk/gxf_extensions/stream_playback/video_stream_replayer.cpp@144: Reach end of file or playback count reaches to the limit. Stop ticking.
   2022-08-24 12:37:16.084 INFO  gxf/std/greedy_scheduler.cpp@329: Scheduler stopped: Some entities are waiting for execution, but there are no periodic or async entities to get out of the deadlock.
   2022-08-24 12:37:16.084 INFO  gxf/std/greedy_scheduler.cpp@353: Scheduler finished.
   [2022-08-24 12:37:16.084] [holoscan] [info] [gxf_executor.cpp:228] Deactivating Graph...
   2022-08-24 12:37:16.085 INFO  /workspace/holoscan-sdk/src/core/executors/gxf/gxf_executor.cpp@49: Destroying context
           

   You can set HOLOSCAN_LOG_LEVEL environment variable to DEBUG (or other levels such as ‘TRACE’) to see more logs.

   Copy

   Copied!

               
   
               
   @LINUX:/workspace/holoscan-sdk/build$ export HOLOSCAN_LOG_LEVEL=DEBUG
   @LINUX:/workspace/holoscan-sdk/build$ LD_LIBRARY_PATH=$(pwd):$(pwd)/lib:$LD_LIBRARY_PATH ./apps/my_recorder_app/my_recorder_app
   2022-08-24 12:41:01.616 INFO  /workspace/holoscan-sdk/src/core/executors/gxf/gxf_executor.cpp@39: Creating context
   [2022-08-24 12:41:01.616] [holoscan] [info] [gxf_executor.cpp:64] Loading extensions...
   [2022-08-24 12:41:01.622] [holoscan] [debug] [main.cpp:12] In App::compose() method
   [2022-08-24 12:41:01.622] [holoscan] [debug] [fragment.hpp:62] Creating operator 'replayer'
   [2022-08-24 12:41:01.622] [holoscan] [debug] [fragment.hpp:84] Creating resource 'entity_serializer'
   [2022-08-24 12:41:01.622] [holoscan] [debug] [fragment.hpp:84] Creating resource 'component_serializer'
   [2022-08-24 12:41:01.622] [holoscan] [debug] [fragment.hpp:84] Creating resource 'allocator'
   [2022-08-24 12:41:01.622] [holoscan] [debug] [fragment.hpp:106] Creating condition 'boolean_scheduling_term'
   [2022-08-24 12:41:01.623] [holoscan] [debug] [fragment.hpp:62] Creating operator 'recorder'
   [2022-08-24 12:41:01.623] [holoscan] [debug] [fragment.hpp:84] Creating resource 'serializer'
   [2022-08-24 12:41:01.623] [holoscan] [debug] [fragment.hpp:84] Creating resource 'component_serializer'
   [2022-08-24 12:41:01.623] [holoscan] [debug] [fragment.hpp:84] Creating resource 'allocator'
   [2022-08-24 12:41:01.623] [holoscan] [info] [main.cpp:20] replayer.directory: /workspace/test_data/endoscopy/video
   [2022-08-24 12:41:01.623] [holoscan] [debug] [gxf_executor.cpp:107] Operator: replayer
   [2022-08-24 12:41:01.623] [holoscan] [debug] [gxf_executor.cpp:115]   Next operator: recorder
   [2022-08-24 12:41:01.623] [holoscan] [debug] [gxf_executor.cpp:125]     Port: output -> input
   [2022-08-24 12:41:01.623] [holoscan] [debug] [gxf_executor.cpp:107] Operator: recorder
   [2022-08-24 12:41:01.623] [holoscan] [info] [gxf_executor.cpp:222] Activating Graph...
   [2022-08-24 12:41:01.713] [holoscan] [info] [gxf_executor.cpp:224] Running Graph...
   [2022-08-24 12:41:01.713] [holoscan] [info] [gxf_executor.cpp:226] Waiting for completion...
   2022-08-24 12:41:01.713 INFO  gxf/std/greedy_scheduler.cpp@170: Scheduling 2 entities
   2022-08-24 12:41:29.093 INFO  /workspace/holoscan-sdk/gxf_extensions/stream_playback/video_stream_replayer.cpp@144: Reach end of file or playback count reaches to the limit. Stop ticking.
   2022-08-24 12:41:29.093 INFO  gxf/std/greedy_scheduler.cpp@329: Scheduler stopped: Some entities are waiting for execution, but there are no periodic or async entities to get out of the deadlock.
   2022-08-24 12:41:29.093 INFO  gxf/std/greedy_scheduler.cpp@353: Scheduler finished.
   [2022-08-24 12:41:29.093] [holoscan] [info] [gxf_executor.cpp:228] Deactivating Graph...
   2022-08-24 12:41:29.209 INFO  /workspace/holoscan-sdk/src/core/executors/gxf/gxf_executor.cpp@49: Destroying context
           

A successful run (it takes about 30 secs) will result in output files (tensor_out.gxf_index and tensor_out.gxf_entities in /tmp) that match the original input files (surgical_video.gxf_index and surgical_video.gxf_entities under test_data/endoscopy/video) exactly.

            

            
@LINUX:/workspace/holoscan-sdk/build$ ls -al /tmp
total 821392
drwxrwxrwt 1 root root      4096 Aug 24 12:36 .
drwxr-xr-x 1 root root      4096 Aug 24 12:36 ..
drwxrwxrwt 2 root root      4096 Aug 11 21:42 .X11-unix
-rw-r--r-- 1 1000 1000    738674 Aug 24 12:41 gxf_log
-rw-r--r-- 1 1000 1000 840054484 Aug 24 12:41 tensor_out.gxf_entities
-rw-r--r-- 1 1000 1000     16392 Aug 24 12:41 tensor_out.gxf_index

@LINUX:/workspace/holoscan-sdk/build$ ls -al ../test_data/endoscopy/video/
total 839116
drwxr-xr-x 2 1000 1000      4096 Aug 24 02:08 .
drwxr-xr-x 4 1000 1000      4096 Aug 24 02:07 ..
-rw-r--r-- 1 1000 1000  19164125 Jun 17 16:31 raw.mp4
-rw-r--r-- 1 1000 1000 840054484 Jun 17 16:31 surgical_video.gxf_entities
-rw-r--r-- 1 1000 1000     16392 Jun 17 16:31 surgical_video.gxf_index