Built-in Operators and Extensions

The units of work of Holoscan applications are implemented within Operators, as described in the core concepts of the SDK. The operators included in the SDK provide domain-agnostic functionalities such as IO, machine learning inference, processing, and visualization, optimized for AI streaming pipelines, relying on a set of Core Technologies.

Operators

The operators below are defined under the holoscan::ops namespace for C++ and CMake, and under the holoscan.operators module in Python.

Class	CMake target/lib	Documentation
AJASourceOp	`aja`	`C++`/`Python`
BayerDemosaicOp	`bayer_demosaic`	`C++`/`Python`
FormatConverterOp	`format_converter`	`C++`/`Python`
HolovizOp	`holoviz`	`C++`/`Python`
MultiAIInferenceOp	`multiai_inference`	`C++`/`Python`
MultiAIPostprocessorOp	`multiai_postprocessor`	`C++`/`Python`
PingRxOp	`ping_rx`	`C++`/`Python`
PingTxOp	`ping_tx`	`C++`/`Python`
SegmentationPostprocessorOp	`segmentation_postprocessor`	`C++`/`Python`
TensorRTInferenceOp †	`tensor_rt`	`C++`/`Python`
VideoStreamRecorderOp	`video_stream_recorder`	`C++`/`Python`
VideoStreamReplayerOp	`video_stream_replayer`	`C++`/`Python`

† deprecated

Given an instance of an operator class, you can print a human-readable description of its specification to inspect the inputs, outputs, and parameters that can be configured on that operator class:

C++
Python

Copy
Copied!

            
            std::cout << operator_object->spec()->description() << std::endl;

Copy
Copied!

            
            print(operator_object.spec)

Note

The Holoscan SDK uses meta-programming with templating and std::any to support arbitrary data types. Because of this, some type information (and therefore values) might not be retrievable by the description API. If more details are needed, we recommend inspecting the list of Parameter members in the operator header to identify their type.

Extensions

The Holoscan SDK also includes some GXF extensions with GXF codelets, which are typically wrapped as operators, or present for legacy reasons. In addition to the core GXF extensions (std, cuda, serialization, multimedia) listed here, the Holoscan SDK includes the following GXF extensions:

bayer_demosaic
gxf_holoscan_wrapper
opengl
stream_playback
tensor_rt
v4l2

Bayer Demosaic

The bayer_demosaic extension includes the nvidia::holoscan::BayerDemosaic codelet. It performs color filter array (CFA) interpolation for 1-channel inputs of 8 or 16-bit unsigned integer and outputs an RGB or RGBA image. It is wrapped by the nvidia::holoscan::ops::BayerDemosaicOp operator.

Note

The BayerDemosaicOp will be converted to a native operator in future releases.

GXF Holoscan Wrapper

The gxf_holoscan_wrapper extension includes the holoscan::gxf::OperatorWrapper codelet. It is used as a utility base class to wrap a holoscan operator to interface with the GXF framework.

Learn more about it in the Using Holoscan Operators in GXF Applications section.

OpenGL

The opengl_renderer extension includes the nvidia::holoscan::OpenGLRenderer codelet. It displays a VideoBuffer, leveraging OpenGL/CUDA interop.

Note

There is no operator currently wrapping this codelet. It is only use to demonstrate the V4L2 example.

Warning

This codelet is deprecated, and will be removed in a future release in favor of a native operator using the visualization module.

Parameter	Description	Type
signal	Input Channel	`gxf::Handle<gxf::Receiver>`
width	Width of the rendering window	`unsigned int`
height	Height of the rendering window	`unsigned int`
window_close_scheduling_term	`BooleanSchedulingTerm` to stop the codelet from ticking after all messages are published	`gxf::Handle<gxf::BooleanSchedulingTerm>`

Stream Playback

The stream_playback extension includes the nvidia::holoscan::stream_playback::VideoStreamSerializer entity serializer to/from a Tensor Object. This extension does not include any codelets: reading and writing video stream (gxf entity files) from the disk was implemented as native operators with VideoStreamRecorderOp and VideoStreamReplayerOp, though they leverage the VideoStreamSerializer from this extension.

Note

The VideoStreamSerializer codelet is based on the nvidia::gxf::StdEntitySerializer with the addition of a repeat feature. (If the repeat parameter is true and the frame count is out of the maximum frame index, unnecessary warning messages are printed with nvidia::gxf::StdEntitySerializer.)

TensorRT

The tensor_rt extension includes the nvidia::holoscan::TensorRtInference codelet. It takes input tensors and feeds them into TensorRT for inference. It is wrapped by the nvidia::holoscan::ops::TensorRTInferenceOp operator.

Note

This codelet is based on nvidia::gxf::TensorRtInference (by GXF), with the addition of the engine_cache_dir to be able to provide a directory of engine files for multiple GPUs instead of a single one.

Warning

This codelet is deprecated, and will be removed in Holoscan 0.6 in favor of a native operator using the inference module.

V4L2

The v4l2_source extension includes the nvidia::holoscan::V4L2Source codelet. It uses V4L2 to get image frames from USB cameras. The output is a VideoBuffer object.

Note

There is no operator currently wrapping this codelet. A native operator also supporting HDMI IN will replace this codelet in future releases,

Parameter	Description	Type	Default
signal	Output channel	`gxf::Handle<gxf::Transmitter>`
allocator	Output Allocator	`gxf::Handle<gxf::Allocator>`
device	Path to the V4L2 device	`std::string`	`/dev/video0`
width	Width of the V4L2 image	`uint32_t`	640
height	Height of the V4L2 image	`uint32_t`	480
numBuffers	Number of V4L2 buffers to use	`uint32_t`	2

HoloHub

Visit the HoloHub repository to find a collection of additional Holoscan operators and extensions.