Built-in Operators and Extensions
The units of work of Holoscan applications are implemented within Operators, as described in the core concepts of the SDK. The operators included in the SDK provide domain-agnostic functionalities such as IO, machine learning inference, processing, and visualization, optimized for AI streaming pipelines, relying on a set of Core Technologies.
The operators below are defined under the holoscan::ops
namespace for C++ and CMake, and under the holoscan.operators
module in Python.
Class |
CMake target/lib |
Documentation |
---|---|---|
AJASourceOp |
|
|
BayerDemosaicOp |
|
|
FormatConverterOp |
|
|
HolovizOp |
|
|
MultiAIInferenceOp |
|
|
MultiAIPostprocessorOp |
|
|
PingRxOp |
|
|
PingTxOp |
|
|
SegmentationPostprocessorOp |
|
|
TensorRTInferenceOp † |
|
|
VideoStreamRecorderOp |
|
|
VideoStreamReplayerOp |
|
† deprecated
Given an instance of an operator class, you can print a human-readable description of its specification to inspect the inputs, outputs, and parameters that can be configured on that operator class:
std::cout << operator_object->spec()->description() << std::endl;
print(operator_object.spec)
The Holoscan SDK uses meta-programming with templating and std::any
to support arbitrary data types. Because of this, some type information (and therefore values) might not be retrievable by the description
API. If more details are needed, we recommend inspecting the list of Parameter
members in the operator header to identify their type.
The Holoscan SDK also includes some GXF extensions with GXF codelets, which are typically wrapped as operators, or present for legacy reasons. In addition to the core GXF extensions (std, cuda, serialization, multimedia) listed here, the Holoscan SDK includes the following GXF extensions:
Bayer Demosaic
The bayer_demosaic
extension includes the nvidia::holoscan::BayerDemosaic
codelet. It performs color filter array (CFA) interpolation for 1-channel inputs of 8 or 16-bit unsigned integer and outputs an RGB or RGBA image. It is wrapped by the nvidia::holoscan::ops::BayerDemosaicOp
operator.
The BayerDemosaicOp
will be converted to a native operator in future releases.
GXF Holoscan Wrapper
The gxf_holoscan_wrapper
extension includes the holoscan::gxf::OperatorWrapper
codelet. It is used as a utility base class to wrap a holoscan operator to interface with the GXF framework.
Learn more about it in the Using Holoscan Operators in GXF Applications section.
OpenGL
The opengl_renderer
extension includes the nvidia::holoscan::OpenGLRenderer
codelet. It displays a VideoBuffer, leveraging OpenGL/CUDA interop.
There is no operator currently wrapping this codelet. It is only use to demonstrate the V4L2 example.
This codelet is deprecated, and will be removed in a future release in favor of a native operator using the visualization module.
Parameter |
Description |
Type |
---|---|---|
signal |
Input Channel |
|
width |
Width of the rendering window |
|
height |
Height of the rendering window |
|
window_close_scheduling_term |
|
|
Stream Playback
The stream_playback
extension includes the nvidia::holoscan::stream_playback::VideoStreamSerializer
entity serializer to/from a Tensor Object.
This extension does not include any codelets: reading and writing video stream (gxf entity files) from the disk was implemented as native operators with VideoStreamRecorderOp
and VideoStreamReplayerOp
, though they leverage the VideoStreamSerializer
from this extension.
The VideoStreamSerializer
codelet is based on the nvidia::gxf::StdEntitySerializer
with the addition of a repeat
feature.
(If the repeat
parameter is true
and the frame count is out of the maximum frame index, unnecessary warning messages are printed with nvidia::gxf::StdEntitySerializer
.)
TensorRT
The tensor_rt
extension includes the nvidia::holoscan::TensorRtInference
codelet. It takes input tensors and feeds them into TensorRT for inference. It is wrapped by the nvidia::holoscan::ops::TensorRTInferenceOp
operator.
This codelet is based on nvidia::gxf::TensorRtInference
(by GXF), with the addition of the engine_cache_dir
to be able to provide a directory of engine files for multiple GPUs instead of a single one.
This codelet is deprecated, and will be removed in Holoscan 0.6 in favor of a native operator using the inference module.
V4L2
The v4l2_source
extension includes the nvidia::holoscan::V4L2Source
codelet. It uses V4L2 to get image frames from USB cameras. The output is a VideoBuffer object.
There is no operator currently wrapping this codelet. A native operator also supporting HDMI IN will replace this codelet in future releases,
Parameter |
Description |
Type |
Default |
---|---|---|---|
signal |
Output channel |
|
|
allocator |
Output Allocator |
|
|
device |
Path to the V4L2 device |
|
|
width |
Width of the V4L2 image |
|
640 |
height |
Height of the V4L2 image |
|
480 |
numBuffers |
Number of V4L2 buffers to use |
|
2 |
HoloHub
Visit the HoloHub repository to find a collection of additional Holoscan operators and extensions.