Clara Holoscan v0.2.0
NVIDIA Docs Hub  NVIDIA Holoscan  Clara Holoscan v0.2.0  SDK Extensions
1.0

SDK Extensions

GXF Standard Extensions

Std

The GXF::std extension provides the most commonly used interfaces and components in Gxf Core.

nvidia::gxf::Broadcast

Messages arrived on the input channel are distributed to all transmitters.

Parameters

  • source: Source channel

    • type: Handle<Receiver>

  • mode: The broadcast mode. Can be Broadcast or RoundRobin (default: 0)

    • type: BroadcastMode

    • value:

      • 0: Broadcast mode. Publishes income message to all transmitters

      • 1: RoundRobin mode. Publishes income message to one of the transmitters in round-robin fashion

Serialization

The GXF::serialization extension provides components for serializing messages.

nvidia::gxf::EntityRecorder

Serializes incoming messages and writes them to a file.

Parameters

  • receiver: Receiver channel to log

    • type: Handle<Receiver>

  • entity_serializer: Serializer for serializing entities

    • type: Handle<EntitySerializer>

  • directory: Directory path for storing files

    • type: std::string

  • basename: User specified file name without extension (optional)

    • type: std::string

  • flush_on_tick: Flushes output buffer on every tick when true (default: false)

    • type: bool

Holoscan SDK GXF Extensions

V4L2

The v4l2_source extension provides a codelet for a realtime Video for Linux 2 source supporting USB cameras and other media inputs. The output is a VideoBuffer object.

nvidia::holoscan::V4L2Source

V4L2 Source Codelet.

Parameters

  • signal: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • allocator: Output Allocator

    • type: gxf::Handle<gxf::Allocator>

  • device: Path to the V4L2 device (default: /dev/video0)

    • type: std::string

  • width: Width of the V4L2 image (default: 640)

    • type: uint32_t

  • height: Height of the V4L2 image (default: 480)

    • type: uint32_t

  • numBuffers: Number of V4L2 buffers to use (default: 2)

    • type: uint32_t

AJA

The aja_source extension provides a codelet for supporting AJA capture card as a source. It offers support for GPUDirect-RDMA on Quadro GPUs. The output is a VideoBuffer object.

nvidia::holoscan::AJASource

AJA Source Codelet.

Parameters

  • signal: Output signal

    • type: gxf::Handle<gxf::Transmitter>

  • device: Device specifier (default: 0)

    • type: std::string

  • channel: NTV2Channel to use (default: 0 (NTV2_CHANNEL1))

    • type: NTV2Channel

  • width: Width of the stream (default: 1920)

    • type: uint32_t

  • height: Height of the stream (default: 1080)

    • type: uint32_t

  • framerate: Framerate of the stream (default: 60)

    • type: uint32_t

  • rdma: Enable RDMA (default: false)

    • type: bool

Stream Playback

The stream_playback extension provides components for the video stream playback module to output video frames as a Tensor object.

nvidia::holoscan::stream_playback::VideoStreamReplayer

VideoStreamReplayer codelet.

Parameters

  • transmitter: Transmitter channel for replaying entities

    • type: gxf::Handle<gxf::Transmitter>

  • entity_serializer: Serializer for serializing entities

    • type: gxf::Handle<gxf::EntitySerializer>

  • boolean_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

  • directory: Directory path for storing files

    • type: std::string

  • basename: User specified file name without extension (optional)

    • type: std::string

  • batch_size: Number of entities to read and publish for one tick (default: 1)

    • type: size_t

  • ignore_corrupted_entities: If an entity could not be deserialized, it is ignored by default; otherwise a failure is generated (default: true)

    • type: bool

  • frame_rate: Frame rate to replay. If zero value is specified, it follows timings in timestamps (default: 0.f)

    • type: float

  • realtime: Playback video in realtime, based on frame_rate or timestamps (default: true)

    • type: bool

  • repeat: Repeat video stream (default: false)

    • type: bool

  • count: Number of frame counts to playback. If zero value is specified, it is ignored. If the count is less than the number of frames in the video, it would be finished early (default: 0)

    • type: uint64_t

nvidia::holoscan::stream_playback::VideoStreamSerializer

The VideoStreamSerializer codelet is based on the nvidia::gxf::StdEntitySerializer with the addition of a repeat feature. (If the repeat parameter is true and the frame count is out of the maximum frame index, unnecessary warning messages are printed with nvidia::gxf::StdEntitySerializer.)

Format Converter

The format_converter extension includes a codelet that provides common video or tensor operations in inference pipelines to change datatypes, resize images, reorder channels, and normalize and scale values.

nvidia::holoscan::formatconverter::FormatConverter

This codelet executes the following processes:

  • Resize the input image before converting data type

    • if resize_width > 0 && resize_height > 0

  • Adjust output shape if the conversion involves the change in the channel dimension

    • if format conversion is one of the following:

      • rgb888 to rgba8888 (out channels: 4)

      • rgba8888 to rgb888 (out channels: 3)

      • rgba8888 to float32 (out channels: 3)

  • Convert data type

    • The following conversions are supported:

      • ""(None): if in_dtype and out_dtype are the same

        • dst_order (default: [0,1,2] or [0,1,2,3] depending on out_dtype) needs to be set

      • uint8(rgb888) to float32

        • scale_min and scale_max need to be set

        • dst_order (default: [0,1,2]) needs to be set

      • float32 to uint8(rgb888)

        • scale_min and scale_max need to be set

        • dst_order (default: [0,1,2]) needs to be set

      • uint8(rgb888) to rgba8888

        • dst_order (default: [0,1,2,3]) and alpha_value (default: 255) need to be set

      • rgba8888 to uint8(rgb888)

        • dst_order (default: [0,1,2]) needs to be set

Parameters

  • in: Input channel

    • type: gxf::Handle<gxf::Receiver>

  • in_tensor_name: Name of the input tensor (default: "")

    • type: std::string

  • in_dtype: Source data type (default: "")

    • type: std::string

    • If not specified, input data type is guessed from the input tensor.

  • out: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • out_tensor_name: Name of the output tensor (default: "")

    • type: std::string

  • out_dtype: Destination data type

    • type: std::string

  • scale_min: Minimum value of the scale (default: 0.f)

    • type: float

  • scale_max: Maximum value of the scale (default: 1.f)

    • type: float

  • alpha_value: Alpha value that can be used to fill the alpha channel when converting RGB888 to RGBA8888 (default: 255)

    • type: uint8_t

  • resize_width: Width for resize. No actions if this value is zero (default: 0)

    • type: int32_t

  • resize_height: Height for resize. No actions if this value is zero (default: 0)

    • type: int32_t

  • resize_mode: Mode for resize. 4 (NPPI_INTER_CUBIC) if this value is zero (default: 0)

    • type: int32_t

      Copy
      Copied!
                  
      
            
                   0 = NPPI_INTER_CUBIC
             1 = NPPI_INTER_NN                  /**<  Nearest neighbor filtering. */
             2 = NPPI_INTER_LINEAR              /**<  Linear interpolation. */
             4 = NPPI_INTER_CUBIC               /**<  Cubic interpolation. */
             5 = NPPI_INTER_CUBIC2P_BSPLINE     /**<  Two-parameter cubic filter (B=1, C=0) */
             6 = NPPI_INTER_CUBIC2P_CATMULLROM  /**<  Two-parameter cubic filter (B=0, C=1/2) */
             7 = NPPI_INTER_CUBIC2P_B05C03,     /**<  Two-parameter cubic filter (B=1/2, C=3/10) */
             8 = NPPI_INTER_SUPER               /**<  Super sampling. */
            16 = NPPI_INTER_LANCZOS            /**<  Lanczos filtering. */
            17 = NPPI_INTER_LANCZOS3_ADVANCED  /**<  Generic Lanczos filtering with order 3. */
(int)0x8000000 = NPPI_SMOOTH_EDGE              /**<  Smooth edge filtering. (0x8000000 = 134217728)*/

  • out_channel_order: Host memory integer array describing how channel values are permutated (default: [])

    • type: std::vector<int>

  • pool: Pool to allocate the output message

    • type: gxf::Handle<gxf::Allocator>

TensorRT

The tensor_rt extension provides the TensorRT inference codelet.

nvidia::holoscan::TensorRtInference

Codelet taking input tensors and feeding them into TensorRT for inference. Based on nvidia::gxf::TensorRtInference, with the addition of the engine_cache_dir to be able to provide a directory of engine files for multiple GPUs instead of a single one.

Parameters

  • model_file_path: Path to ONNX model to be loaded

    • type: std::string

  • engine_cache_dir: Path to a directory containing cached generated engines to be serialized and loaded from

    • type: std::string

  • plugins_lib_namespace: Namespace used to register all the plugins in this library (default: "")

    • type: std::string

  • force_engine_update: Always update engine regard less of existing engine file. Such conversion may take minutes (default: false)

    • type: bool

  • input_tensor_names: Names of input tensors in the order to be fed into the model

    • type: std::vector<std::string>

  • input_binding_names: Names of input bindings as in the model in the same order of what is provided in input_tensor_names

    • type: std::vector<std::string>

  • output_tensor_names: Names of output tensors in the order to be retrieved from the model

    • type: std::vector<std::string>

  • output_binding_names: Names of output bindings in the model in the same order of of what is provided in output_tensor_names

    • type: std::vector<std::string>

  • pool: Allocator instance for output tensors

    • type: gxf::Handle<gxf::Allocator>

  • cuda_stream_pool: Instance of gxf::CudaStreamPool to allocate CUDA stream

    • type: gxf::Handle<gxf::CudaStreamPool>

  • max_workspace_size: Size of working space in bytes (default: 67108864 (64MB))

    • type: int64_t

  • dla_core: DLA Core to use. Fallback to GPU is always enabled. Default to use GPU only (optional)

    • type: int64_t

  • max_batch_size: Maximum possible batch size in case the first dimension is dynamic and used as batch size (default: 1)

    • type: int32_t

  • enable_fp16_: Enable inference with FP16 and FP32 fallback (default: false)

    • type: bool

  • verbose: Enable verbose logging on console (default: false)

    • type: bool

  • relaxed_dimension_check: Ignore dimensions of 1 for input tensor dimension check (default: true)

    • type: bool

  • clock: Instance of clock for publish time (optional)

    • type: gxf::Handle<gxf::Clock>

  • rx: List of receivers to take input tensors

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • tx: Transmitter to publish output tensors

    • type: gxf::Handle<gxf::Transmitter>

OpenGL

The opengl_renderer extension provides a codelet that displays a VideoBuffer, leveraging OpenGL/CUDA interop.

nvidia::holoscan::OpenGLRenderer

OpenGL Renderer Codelet.

Parameters

  • signal: Input Channel

    • type: gxf::Handle<gxf::Receiver>

  • width: Width of the rendering window

    • type: unsigned int

  • height: Height of the rendering window

    • type: unsigned int

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

Segmentation Post Processor

The segmentation_postprocessor extension provides a codelet that converts inference output to the highest-probability class index, including support for sigmoid, softmax, and activations.

nvidia::holoscan::segmentation_postprocessor::Postprocessor

Segmentation Postprocessor codelet.

Parameters

  • in: Input channel

    • type: gxf::Handle<gxf::Receiver>

  • in_tensor_name: Name of the input tensor (default: "")

    • type: std::string

  • network_output_type: Network output type (default: softmax)

    • type: std::string

  • out: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • data_format: Data format of network output (default: hwc)

    • type: std::string

  • allocator: Output Allocator

    • type: gxf::Handle<gxf::Allocator>

Segmentation Visualizer

The segmentation_visualizer extension provides an OpenGL renderer codelet that combines segmentation output overlayed on video input, using CUDA/OpenGL interop.

nvidia::holoscan::segmentation_visualizer::Visualizer

OpenGL Segmentation Visualizer codelet.

Parameters

  • image_in: Tensor input

    • type: gxf::Handle<gxf::Receiver>

  • image_width: Width of the input image (default: 1920)

    • type: int32_t

  • image_height: Height of the input image (default: 1080)

    • type: int32_t

  • class_index_in: Tensor input

    • type: gxf::Handle<gxf::Receiver>

  • class_index_width: Width of the segmentation class index tensor (default: 1920)

    • type: int32_t

  • class_index_height: Height of the segmentation class index tensor (default: 1080)

    • type: int32_t

  • class_color_lut: Overlay Image Segmentation Class Colormap

    • type: std::vector<std::vector<float>>

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

Custom LSTM Inference

The custom_lstm_inference extension provides LSTM (Long-Short Term Memory) stateful inference module using TensorRT.

nvidia::holoscan::custom_lstm_inference::TensorRtInference

Codelet, taking input tensors and feeding them into TensorRT for LSTM inference.

This implementation is based on nvidia::gxf::TensorRtInference. input_state_tensor_names and output_state_tensor_names parameters are added to specify tensor names for states in LSTM model.

Parameters

  • model_file_path: Path to ONNX model to be loaded

    • type: std::string

  • engine_cache_dir: Path to a directory containing cached generated engines to be serialized and loaded from

    • type: std::string

  • plugins_lib_namespace: Namespace used to register all the plugins in this library (default: "")

    • type: std::string

  • force_engine_update: Always update engine regard less of existing engine file. Such conversion may take minutes (default: false)

    • type: bool

  • input_tensor_names: Names of input tensors in the order to be fed into the model

    • type: std::vector<std::string>

  • input_state_tensor_names: Names of input state tensors that are used internally by TensorRT

    • type: std::vector<std::string>

  • input_binding_names: Names of input bindings as in the model in the same order of what is provided in input_tensor_names

    • type: std::vector<std::string>

  • output_tensor_names: Names of output tensors in the order to be retrieved from the model

    • type: std::vector<std::string>

  • input_state_tensor_names: Names of output state tensors that are used internally by TensorRT

    • type: std::vector<std::string>

  • output_binding_names: Names of output bindings in the model in the same order of of what is provided in output_tensor_names

    • type: std::vector<std::string>

  • pool: Allocator instance for output tensors

    • type: gxf::Handle<gxf::Allocator>

  • cuda_stream_pool: Instance of gxf::CudaStreamPool to allocate CUDA stream

    • type: gxf::Handle<gxf::CudaStreamPool>

  • max_workspace_size: Size of working space in bytes (default: 67108864l (64MB))

    • type: int64_t

  • dla_core: DLA Core to use. Fallback to GPU is always enabled. Default to use GPU only (optional)

    • type: int64_t

  • max_batch_size: Maximum possible batch size in case the first dimension is dynamic and used as batch size (default: 1)

    • type: int32_t

  • enable_fp16_: Enable inference with FP16 and FP32 fallback (default: false)

    • type: bool

  • verbose: Enable verbose logging on console (default: false)

    • type: bool

  • relaxed_dimension_check: Ignore dimensions of 1 for input tensor dimension check (default: true)

    • type: bool

  • clock: Instance of clock for publish time (optional)

    • type: gxf::Handle<gxf::Clock>

  • rx: List of receivers to take input tensors

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • tx: Transmitter to publish output tensors

    • type: gxf::Handle<gxf::Transmitter>

Visualizer Tool Tracking

The visualizer_tool_tracking extension provides a custom visualizer codelet that handles compositing, blending, and visualization of tool labels, tips, and masks given the output tensors of the custom_lstm_inference.

nvidia::holoscan::visualizer_tool_tracking::Sink

Surgical Tool Tracking Viz codelet.

Parameters

  • videoframe_vertex_shader_path: Path to vertex shader to be loaded

    • type: std::string

  • videoframe_fragment_shader_path: Path to fragment shader to be loaded

    • type: std::string

  • tooltip_vertex_shader_path: Path to vertex shader to be loaded

    • type: std::string

  • tooltip_fragment_shader_path: Path to fragment shader to be loaded

    • type: std::string

  • num_tool_classes: Number of different tool classes

    • type: int32_t

  • num_tool_pos_components: Number of components of the tool position vector (default: 2)

    • type: int32_t

  • tool_tip_colors: Color of the tool tips, a list of RGB values with components between 0 and 1 (default: 12 qualitative classes color scheme from colorbrewer2)

    • type: std::vector<std::vector<float>>

  • overlay_img_vertex_shader_path: Path to vertex shader to be loaded

    • type: std::string

  • overlay_img_fragment_shader_path: Path to fragment shader to be loaded

    • type: std::string

  • overlay_img_width: Width of overlay image

    • type: int32_t

  • overlay_img_height: Height of overlay image

    • type: int32_t

  • overlay_img_channels: Number of Overlay Image Channels

    • type: int32_t

  • overlay_img_layers: Number of Overlay Image Layers

    • type: int32_t

  • overlay_img_colors: Color of the image overlays, a list of RGB values with components between 0 and 1 (default: 12 qualitative classes color scheme from colorbrewer2)

    • type: std::vector<std::vector<float>>

  • tool_labels: List of tool names (default: [])

    • type: std::vector<std::string>

  • label_sans_font_path: Path for sans font to be loaded

    • type: std::string

  • label_sans_bold_font_path: Path for sans bold font to be loaded

    • type: std::string

  • in: List of input channels

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • in_tensor_names: Names of input tensors (default: "")

    • type: std::vector<std::string>

  • in_width: Width of the image (default: 640)

    • type: int32_t

  • in_height: Height of the image (default: 480)

    • type: int32_t

  • in_channels: Number of channels (default: 3)

    • type: int16_t

  • in_bytes_per_pixel: Number of bytes per pixel of the image (default: 1)

    • type: uint8_t

  • alpha_value: Alpha value that can be used when converting RGB888 to RGBA8888 (default: 255)

    • type: uint8_t

  • pool: Pool to allocate the output message.

    • type: gxf::Handle<gxf::Allocator>

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published.

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

Holoscan Test Mock

The mocks extension provides mock codelets that can be used for testing GXF applications.

nvidia::holoscan::test::VideoBufferMock

VideoBuffer Mock codelet. It creates RGB strips as an output message of gxf::VideoBuffer type to mimic the output of AJA extension.

Parameters

  • in_width: Width of the image (default: 640)

    • type: int32_t

  • in_height: Height of the image (default: 480)

    • type: int32_t

  • in_channels: Number of input channels (default: 3)

    • type: int16_t

  • in_bytes_per_pixel: Number of bytes per pixel of the image (default: 1)

    • type: int8_t

  • out_tensor_name: Name of the output tensor (default: "")

    • type: std::string

  • out: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • pool: Pool to allocate the output message

    • type: gxf::Handle<gxf::Allocator>
© Copyright 2022, NVIDIA. Last updated on Jun 28, 2023.
content here