Holoscan GXF Extensions
NVIDIA Docs Hub  NVIDIA Holoscan  NVIDIA Holoscan SDK v0.4.0  Holoscan GXF Extensions

GXF Built-in Extensions

Std

The GXF::std extension provides the most commonly used interfaces and components in Gxf Core.

Please see GXF Standard Extension for more details.

nvidia::gxf::Broadcast

Messages arrived on the input channel are distributed to all transmitters.

Parameters

  • source: Source channel

    • type: Handle<Receiver>

  • mode: The broadcast mode. Can be Broadcast or RoundRobin (default: 0)

    • type: BroadcastMode

    • value:

      • 0: Broadcast mode. Publishes income message to all transmitters

      • 1: RoundRobin mode. Publishes income message to one of the transmitters in round-robin fashion

Serialization

The GXF::serialization extension provides components for serializing messages.

Please see GXF Serialization Extension for more details.

nvidia::gxf::EntityRecorder

Serializes incoming messages and writes them to a file.

Parameters

  • receiver: Receiver channel to log

    • type: Handle<Receiver>

  • entity_serializer: Serializer for serializing entities

    • type: Handle<EntitySerializer>

  • directory: Directory path for storing files

    • type: std::string

  • basename: User specified file name without extension (optional)

    • type: std::string

  • flush_on_tick: Flushes output buffer on every tick when true (default: false)

    • type: bool

Holoscan SDK GXF Extensions

V4L2

The v4l2_source extension provides a codelet for a realtime Video for Linux 2 source supporting USB cameras and other media inputs. The output is a VideoBuffer object.

nvidia::holoscan::V4L2Source

V4L2 Source Codelet.

Parameters

  • signal: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • allocator: Output Allocator

    • type: gxf::Handle<gxf::Allocator>

  • device: Path to the V4L2 device (default: /dev/video0)

    • type: std::string

  • width: Width of the V4L2 image (default: 640)

    • type: uint32_t

  • height: Height of the V4L2 image (default: 480)

    • type: uint32_t

  • numBuffers: Number of V4L2 buffers to use (default: 2)

    • type: uint32_t

AJA

The aja_source extension provides a codelet for supporting AJA capture card as a source. It offers support for GPUDirect-RDMA on Quadro/NVIDIA RTX GPUs. The output is a VideoBuffer object.

nvidia::holoscan::AJASource

AJA Source Codelet.

Parameters

  • signal: Output signal

    • type: gxf::Handle<gxf::Transmitter>

  • device: Device specifier (default: 0)

    • type: std::string

  • channel: NTV2Channel to use (default: 0 (NTV2_CHANNEL1))

    • type: NTV2Channel

  • width: Width of the stream (default: 1920)

    • type: uint32_t

  • height: Height of the stream (default: 1080)

    • type: uint32_t

  • framerate: Framerate of the stream (default: 60)

    • type: uint32_t

  • rdma: Enable RDMA (default: false)

    • type: bool

Stream Playback

The stream_playback extension provides components for the video stream playback module to output video frames as a Tensor object.

nvidia::holoscan::stream_playback::VideoStreamReplayer

VideoStreamReplayer codelet.

Parameters

  • transmitter: Transmitter channel for replaying entities

    • type: gxf::Handle<gxf::Transmitter>

  • entity_serializer: Serializer for serializing entities

    • type: gxf::Handle<gxf::EntitySerializer>

  • boolean_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

  • directory: Directory path for storing files

    • type: std::string

  • basename: User specified file name without extension (optional)

    • type: std::string

  • batch_size: Number of entities to read and publish for one tick (default: 1)

    • type: size_t

  • ignore_corrupted_entities: If an entity could not be deserialized, it is ignored by default; otherwise a failure is generated (default: true)

    • type: bool

  • frame_rate: Frame rate to replay. If zero value is specified, it follows timings in timestamps (default: 0.f)

    • type: float

  • realtime: Playback video in realtime, based on frame_rate or timestamps (default: true)

    • type: bool

  • repeat: Repeat video stream (default: false)

    • type: bool

  • count: Number of frame counts to playback. If zero value is specified, it is ignored. If the count is less than the number of frames in the video, it would be finished early (default: 0)

    • type: uint64_t

nvidia::holoscan::stream_playback::VideoStreamSerializer

The VideoStreamSerializer codelet is based on the nvidia::gxf::StdEntitySerializer with the addition of a repeat feature. (If the repeat parameter is true and the frame count is out of the maximum frame index, unnecessary warning messages are printed with nvidia::gxf::StdEntitySerializer.)

Format Converter

The format_converter extension includes a codelet that provides common video or tensor operations in inference pipelines to change datatypes, resize images, reorder channels, and normalize and scale values.

nvidia::holoscan::formatconverter::FormatConverter

This codelet executes the following processes:

  • Resize the input image before converting data type

    • if resize_width > 0 && resize_height > 0

  • Adjust output shape if the conversion involves the change in the channel dimension

    • if format conversion is one of the following:

      • rgb888 to rgba8888 (out channels: 4)

      • rgba8888 to rgb888 (out channels: 3)

      • rgba8888 to float32 (out channels: 3)

  • Convert data type

    • The following conversions are supported:

      • ""(None): if in_dtype and out_dtype are the same

        • dst_order (default: [0,1,2] or [0,1,2,3] depending on out_dtype) needs to be set

      • uint8(rgb888) to float32

        • scale_min and scale_max need to be set

        • dst_order (default: [0,1,2]) needs to be set

      • float32 to uint8(rgb888)

        • scale_min and scale_max need to be set

        • dst_order (default: [0,1,2]) needs to be set

      • uint8(rgb888) to rgba8888

        • dst_order (default: [0,1,2,3]) and alpha_value (default: 255) need to be set

      • rgba8888 to uint8(rgb888)

        • dst_order (default: [0,1,2]) needs to be set

Parameters

  • in: Input channel

    • type: gxf::Handle<gxf::Receiver>

  • in_tensor_name: Name of the input tensor (default: "")

    • type: std::string

  • in_dtype: Source data type (default: "")

    • type: std::string

    • If not specified, input data type is guessed from the input tensor.

  • out: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • out_tensor_name: Name of the output tensor (default: "")

    • type: std::string

  • out_dtype: Destination data type

    • type: std::string

  • scale_min: Minimum value of the scale (default: 0.f)

    • type: float

  • scale_max: Maximum value of the scale (default: 1.f)

    • type: float

  • alpha_value: Alpha value that can be used to fill the alpha channel when converting RGB888 to RGBA8888 (default: 255)

    • type: uint8_t

  • resize_width: Width for resize. No actions if this value is zero (default: 0)

    • type: int32_t

  • resize_height: Height for resize. No actions if this value is zero (default: 0)

    • type: int32_t

  • resize_mode: Mode for resize. 4 (NPPI_INTER_CUBIC) if this value is zero (default: 0)

    • type: int32_t

      Copy
      Copied!
                  
      
            
                   0 = NPPI_INTER_CUBIC
             1 = NPPI_INTER_NN                  /**<  Nearest neighbor filtering. */
             2 = NPPI_INTER_LINEAR              /**<  Linear interpolation. */
             4 = NPPI_INTER_CUBIC               /**<  Cubic interpolation. */
             5 = NPPI_INTER_CUBIC2P_BSPLINE     /**<  Two-parameter cubic filter (B=1, C=0) */
             6 = NPPI_INTER_CUBIC2P_CATMULLROM  /**<  Two-parameter cubic filter (B=0, C=1/2) */
             7 = NPPI_INTER_CUBIC2P_B05C03,     /**<  Two-parameter cubic filter (B=1/2, C=3/10) */
             8 = NPPI_INTER_SUPER               /**<  Super sampling. */
            16 = NPPI_INTER_LANCZOS            /**<  Lanczos filtering. */
            17 = NPPI_INTER_LANCZOS3_ADVANCED  /**<  Generic Lanczos filtering with order 3. */
(int)0x8000000 = NPPI_SMOOTH_EDGE              /**<  Smooth edge filtering. (0x8000000 = 134217728)*/

  • out_channel_order: Host memory integer array describing how channel values are permutated (default: [])

    • type: std::vector<int>

  • pool: Pool to allocate the output message

    • type: gxf::Handle<gxf::Allocator>

TensorRT

The tensor_rt extension provides the TensorRT inference codelet.

nvidia::holoscan::TensorRtInference

Codelet taking input tensors and feeding them into TensorRT for inference. Based on nvidia::gxf::TensorRtInference, with the addition of the engine_cache_dir to be able to provide a directory of engine files for multiple GPUs instead of a single one.

Parameters

  • model_file_path: Path to ONNX model to be loaded

    • type: std::string

  • engine_cache_dir: Path to a directory containing cached generated engines to be serialized and loaded from

    • type: std::string

  • plugins_lib_namespace: Namespace used to register all the plugins in this library (default: "")

    • type: std::string

  • force_engine_update: Always update engine regard less of existing engine file. Such conversion may take minutes (default: false)

    • type: bool

  • input_tensor_names: Names of input tensors in the order to be fed into the model

    • type: std::vector<std::string>

  • input_binding_names: Names of input bindings as in the model in the same order of what is provided in input_tensor_names

    • type: std::vector<std::string>

  • output_tensor_names: Names of output tensors in the order to be retrieved from the model

    • type: std::vector<std::string>

  • output_binding_names: Names of output bindings in the model in the same order of of what is provided in output_tensor_names

    • type: std::vector<std::string>

  • pool: Allocator instance for output tensors

    • type: gxf::Handle<gxf::Allocator>

  • cuda_stream_pool: Instance of gxf::CudaStreamPool to allocate CUDA stream

    • type: gxf::Handle<gxf::CudaStreamPool>

  • max_workspace_size: Size of working space in bytes (default: 67108864 (64MB))

    • type: int64_t

  • dla_core: DLA Core to use. Fallback to GPU is always enabled. Default to use GPU only (optional)

    • type: int64_t

  • max_batch_size: Maximum possible batch size in case the first dimension is dynamic and used as batch size (default: 1)

    • type: int32_t

  • enable_fp16_: Enable inference with FP16 and FP32 fallback (default: false)

    • type: bool

  • verbose: Enable verbose logging on console (default: false)

    • type: bool

  • relaxed_dimension_check: Ignore dimensions of 1 for input tensor dimension check (default: true)

    • type: bool

  • clock: Instance of clock for publish time (optional)

    • type: gxf::Handle<gxf::Clock>

  • rx: List of receivers to take input tensors

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • tx: Transmitter to publish output tensors

    • type: gxf::Handle<gxf::Transmitter>

OpenGL

The opengl_renderer extension provides a codelet that displays a VideoBuffer, leveraging OpenGL/CUDA interop.

nvidia::holoscan::OpenGLRenderer

OpenGL Renderer Codelet.

Parameters

  • signal: Input Channel

    • type: gxf::Handle<gxf::Receiver>

  • width: Width of the rendering window

    • type: unsigned int

  • height: Height of the rendering window

    • type: unsigned int

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

Segmentation Post Processor

The segmentation_postprocessor extension provides a codelet that converts inference output to the highest-probability class index, including support for sigmoid, softmax, and activations.

nvidia::holoscan::segmentation_postprocessor::Postprocessor

Segmentation Postprocessor codelet.

Parameters

  • in: Input channel

    • type: gxf::Handle<gxf::Receiver>

  • in_tensor_name: Name of the input tensor (default: "")

    • type: std::string

  • network_output_type: Network output type (default: softmax)

    • type: std::string

  • out: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • data_format: Data format of network output (default: hwc)

    • type: std::string

  • allocator: Output Allocator

    • type: gxf::Handle<gxf::Allocator>

Segmentation Visualizer

The segmentation_visualizer extension provides an OpenGL renderer codelet that combines segmentation output overlaid on video input, using CUDA/OpenGL interop.

nvidia::holoscan::segmentation_visualizer::Visualizer

OpenGL Segmentation Visualizer codelet.

Parameters

  • image_in: Tensor input

    • type: gxf::Handle<gxf::Receiver>

  • image_width: Width of the input image (default: 1920)

    • type: int32_t

  • image_height: Height of the input image (default: 1080)

    • type: int32_t

  • class_index_in: Tensor input

    • type: gxf::Handle<gxf::Receiver>

  • class_index_width: Width of the segmentation class index tensor (default: 1920)

    • type: int32_t

  • class_index_height: Height of the segmentation class index tensor (default: 1080)

    • type: int32_t

  • class_color_lut: Overlay Image Segmentation Class Colormap

    • type: std::vector<std::vector<float>>

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

Custom LSTM Inference

The custom_lstm_inference extension provides LSTM (Long-Short Term Memory) stateful inference module using TensorRT.

nvidia::holoscan::custom_lstm_inference::TensorRtInference

Codelet, taking input tensors and feeding them into TensorRT for LSTM inference.

This implementation is based on nvidia::gxf::TensorRtInference. input_state_tensor_names and output_state_tensor_names parameters are added to specify tensor names for states in LSTM model.

Parameters

  • model_file_path: Path to ONNX model to be loaded

    • type: std::string

  • engine_cache_dir: Path to a directory containing cached generated engines to be serialized and loaded from

    • type: std::string

  • plugins_lib_namespace: Namespace used to register all the plugins in this library (default: "")

    • type: std::string

  • force_engine_update: Always update engine regard less of existing engine file. Such conversion may take minutes (default: false)

    • type: bool

  • input_tensor_names: Names of input tensors in the order to be fed into the model

    • type: std::vector<std::string>

  • input_state_tensor_names: Names of input state tensors that are used internally by TensorRT

    • type: std::vector<std::string>

  • input_binding_names: Names of input bindings as in the model in the same order of what is provided in input_tensor_names

    • type: std::vector<std::string>

  • output_tensor_names: Names of output tensors in the order to be retrieved from the model

    • type: std::vector<std::string>

  • input_state_tensor_names: Names of output state tensors that are used internally by TensorRT

    • type: std::vector<std::string>

  • output_binding_names: Names of output bindings in the model in the same order of of what is provided in output_tensor_names

    • type: std::vector<std::string>

  • pool: Allocator instance for output tensors

    • type: gxf::Handle<gxf::Allocator>

  • cuda_stream_pool: Instance of gxf::CudaStreamPool to allocate CUDA stream

    • type: gxf::Handle<gxf::CudaStreamPool>

  • max_workspace_size: Size of working space in bytes (default: 67108864l (64MB))

    • type: int64_t

  • dla_core: DLA Core to use. Fallback to GPU is always enabled. Default to use GPU only (optional)

    • type: int64_t

  • max_batch_size: Maximum possible batch size in case the first dimension is dynamic and used as batch size (default: 1)

    • type: int32_t

  • enable_fp16_: Enable inference with FP16 and FP32 fallback (default: false)

    • type: bool

  • verbose: Enable verbose logging on console (default: false)

    • type: bool

  • relaxed_dimension_check: Ignore dimensions of 1 for input tensor dimension check (default: true)

    • type: bool

  • clock: Instance of clock for publish time (optional)

    • type: gxf::Handle<gxf::Clock>

  • rx: List of receivers to take input tensors

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • tx: Transmitter to publish output tensors

    • type: gxf::Handle<gxf::Transmitter>

Visualizer Tool Tracking

The visualizer_tool_tracking extension provides a custom visualizer codelet that handles compositing, blending, and visualization of tool labels, tips, and masks given the output tensors of the custom_lstm_inference.

nvidia::holoscan::visualizer_tool_tracking::Sink

Surgical Tool Tracking Viz codelet.

Parameters

  • videoframe_vertex_shader_path: Path to vertex shader to be loaded

    • type: std::string

  • videoframe_fragment_shader_path: Path to fragment shader to be loaded

    • type: std::string

  • tooltip_vertex_shader_path: Path to vertex shader to be loaded

    • type: std::string

  • tooltip_fragment_shader_path: Path to fragment shader to be loaded

    • type: std::string

  • num_tool_classes: Number of different tool classes

    • type: int32_t

  • num_tool_pos_components: Number of components of the tool position vector (default: 2)

    • type: int32_t

  • tool_tip_colors: Color of the tool tips, a list of RGB values with components between 0 and 1 (default: 12 qualitative classes color scheme from colorbrewer2)

    • type: std::vector<std::vector<float>>

  • overlay_img_vertex_shader_path: Path to vertex shader to be loaded

    • type: std::string

  • overlay_img_fragment_shader_path: Path to fragment shader to be loaded

    • type: std::string

  • overlay_img_width: Width of overlay image

    • type: int32_t

  • overlay_img_height: Height of overlay image

    • type: int32_t

  • overlay_img_channels: Number of Overlay Image Channels

    • type: int32_t

  • overlay_img_layers: Number of Overlay Image Layers

    • type: int32_t

  • overlay_img_colors: Color of the image overlays, a list of RGB values with components between 0 and 1 (default: 12 qualitative classes color scheme from colorbrewer2)

    • type: std::vector<std::vector<float>>

  • tool_labels: List of tool names (default: [])

    • type: std::vector<std::string>

  • label_sans_font_path: Path for sans font to be loaded

    • type: std::string

  • label_sans_bold_font_path: Path for sans bold font to be loaded

    • type: std::string

  • in: List of input channels

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • in_tensor_names: Names of input tensors (default: "")

    • type: std::vector<std::string>

  • in_width: Width of the image (default: 640)

    • type: int32_t

  • in_height: Height of the image (default: 480)

    • type: int32_t

  • in_channels: Number of channels (default: 3)

    • type: int16_t

  • in_bytes_per_pixel: Number of bytes per pixel of the image (default: 1)

    • type: uint8_t

  • alpha_value: Alpha value that can be used when converting RGB888 to RGBA8888 (default: 255)

    • type: uint8_t

  • pool: Pool to allocate the output message.

    • type: gxf::Handle<gxf::Allocator>

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking after all messages are published.

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

Holoscan Test Mock

The mocks extension provides mock codelets that can be used for testing GXF applications.

nvidia::holoscan::mocks::VideoBufferMock

VideoBuffer Mock codelet. It creates RGB strips as an output message of gxf::VideoBuffer type to mimic the output of AJA extension.

Parameters

  • in_width: Width of the image (default: 640)

    • type: int32_t

  • in_height: Height of the image (default: 480)

    • type: int32_t

  • in_channels: Number of input channels (default: 3)

    • type: int16_t

  • in_bytes_per_pixel: Number of bytes per pixel of the image (default: 1)

    • type: int8_t

  • out_tensor_name: Name of the output tensor (default: "")

    • type: std::string

  • out: Output channel

    • type: gxf::Handle<gxf::Transmitter>

  • pool: Pool to allocate the output message

    • type: gxf::Handle<gxf::Allocator>

Emergent

The emergent_source extension supports an Emergent Vision Technologies camera as the video source. The datastream from this camera is transferred through Mellanox ConnectX SmartNIC using Rivermax SDK.

nvidia::holoscan::EmergentSource

Emergent Source codelet

Parameters

  • signal: Output signal

    • type: gxf::Handle<gxf::Transmitter>

  • width: Width of the stream (default: 4200)

    • type: uint32_t

  • height: Height of the stream (default: 2160)

    • type: uint32_t

  • framerate: Framerate of the stream (default: 240)

    • type: uint32_t

  • rdma: Enable RDMA (default: false)

    • type: bool

Bayer Demosaic

The bayer_demosaic extension performs color filter array (CFA) interpolation for 1-channel inputs of 8 or 16-bit unsigned integer and outputs an RGB or RGBA image.

nvidia::holoscan::BayerDemosaic

Bayer Demosaic codelet

Parameters

  • receiver: Input queue to component accepting gxf::Tensor or gxf::VideoBuffer

    • type: gxf::Handle<gxf::Receiver>

  • transmitter: Output queue of component for gxf::Tensor types

    • type: gxf::Handle<gxf::Transmitter>

  • in_tensor_name: Name of the expected input tensor (default: "")

    • type: std::string

  • out_tensor_name: Name of the output tensor generated by component (default: "")

    • type: std::string

  • pool: Pool to allocate output message contents

    • type: gxf::Handle<gxf::Allocator>

  • cuda_stream_pool: Instance of gxf::CudaStreamPool to allocate CUDA streams

    • type: gxf::Handle<gxf::CudaStreamPool>

  • interpolation_mode: Interpolation model to be used for demosaicing (default: 0 (UNDEFINED)). For details on possible interpolation modes consult NPP documentation. Currently NPP only supports UNDEFINED as interpolation mode

    • type: int

  • bayer_grid_pos: Bayer grid position (default: 2 (GBRG)). For details on possible Bayer grid position values consult NPP documentation

    • type: int

  • generate_alpha: Boolean value indicating whether output image should be RGB (false) or RGBA (true) (default: false)

    • type: bool

  • alpha_value: Alpha value desired at the output of the component when generate_alpha is set to true

    • type: int

Holoviz

The holoviz extension provides a high-speed viewer (built on Vulkan SDK) that handles compositing, blending, and visualization of RGB or RGBA images, masks, geometric primitives and text. The extension can auto detect the format of the input tensors when only the receivers list is specified. By default a window is opened to display the rendering, but the extension can also be run in headless mode. Using a display in exclusive mode is also supported, this reduces the latency by avoiding the desktop compositor. The rendered framebuffer can be output to render_buffer_output.

nvidia::holoscan::Holoviz

Holoviz codelet

Parameters

  • receivers: List of input queues to component accepting gxf::Tensor or gxf::VideoBuffer

    • type: std::vector<gxf::Handle<gxf::Receiver>>

  • enable_render_buffer_input: Enable render_buffer_input, (default: false)

    • type: bool

  • render_buffer_input: Input for an empty render buffer, type gxf::VideoBuffer

    • type: gxf::Handle<gxf::Receiver>

  • enable_render_buffer_output: Enable render_buffer_output, (default: false)

    • type: bool

  • render_buffer_output: Output for a filled render buffer. If an input render buffer is specified at render_buffer_input it uses that one, otherwise it allocates a new buffer.

    • type: gxf::Handle<gxf::Transmitter>

  • tensors: List of input tensor specifications (default: [])

    • type: std::vector<InputSpec>

      • name: name of the tensor containing the input data to display

        • type: std::string

      • type: input type (default "unknown")

        • type: std::string

        • possible values: unknown: unknown type, the operator tries to guess the type by inspecting the tensor color: RGB or RGBA color 2d image color_lut: single channel 2d image, color is looked up points: point primitives, one coordinate (x, y) per primitive lines: line primitives, two coordinates (x0, y0) and (x1, y1) per primitive line_strip: line strip primitive, a line primitive i is defined by each coordinate (xi, yi) and the following (xi+1, yi+1) triangles: triangle primitive, three coordinates (x0, y0), (x1, y1) and (x2, y2) per primitive crosses: cross primitive, a cross is defined by the center coordinate and the size (xi, yi, si) rectangles: axis aligned rectangle primitive, each rectangle is defined by two coordinates (xi, yi) and (xi+1, yi+1) ovals: oval primitive, an oval primitive is defined by the center coordinate and the axis sizes (xi, yi, sxi, syi) text: text is defined by the top left coordinate and the size (x, y, s) per string, text strings are defined by InputSpec member text

      • opacity: layer opacity, 1.0 is fully opaque, 0.0 is fully transparent (default: 1.0)

        • type: float

      • priority: layer priority, determines the render order, layers with higher priority values are rendered on top of layers with lower priority values (default: 0)

        • type: int32_t

      • color: RGBA color of rendered geometry (default: [1.f, 1.f, 1.f, 1.f])

        • type: std::vector<float>

      • line_width: name of the tensor containing the input data to display (default: 1.0)

        • type: float

      • point_size: point size for geometry made of points (default: 1.0)

        • type: float

      • text: array of text strings, used when type_ is text. (default: [])

        • type: std::vector<std::string>

  • color_lut: Color lookup table for tensors of type ‘color_lut’, vector of four float RGBA values

    • type: std::vector<std::vector<float>>

  • window_title: Title on window canvas (default: Holoviz)

    • type: std::string

  • display_name: In exclusive mode, name of display to use as shown with xrandr (default: DP-0)

    • type: std::string

  • width: Window width or display resolution width if in exclusive or fullscreen mode (default: 1920)

    • type: uint32_t

  • height: Window height or display resolution height if in exclusive or fullscreen mode (default: 1080)

    • type: uint32_t

  • framerate: Display framerate if in exclusive mode (default: 60)

    • type: uint32_t

  • use_exclusive_display: Enable exclusive display (default: false)

    • type: bool

  • fullscreen: Enable fullscreen window (default: false)

    • type: bool

  • headless: Enable headless mode. No window is opened, the render buffer is output to render_buffer_output. (default: false)

    • type: bool

  • window_close_scheduling_term: BooleanSchedulingTerm to stop the codelet from ticking when the window is closed

    • type: gxf::Handle<gxf::BooleanSchedulingTerm>

  • allocator: Allocator used to allocate memory for render_buffer_output

    • type: gxf::Handle<gxf::Allocator>

Tool tracking postprocessor

The tool_tracking_postprocessor extension provides a codelet that converts inference output of custom_lstm_inference used in the endoscopy tool tracking pipeline to be consumed by the holoviz codelet.

nvidia::holoscan::tool_tracking_postprocessor

Tool tracking postprocessor codelet

Parameters

  • in: Input channel, type gxf::Tensor

    • type: gxf::Handle<gxf::Receiver>

  • out: Output channel, type gxf::Tensor

    • type: gxf::Handle<gxf::Transmitter>

  • min_prob: Minimum probability, (default: 0.5)

    • type: float

  • overlay_img_colors: Color of the image overlays, a list of RGB values with components between 0 and 1, (default: 12 qualitative classes color scheme from colorbrewer2)

    • type: std::vector<std::vector<float>>

  • host_allocator: Output Allocator

    • type: gxf::Handle<gxf::Allocator>

  • device_allocator: Output Allocator

    • type: gxf::Handle<gxf::Allocator>

Multi AI Inference

The multiai_inference extension provides capabilities to execute inference on multiple models.

nvidia::holoscan::multiai::MultiAIInference

Multi AI Inference codelet takes multiple AI inference related parameters as the input from the configuration. It can ingest multiple GXF messages and transmits multiple tensors as the output via a single GXF Transmitter.

The codelet uses APIs from the Holoscan Inference component to execute all the tasks.

Parameters

  • model_path_map_: Map with key as the model name and value as the model file path.

    • type: HoloInfer::Mappings

  • pre_processor_map_: Map with key as the model name and value as a vector of the input tensor names.

    • type: HoloInfer::MultiMappings

  • inference_map_: Map with key as the model name and value as the inferred tensor name.

    • type: HoloInfer::Mappings

  • is_engine_path_: Flag to show if the input model path mapping is for the cached TensorRT engine files.

    • type: bool

  • in_tensor_names_: Input tensor names.

    • type: std::vector<std::string>

  • out_tensor_names_: Output tensor names.

    • type: std::vector<std::string>

  • allocator_: Memory allocator.

    • type: gxf::Handle<gxf::Allocator>

  • infer_on_cpu_: Flag to enable inference on the CPU (only supported by onnxruntime).

    • type: bool

  • parallel_inference_: Flag to enable parallel inference. Default is True.

    • type: bool

  • enable_fp16_: Flag showing if the TensorRT engine file conversion will use FP16. Default is False.

    • type: bool

  • backend_: Backend to do the inference on. Supported values: “trt”, “onnxrt”.

    • type: std::string

  • receivers_: Vector of input receivers. Multiple receivers supported.

    • type: HoloInfer::GXFReceivers

  • transmitter_: Output transmitter. Single transmitter supported.

    • type: HoloInfer::GXFTransmitters

  • input_on_cuda_: Flag showing if the input buffers (to the inference) are on CUDA. Default is True.

    • type: bool

  • output_on_cuda_: Flag showing if the output buffers (from the inference) are on CUDA. Default is True.

    • type: bool

  • transmit_on_cuda_: Flag showing if the data transmission is on CUDA. Default is True.

    • type: bool

Multi AI Postprocessor

The multiai_postprocessor extension provides processing capabilities to multiple input tensors.

nvidia::holoscan::multiai::MultiAIPostprocessor

Multi AI Postprocessing codelet takes multiple tensors as input mapped to the corresponding operations to perform on them. The codelet can ingest multiple GXF messages and transmits multiple tensors as output via a single GXF Transmitter.

The codelet uses APIs from the Holoscan Inference component to execute all the tasks.

Parameters

  • process_operations_: Map with key as the tensor name and value as a vector of the operations.

    • type: HoloInfer::MultiMappings

  • processed_map_: Map with key as the tensor name and value as the processed tensor name.

    • type: HoloInfer::Mappings

  • in_tensor_names_: Input tensor names.

    • type: std::vector<std::string>

  • out_tensor_names_: Output tensor names.

    • type: std::vector<std::string>

  • allocator_: Memory allocator.

    • type: gxf::Handle<gxf::Allocator>

  • receivers_: Vector of input receivers. Multiple receivers supported.

    • type: HoloInfer::GXFReceivers

  • transmitter_: Output transmitter. Single transmitter supported.

    • type: HoloInfer::GXFTransmitters

  • input_on_cuda_: Flag showing if the input buffers (to the processor) are on CUDA. Supported value is False.

    • type: bool

  • output_on_cuda_: Flag showing if the output buffers (from inference) are on CUDA. Supported value is False.

    • type: bool

  • transmit_on_cuda_: Flag showing if the data transmission is on CUDA. Supported value is False.

    • type: bool

Visualizer iCardio

The visualizer_icardio extension generates the visualization components from the processed results of the plax chamber model.

nvidia::holoscan::multiai::VisualizerICardio

Visualizer iCardio extension ingests the processed results of the plax chamber model and generates the key points, the key areas and the lines that are transmitted to the HoloViz codelet.

Parameters

  • in_tensor_names_: Input tensor names

    • type: std::vector<std::string>

  • out_tensor_names_: Output tensor names

    • type: std::vector<std::string>

  • allocator_: Memory allocator

    • type: gxf::Handle<gxf::Allocator>

  • receivers_: Vector of input receivers. Multiple receivers supported.

    • type: HoloInfer::GXFReceivers

  • transmitter_: Output transmitter. Single transmitter supported.

    • type: HoloInfer::GXFTransmitters

© Copyright 2022, NVIDIA. Last updated on Jun 28, 2023.