Class HolovizOp

• Defined in File holoviz.hpp

Nested Types

• Struct HolovizOp::InputSpec

• Struct InputSpec::View

Base Type

• public holoscan::Operator (Class Operator)

class HolovizOp : public holoscan::Operator

Operator class for data visualization.

This high-speed viewer handles compositing, blending, and visualization of RGB or RGBA images, masks, geometric primitives, text and depth maps. The operator can auto detect the format of the input tensors acquired at the receivers port. Else the input specification can be set at creation time using the tensors parameter or at runtime when passing input specifications to the input_specs port.

Depth maps and 3D geometry are rendered in 3D and support camera movement. The camera is controlled using the mouse:

• Orbit (LMB)

• Pan (LMB + CTRL | MMB)

• Dolly (LMB + SHIFT | RMB | Mouse wheel)

• Look Around (LMB + ALT | LMB + CTRL + SHIFT)

• Zoom (Mouse wheel + SHIFT) Or by providing new values at the camera_eye_input, camera_look_at_input or camera_up_input input ports. The camera pose can be output at the camera_pose_output port when enable_camera_pose_output is set to true.

==Named Inputs==

• receivers : multi-receiver accepting nvidia::gxf::Tensor and/or nvidia::gxf::VideoBuffer

  • Any number of upstream ports may be connected to this receivers port. This port can accept either VideoBuffers or Tensors. These inputs can be in either host or device memory. Each tensor or video buffer will result in a layer. The operator autodetects the layer type for certain input types (e.g. a video buffer will result in an image layer). For other input types or more complex use cases, input specifications can be provided either at initialization time as a parameter or dynamically at run time (via input_specs). On each call to compute, tensors corresponding to all names specified in the tensors parameter must be found or an exception will be raised. Any extra, named tensors not present in the tensors parameter specification (or optional, dynamic input_specs input) will be ignored.

• input_specs : std::vector<holoscan::ops::HolovizOp::InputSpec> (optional)

  • A list of InputSpec objects. This port can be used to dynamically update the overlay specification at run time. No inputs are required on this port in order for the operator to compute.

• render_buffer_input : nvidia::gxf::VideoBuffer (optional)

  • An empty render buffer can optionally be provided. The video buffer must have format GXF_VIDEO_FORMAT_RGBA and be in device memory. This input port only exists if enable_render_buffer_input was set to true, in which case compute will only be called when a message arrives on this input.

• camera_eye_input : std::array<float, 3> (optional)

  • Camera eye position. The camera is animated to reach the new position.

• camera_look_at_input : std::array<float, 3> (optional)

  • Camera look at position. The camera is animated to reach the new position.

• camera_up_input : : std::array<float, 3> (optional)

  • Camera up vector. The camera is animated to reach the new vector.

==Named Outputs==

• render_buffer_output : nvidia::gxf::VideoBuffer (optional)

  • Output for a filled render buffer. If an input render buffer is specified, it is using that one, else it allocates a new buffer. The video buffer will have format GXF_VIDEO_FORMAT_RGBA and will be in device memory. This output is useful for offline rendering or headless mode. This output port only exists if enable_render_buffer_output was set to true.

• camera_pose_output : std::array<float, 16> or nvidia::gxf::Pose3D (optional)

  • Output the camera pose. Depending on the value of camera_pose_output_type this outputs a 4x4 row major projection matrix (type std::array<float, 16>) or the camera extrinsics model (type nvidia::gxf::Pose3D). This output port only exists if enable_camera_pose_output was set to True.

==Parameters==

• receivers: List of input queues to component accepting gxf::Tensor or gxf::VideoBuffer.

  • type: std::vector<gxf::Handle<gxf::Receiver>>

• enable_render_buffer_input: Enable render_buffer_input (default: false)

  • type: bool

• enable_render_buffer_output: Enable render_buffer_output (default: false)

  • type: bool

• enable_camera_pose_output: Enable camera_pose_output (default: false)

  • type: bool

• tensors: List of input tensor specifications (default: [])

  • type: std::vector<InputSpec>

    • name: name of the tensor containing the input data to display

      • type: std::string

    • type: input type (default "unknown")

      • type: std::string

      • possible values:

        • unknown: unknown type, the operator tries to guess the type by inspecting the tensor.

        • color: RGB or RGBA color 2d image.

        • color_lut: single channel 2d image, color is looked up.

        • points: point primitives, one coordinate (x, y) per primitive.

        • lines: line primitives, two coordinates (x0, y0) and (x1, y1) per primitive.

        • line_strip: line strip primitive, a line primitive i is defined by each coordinate (xi, yi) and the following (xi+1, yi+1).

        • triangles: triangle primitive, three coordinates (x0, y0), (x1, y1) and (x2, y2) per primitive.

        • crosses: cross primitive, a cross is defined by the center coordinate and the size (xi, yi, si).

        • rectangles: axis aligned rectangle primitive, each rectangle is defined by two coordinates (xi, yi) and (xi+1, yi+1).

        • ovals: oval primitive, an oval primitive is defined by the center coordinate and the axis sizes (xi, yi, sxi, syi).

        • text: text is defined by the top left coordinate and the size (x, y, s) per string, text strings are defined by InputSpec member text.

        • depth_map: single channel 2d array where each element represents a depth value. The data is rendered as a 3d object using points, lines or triangles. The color for the elements can be specified through depth_map_color. Supported formats for the depth map:

          • 8-bit unsigned normalized format that has a single 8-bit depth component

          • 32-bit signed float format that has a single 32-bit depth component

        • depth_map_color: RGBA 2d image, same size as the depth map. One color value for each element of the depth map grid. Supported format: 32-bit unsigned normalized format that has an 8-bit R component in byte 0, an 8-bit G component in byte 1, an 8-bit B component in byte 2, and an 8-bit A component in byte 3.

    • opacity: layer opacity, 1.0 is fully opaque, 0.0 is fully transparent (default: 1.0)

      • type: float

    • priority: layer priority, determines the render order, layers with higher priority values are rendered on top of layers with lower priority values (default: 0)

      • type: int32_t

    • image_format: color image format, used if type is color, color_lut or depth_map_color. (default: auto_detect).

      • type: std::string

    • color: RGBA color of rendered geometry (default: [1.f, 1.f, 1.f, 1.f])

      • type: std::vector<float>

    • line_width: line width for geometry made of lines (default: 1.0)

      • type: float

    • point_size: point size for geometry made of points (default: 1.0)

      • type: float

    • text: array of text strings, used when type is text. (default: [])

      • type: std::vector<std::string>

    • depth_map_render_mode: depth map render mode (default: points)

      • type: std::string

      • possible values:

        • points: render as points

        • lines: render as lines

        • triangles: render as triangles

• color_lut: Color lookup table for tensors of type ‘color_lut’, vector of four float RGBA values

  • type: std::vector<std::vector<float>>

• window_title: Title on window canvas (default: "Holoviz")

  • type: std::string

• display_name: In exclusive display or fullscreen mode, name of display to use as shown with xrandr or hwinfo --monitor (default: `)

• : Window width or display resolution width if in exclusive display or fullscreen mode (default: )

• : Window height or display resolution height if in exclusive display or fullscreen mode (default: )

• : Display framerate if in exclusive display mode (default: )

• : Enable exclusive display mode (default: )

• : Enable fullscreen window (default: )

• : Enable headless mode. No window is opened, the render buffer is output to . (default: )

• : Enable sRGB framebuffer. If set to true, the operator will use an sRGB framebuffer for rendering. If set to false, the operator will use a linear framebuffer. (default: )

• : Enable vertical sync. If set to true the operator waits for the next vertical blanking period of the display to update the current image. (default: )

• : BooleanSchedulingTerm to stop the codelet from ticking when the window is closed

• : Allocator used to allocate memory for

• : File path for the font used for rendering text (default: )

• : Instance of gxf::CudaStreamPool

• : Type of data output at . Supported values are and . Default value is .

• : Initial camera eye position.

• : Initial camera look at position.

• : Initial camera up vector.

==Device Memory Requirements==

If is enabled, the provided buffer is used and no memory block will be allocated. Otherwise, when using this operator with a , a single device memory block is needed ( = 1). The size of this memory block can be determined by rounding the width and height up to the nearest even size and then padding the rows as needed so that the row stride is a multiple of 256 bytes. C++ code to calculate the block size is as follows:

==Notes==

Public Types

(span-pre-substitution) (span-pre-substitution) (span-pre-substitution)

Input type.

All geometric primitives expect a 1d array of coordinates. Coordinates range from 0.0 (left, top) to 1.0 (right, bottom).

Values:

(span-pre-substitution) (span-pre-substitution)

unknown type, the operator tries to guess the type by inspecting the tensor

(span-pre-substitution) (span-pre-substitution)

GRAY, RGB or RGBA 2d color image.

(span-pre-substitution) (span-pre-substitution)

single channel 2d image, color is looked up

(span-pre-substitution) (span-pre-substitution)

point primitives, one coordinate (x, y) per primitive

(span-pre-substitution) (span-pre-substitution)

line primitives, two coordinates (x0, y0) and (x1, y1) per primitive

(span-pre-substitution) (span-pre-substitution)

line strip primitive, a line primitive i is defined by each coordinate (xi, yi) and the following (xi+1, yi+1)

(span-pre-substitution) (span-pre-substitution)

triangle primitive, three coordinates (x0, y0), (x1, y1) and (x2, y2) per primitive

(span-pre-substitution) (span-pre-substitution)

cross primitive, a cross is defined by the center coordinate and the size (xi, yi, si)

(span-pre-substitution) (span-pre-substitution)

axis aligned rectangle primitive, each rectangle is defined by two coordinates (xi, yi) and (xi+1, yi+1)

(span-pre-substitution) (span-pre-substitution)

oval primitive, an oval primitive is defined by the center coordinate and the axis sizes (xi, yi, sxi, syi)

(span-pre-substitution) (span-pre-substitution)

text is defined by the top left coordinate and the size (x, y, s) per string, text strings are define by InputSpec::text_

(span-pre-substitution) (span-pre-substitution)

single channel 2d array where each element represents a depth value. The data is rendered as a 3d object using points, lines or triangles. The color for the elements can be specified through enum. Supported format: 8-bit unsigned normalized format that has a single 8-bit depth component

enumerator DEPTH_MAP_COLOR

RGBA 2d image, same size as the depth map. One color value for each element of the depth map grid. Supported format: 32-bit unsigned normalized format that has an 8-bit R component in byte > 0, an 8-bit G component in byte 1, an 8-bit B component in byte 2, and an 8-bit A component in byte 3

enumerator POINTS_3D

3D point primitives, one coordinate (x, y, z) per primitive

enumerator LINES_3D

3D line primitives, two coordinates (x0, y0, z0) and (x1, y1, z1) per primitive

enumerator LINE_STRIP_3D

3D line strip primitive, a line primitive i is defined by each coordinate (xi, yi, zi) and the following (xi+1, yi+1, zi+1)

enumerator TRIANGLES_3D

3D triangle primitive, three coordinates (x0, y0, z0), (x1, y1, z1) and (x2, y2, z2) per primitive

enum class ImageFormat

Image formats.

{component format}_{numeric format}

• component format

  • indicates the size in bits of the R, G, B and A components if present

• numeric format

  • UNORM - unsigned normalize values, range [0, 1]

  • SNORM - signed normalized values, range [-1,1]

  • UINT - unsigned integer values, range [0,2n-1]

  • SINT - signed integer values, range [-2n-1,2n-1-1]

  • SFLOAT - signed floating-point numbers

  • SRGB - the R, G, and B components are unsigned normalized values that represent values using sRGB nonlinear encoding, while the A component (if one exists) is a regular unsigned normalized value

Note: this needs to match the viz::ImageFormat enum (except the AUTO_DETECT value).

Values:

enumerator R8_UINT

specifies a one-component, 8-bit unsigned integer format that has a single 8-bit R component

enumerator R8_SINT

specifies a one-component, 8-bit signed integer format that has a single 8-bit R component

enumerator R8_UNORM

specifies a one-component, 8-bit unsigned normalized format that has a single 8-bit R component

enumerator R8_SNORM

specifies a one-component, 8-bit signed normalized format that has a single 8-bit R component

enumerator R8_SRGB

specifies a one-component, 8-bit unsigned normalized format that has a single 8-bit R component stored with sRGB nonlinear encoding

enumerator R16_UINT

specifies a one-component, 16-bit unsigned integer format that has a single 16-bit R component

enumerator R16_SINT

specifies a one-component, 16-bit signed integer format that has a single 16-bit R component

enumerator R16_UNORM

specifies a one-component, 16-bit unsigned normalized format that has a single 16-bit R component

enumerator R16_SNORM

specifies a one-component, 16-bit signed normalized format that has a single 16-bit R component

enumerator R16_SFLOAT

specifies a one-component, 16-bit signed floating-point format that has a single 16-bit R component

enumerator R32_UINT

specifies a one-component, 16-bit unsigned integer format that has a single 16-bit R component

enumerator R32_SINT

specifies a one-component, 16-bit signed integer format that has a single 16-bit R component

enumerator R32_SFLOAT

specifies a one-component, 32-bit signed floating-point format that has a single 32-bit R component

enumerator R8G8B8_UNORM

specifies a three-component, 24-bit unsigned normalized format that has a 8-bit R component in byte 0, a 8-bit G component in byte 1, and a 8-bit B component in byte 2

enumerator R8G8B8_SNORM

specifies a three-component, 24-bit signed normalized format that has a 8-bit R component in byte 0, a 8-bit G component in byte 1, and a 8-bit B component in byte 2

enumerator R8G8B8_SRGB

specifies a three-component, 24-bit unsigned normalized format that has a 8-bit R component stored with sRGB nonlinear encoding in byte 0, a 8-bit G component stored with sRGB nonlinear encoding in byte 1, and a 8-bit B component stored with sRGB nonlinear encoding in byte 2

enumerator R8G8B8A8_UNORM

specifies a four-component, 32-bit unsigned normalized format that has a 8-bit R component in byte 0, a 8-bit G component in byte 1, a 8-bit B component in byte 2, and a 8-bit A component in byte 3

enumerator R8G8B8A8_SNORM

specifies a four-component, 32-bit signed normalized format that has a 8-bit R component in byte 0, a 8-bit G component in byte 1, a 8-bit B component in byte 2, and a 8-bit A component in byte 3

enumerator R8G8B8A8_SRGB

specifies a four-component, 32-bit unsigned normalized format that has a 8-bit R component stored with sRGB nonlinear encoding in byte 0, a 8-bit G component stored with sRGB nonlinear encoding in byte 1, a 8-bit B component stored with sRGB nonlinear encoding in byte 2, and a 8-bit A component in byte 3

enumerator R16G16B16A16_UNORM

specifies a four-component, 64-bit unsigned normalized format that has a 16-bit R component in bytes 0..1, a 16-bit G component in bytes 2..3, a 16-bit B component in bytes 4..5, and a 16-bit A component in bytes 6..7

enumerator R16G16B16A16_SNORM

specifies a four-component, 64-bit signed normalized format that has a 16-bit R component in bytes 0..1, a 16-bit G component in bytes 2..3, a 16-bit B component in bytes 4..5, and a 16-bit A component in bytes 6..7

enumerator R16G16B16A16_SFLOAT

specifies a four-component, 64-bit signed floating-point format that has a 16-bit R component in bytes 0..1, a 16-bit G component in bytes 2..3, a 16-bit B component in bytes 4..5, and a 16-bit A component in bytes 6..7

enumerator R32G32B32A32_SFLOAT

specifies a four-component, 128-bit signed floating-point format that has a 32-bit R component in bytes 0..3, a 32-bit G component in bytes 4..7, a 32-bit B component in bytes 8..11, and a 32-bit A component in bytes 12..15

enumerator D16_UNORM

specifies a one-component, 16-bit unsigned normalized format that has a single 16-bit depth component

enumerator X8_D24_UNORM

specifies a two-component, 32-bit format that has 24 unsigned normalized bits in the depth component, and, optionally, 8 bits that are unused

enumerator D32_SFLOAT

specifies a one-component, 32-bit signed floating-point format that has 32 bits in the depth component

enumerator A2B10G10R10_UNORM_PACK32

specifies a four-component, 32-bit packed unsigned normalized format that has a 2-bit A component in bits 30..31, a 10-bit B component in bits 20..29, a 10-bit G component in bits 10..19, and a 10-bit R component in bits 0..9.

enumerator A2R10G10B10_UNORM_PACK32

specifies a four-component, 32-bit packed unsigned normalized format that has a 2-bit A component in bits 30..31, a 10-bit R component in bits 20..29, a 10-bit G component in bits 10..19, and a 10-bit B component in bits 0..9.

enumerator B8G8R8A8_UNORM

specifies a four-component, 32-bit unsigned normalized format that has a 8-bit B component in byte 0, a 8-bit G component in byte 1, a 8-bit R component in byte 2, and a 8-bit A component in byte 3

enumerator B8G8R8A8_SRGB

specifies a four-component, 32-bit unsigned normalized format that has a 8-bit B component stored with sRGB nonlinear encoding in byte 0, a 8-bit G component stored with sRGB nonlinear encoding in byte 1, a 8-bit R component stored with sRGB nonlinear encoding in byte 2, and a 8-bit A component in byte 3

enumerator A8B8G8R8_UNORM_PACK32

specifies a four-component, 32-bit packed unsigned normalized format that has an 8-bit A component in bits 24..31, an 8-bit B component in bits 16..23, an 8-bit G component in bits 8..15, and an 8-bit R component in bits 0..7.

enumerator A8B8G8R8_SRGB_PACK32

specifies a four-component, 32-bit packed unsigned normalized format that has an 8-bit A component in bits 24..31, an 8-bit B component stored with sRGB nonlinear encoding in bits 16..23, an 8-bit G component stored with sRGB nonlinear encoding in bits 8..15, and an 8-bit R component stored with sRGB nonlinear encoding in bits 0..7.

enumerator AUTO_DETECT

Auto detect the image format. If the input is a video buffer the format of the video buffer is used, if the input is a tensor then the format depends on the component count

• one component : gray level image

• three components : RGB image

• four components : RGBA image and the component type.

enum class DepthMapRenderMode

Depth map render mode.

Values:

enumerator POINTS

render points

enumerator LINES

render lines

enumerator TRIANGLES

render triangles

Public Functions

HOLOSCAN_OPERATOR_FORWARD_ARGS (HolovizOp) HolovizOp()=default

virtual void setup(OperatorSpec &spec) override

Define the operator specification.

Parameters

spec – The reference to the operator specification.

virtual void initialize() override

Initialize the operator.

This function is called when the fragment is initialized by Executor::initialize_fragment().

virtual void start() override

Implement the startup logic of the operator.

This method is called multiple times over the lifecycle of the operator according to the order defined in the lifecycle, and used for heavy initialization tasks such as allocating memory resources.

virtual void compute(InputContext &op_input, OutputContext &op_output, ExecutionContext &context) override

Implement the compute method.

This method is called by the runtime multiple times. The runtime calls this method until the operator is stopped.

Parameters

• op_input – The input context of the operator.

• op_output – The output context of the operator.

• context – The execution context of the operator.

virtual void stop() override

Implement the shutdown logic of the operator.

This method is called multiple times over the lifecycle of the operator according to the order defined in the lifecycle, and used for heavy deinitialization tasks such as deallocation of all resources previously assigned in start.

struct InputSpec

Input specification

Public Functions

InputSpec() = default

inline InputSpec(const std::string &tensor_name, InputType type)

InputSpec(const std::string &tensor_name, const std::string &type_str)

explicit InputSpec(const std::string &yaml_description)

Returns

an InputSpec from the YAML form output by description()

inline explicit operator bool() const noexcept

Returns

true if the input spec is valid

std::string description() const

Returns

a YAML string representation of the InputSpec

Public Members

std::string tensor_name_

name of the tensor/video buffer containing the input data

InputType type_ = InputType::UNKNOWN

input type

float opacity_ = 1.f

layer opacity, 1.0 is fully opaque, 0.0 is fully transparent

int32_t priority_ = 0

layer priority, determines the render order, layers with higher priority values are rendered on top of layers with lower priority values

ImageFormat image_format_ = ImageFormat::AUTO_DETECT

image format

std::vector<float> color_ = {1.f, 1.f, 1.f, 1.f}

color of rendered geometry

float line_width_ = 1.f

line width for geometry made of lines

float point_size_ = 1.f

point size for geometry made of points

std::vector<std::string> text_

array of text strings, used when type_ is TEXT.

DepthMapRenderMode depth_map_render_mode_ = DepthMapRenderMode::POINTS

depth map render mode, used if type_ is DEPTH_MAP or DEPTH_MAP_COLOR.

std::vector<View> views_

struct View

Layer view.

By default a layer will fill the whole window. When using a view the layer can be placed freely within the window.

Layers can also be placed in 3D space by specifying a 3D transformation matrix. Note that for geometry layers there is a default matrix which allows coordinates in the range of [0 … 1] instead of the Vulkan [-1 … 1] range. When specifying a matrix for a geometry layer, this default matrix is overwritten.

When multiple views are specified the layer is drawn multiple times using the specified layer views.

It’s possible to specify a negative term for height, which flips the image. When using a negative height, one should also adjust the y value to point to the lower left corner of the viewport instead of the upper left corner.

Public Members

float offset_x_ = 0.f

float offset_y_ = 0.f

offset of top-left corner of the view. Top left coordinate of the window area is (0, 0) bottom right coordinate is (1, 1).

float width_ = 1.f

float height_ = 1.f

width and height of the view in normalized range. 1.0 is full size.

std::optional<std::array<float, 16>> matrix_

row major 4x4 transform matrix (optional, can be nullptr)