Class HolovizOp
Defined in File holoviz.hpp
Base Type
public holoscan::Operator
(Class Operator)
-
class HolovizOp : public holoscan::Operator
Operator class for data visualization.
This high-speed viewer handles compositing, blending, and visualization of RGB or RGBA images, masks, geometric primitives, text and depth maps. The operator can auto detect the format of the input tensors when only the
receivers
parameter list is specified. Else the input specification can be set at creation time using thetensors
parameter or at runtime when passing input specifications to theinput_specs
port.Named inputs:
receivers: multi-receiver accepting
nvidia::gxf::Tensor
and/ornvidia::gxf::VideoBuffer
Any number of upstream ports may be connected to this
receivers
port. This port can accept either VideoBuffers or Tensors. These inputs can be in either host or device memory. Each tensor or video buffer will result in a layer. The operator autodetects the layer type for certain input types (e.g. a video buffer will result in an image layer). For other input types or more complex use cases, input specifications can be provided either at initialization time as a parameter or dynamically at run time (viainput_specs
). On each call tocompute
, tensors corresponding to all names specified in thetensors
parameter must be found or an exception will be raised. Any extra, named tensors not present in thetensors
parameter specification (or optional, dynamicinput_specs
input) will be ignored.
input_specs:
std::vector<<a class="reference internal" href="structholoscan_1_1ops_1_1HolovizOp_1_1InputSpec.html#structholoscan_1_1ops_1_1HolovizOp_1_1InputSpec" target="_self">holoscan::ops::HolovizOp::InputSpec</a>>
(optional)A list of
<a class="reference internal" href="structholoscan_1_1ops_1_1HolovizOp_1_1InputSpec.html#structholoscan_1_1ops_1_1HolovizOp_1_1InputSpec" target="_self">InputSpec</a>
objects. This port can be used to dynamically update the overlay specification at run time. No inputs are required on this port in order for the operator tocompute
.
render_buffer_input:
nvidia::gxf::VideoBuffer
(optional)An empty render buffer can optionally be provided. The video buffer must have format GXF_VIDEO_FORMAT_RGBA and be in device memory. This input port only exists if
enable_render_buffer_input
was set to true, in which casecompute
will only be called when a message arrives on this input.
Named outputs:
render_buffer_output:
nvidia::gxf::VideoBuffer
(optional)Output for a filled render buffer. If an input render buffer is specified, it is using that one, else it allocates a new buffer. The video buffer will have format GXF_VIDEO_FORMAT_RGBA and will be in device memory. This output is useful for offline rendering or headless mode. This output port only exists if
enable_render_buffer_output
was set to true.
camera_pose_output:
std::array<float, 16>
(optional)The camera pose. The parameters returned represent the values of a 4x4 row major projection matrix. This output port only exists if
enable_camera_pose_output
was set to true.
Parameters
**
receivers
**: List of input queues to component acceptinggxf::Tensor
orgxf::VideoBuffer
type:
std::vector<gxf::Handle<gxf::Receiver>>
**
enable_render_buffer_input
**: Enablerender_buffer_input
, (default:false
)type:
bool
**
render_buffer_input
**: Input for an empty render buffer, typegxf::VideoBuffer
type:
gxf::Handle<gxf::Receiver>
**
enable_render_buffer_output
**: Enablerender_buffer_output
, (default:false
)type:
bool
**
render_buffer_output
**: Output for a filled render buffer. If an input render buffer is specified atrender_buffer_input
it uses that one, otherwise it allocates a new buffer.type:
gxf::Handle<gxf::Transmitter>
**
enable_camera_pose_output
**: Enablecamera_pose_output
, (default:false
)type:
bool
**
camera_pose_output
**: Output the camera pose. The camera parameters are returned in a 4x4 row major projection matrix.type:
std::array<float, 16>
**
tensors
**: List of input tensor specifications (default:[]
)type:
std::vector<<a class="reference internal" href="structholoscan_1_1ops_1_1HolovizOp_1_1InputSpec.html#structholoscan_1_1ops_1_1HolovizOp_1_1InputSpec" target="_self">InputSpec</a>>
**
name
**: name of the tensor containing the input data to displaytype:
std::string
**
type
**: input type (default"unknown"
)type:
std::string
possible values: **
unknown
**: unknown type, the operator tries to guess the type by inspecting the tensor **color
**: RGB or RGBA color 2d image **color_lut
**: single channel 2d image, color is looked up **points
**: point primitives, one coordinate (x, y) per primitive **lines
**: line primitives, two coordinates (x0, y0) and (x1, y1) per primitive **line_strip
**: line strip primitive, a line primitive i is defined by each coordinate (xi, yi) and the following (xi+1, yi+1) **triangles
**: triangle primitive, three coordinates (x0, y0), (x1, y1) and (x2, y2) per primitive **crosses
**: cross primitive, a cross is defined by the center coordinate and the size (xi, yi, si) **rectangles
**: axis aligned rectangle primitive, each rectangle is defined by two coordinates (xi, yi) and (xi+1, yi+1) **ovals
**: oval primitive, an oval primitive is defined by the center coordinate and the axis sizes (xi, yi, sxi, syi) **text
**: text is defined by the top left coordinate and the size (x, y, s) per string, text strings are defined by InputSpec member **text
** **depth_map
**: single channel 2d array where each element represents a depth value. The data is rendered as a 3d object using points, lines or triangles. The color for the elements can be specified throughdepth_map_color
. Supported format: 8-bit unsigned normalized format that has a single 8-bit depth component **depth_map_color
**: RGBA 2d image, same size as the depth map. One color value for each element of the depth map grid. Supported format: 32-bit unsigned normalized format that has an 8-bit R component in byte 0, an 8-bit G component in byte 1, an 8-bit B component in byte 2, and an 8-bit A component in byte 3.
**
opacity
**: layer opacity, 1.0 is fully opaque, 0.0 is fully transparent (default:1.0
)type:
float
**
priority
**: layer priority, determines the render order, layers with higher priority values are rendered on top of layers with lower priority values (default:0
)type:
int32_t
**
color
**: RGBA color of rendered geometry (default:[1.f, 1.f, 1.f, 1.f]
)type:
std::vector<float>
**
line_width
**: line width for geometry made of lines (default:1.0
)type:
float
**
point_size
**: point size for geometry made of points (default:1.0
)type:
float
**
text
**: array of text strings, used whentype
is text. (default:[]
)type:
std::vector<std::string>
**
depth_map_render_mode
**: depth map render mode (default:points
) -typestd::string
possible values: **
points
**: render as points **lines
**: render as lines **triangles
**: render as triangles
**
color_lut
**: Color lookup table for tensors of type ‘color_lut’, vector of four float RGBA valuestype:
std::vector<std::vector<float>>
**
window_title
**: Title on window canvas (default:Holoviz
)type:
std::string
**
display_name
**: In exclusive mode, name of display to use as shown with xrandr (default:DP-0
)type:
std::string
**
width
**: Window width or display resolution width if in exclusive or fullscreen mode (default:1920
)type:
uint32_t
**
height
**: Window height or display resolution height if in exclusive or fullscreen mode (default:1080
)type:
uint32_t
**
framerate
**: Display framerate if in exclusive mode (default:60
)type:
uint32_t
**
use_exclusive_display
**: Enable exclusive display (default:false
)type:
bool
**
fullscreen
**: Enable fullscreen window (default:false
)type:
bool
**
headless
**: Enable headless mode. No window is opened, the render buffer is output torender_buffer_output
. (default:false
)type:
bool
**
window_close_scheduling_term
**: BooleanSchedulingTerm to stop the codelet from ticking when the window is closedtype:
gxf::Handle<gxf::BooleanSchedulingTerm>
**
allocator
**: Allocator used to allocate memory forrender_buffer_output
type:
gxf::Handle<gxf::Allocator>
**
font_path
**: File path for the font used for rendering text.type:
std::string
**
cuda_stream_pool
**: Instance of gxf::CudaStreamPooltype:
gxf::Handle<gxf::CudaStreamPool>
Displaying Color Images
Image data can either be on host or device (GPU). Multiple image formats are supported
R 8 bit unsigned
R 16 bit unsigned
R 16 bit float
R 32 bit unsigned
R 32 bit float
RGB 8 bit unsigned
BGR 8 bit unsigned
RGBA 8 bit unsigned
BGRA 8 bit unsigned
RGBA 16 bit unsigned
RGBA 16 bit float
RGBA 32 bit float
When the
type
parameter is set tocolor_lut
the final color is looked up using the values from thecolor_lut
parameter. For color lookups these image formats are supportedR 8 bit unsigned
R 16 bit unsigned
R 32 bit unsigned
Drawing Geometry
In all cases,
x
andy
are normalized coordinates in the range[0, 1]
. Thex
andy
correspond to the horizontal and vertical axes of the display, respectively. The origin(0, 0)
is at the top left of the display. Geometric primitives outside of the visible area are clipped. Coordinate arrays are expected to have the shape(N, C)
whereN
is the coordinate count andC
is the component count for each coordinate.Points are defined by a
(x, y)
coordinate pair.Lines are defined by a set of two
(x, y)
coordinate pairs.Lines strips are defined by a sequence of
(x, y)
coordinate pairs. The first two coordinates define the first line, each additional coordinate adds a line connecting to the previous coordinate.Triangles are defined by a set of three
(x, y)
coordinate pairs.Crosses are defined by
(x, y, size)
tuples.size
specifies the size of the cross in thex
direction and is optional, if omitted it’s set to0.05
. The size in they
direction is calculated using the aspect ratio of the window to make the crosses square.Rectangles (bounding boxes) are defined by a pair of 2-tuples defining the upper-left and lower-right coordinates of a box:
(x1, y1), (x2, y2)
.Ovals are defined by
(x, y, size_x, size_y)
tuples.size_x
andsize_y
are optional, if omitted they are set to0.05
.Texts are defined by
(x, y, size)
tuples.size
specifies the size of the text iny
direction and is optional, if omitted it’s set to0.05
. The size in thex
direction is calculated using the aspect ratio of the window. The index of each coordinate references a text string from thetext
parameter and the index is clamped to the size of the text array. For example, if there is one item set for thetext
parameter, e.g. text=[‘my_text]and three coordinates, then
my_textis rendered three times. If
text=[‘first text’, ‘second text’]and three coordinates are specified, then
first textis rendered at the first coordinate,
second textat the second coordinate and then
second textagain at the third coordinate. The
text` string array is fixed and can’t be changed after initialization. To hide text which should not be displayed, specify coordinates greater than(1.0, 1.0)
for the text item, the text is then clipped away.3D Points are defined by a
(x, y, z)
coordinate tuple.3D Lines are defined by a set of two
(x, y, z)
coordinate tuples.3D Lines strips are defined by a sequence of
(x, y, z)
coordinate tuples. The first two coordinates define the first line, each additional coordinate adds a line connecting to the previous coordinate.3D Triangles are defined by a set of three
(x, y, z)
coordinate tuples.
Displaying Depth Maps
When
type
isdepth_map
the provided data is interpreted as a rectangular array of depth values. Additionally a 2d array with a color value for each point in the grid can be specified by settingtype
todepth_map_color
.The type of geometry drawn can be selected by setting
depth_map_render_mode
.Depth maps are rendered in 3D and support camera movement. The camera is controlled using the mouse:
Orbit (LMB)
Pan (LMB + CTRL | MMB)
Dolly (LMB + SHIFT | RMB | Mouse wheel)
Look Around (LMB + ALT | LMB + CTRL + SHIFT)
Zoom (Mouse wheel + SHIFT)
Output
By default a window is opened to display the rendering, but the extension can also be run in headless mode with the
headless
parameter.Using a display in exclusive mode is also supported with the
use_exclusive_display
parameter. This reduces the latency by avoiding the desktop compositor.The rendered framebuffer can be output to
render_buffer_output
.Public Types
-
enum class InputType
Input type.
All geometric primitives expect a 1d array of coordinates. Coordinates range from 0.0 (left, top) to 1.0 (right, bottom).
Values:
- enumerator UNKNOWN
unknown type, the operator tries to guess the type by inspecting the tensor
- enumerator COLOR
GRAY, RGB or RGBA 2d color image.
- enumerator COLOR_LUT
single channel 2d image, color is looked up
- enumerator POINTS
point primitives, one coordinate (x, y) per primitive
- enumerator LINES
line primitives, two coordinates (x0, y0) and (x1, y1) per primitive
- enumerator LINE_STRIP
line strip primitive, a line primitive i is defined by each coordinate (xi, yi) and the following (xi+1, yi+1)
- enumerator TRIANGLES
triangle primitive, three coordinates (x0, y0), (x1, y1) and (x2, y2) per primitive
- enumerator CROSSES
cross primitive, a cross is defined by the center coordinate and the size (xi, yi, si)
- enumerator RECTANGLES
axis aligned rectangle primitive, each rectangle is defined by two coordinates (xi, yi) and (xi+1, yi+1)
- enumerator OVALS
oval primitive, an oval primitive is defined by the center coordinate and the axis sizes (xi, yi, sxi, syi)
- enumerator TEXT
text is defined by the top left coordinate and the size (x, y, s) per string, text strings are define by InputSpec::text_
- enumerator DEPTH_MAP
single channel 2d array where each element represents a depth value. The data is rendered as a 3d object using points, lines or triangles. The color for the elements can be specified through
DEPTH_MAP_COLOR
. Supported format: 8-bit unsigned normalized format that has a single 8-bit depth component- enumerator DEPTH_MAP_COLOR
RGBA 2d image, same size as the depth map. One color value for each element of the depth map grid. Supported format: 32-bit unsigned normalized format that has an 8-bit R component in byte > 0, an 8-bit G component in byte 1, an 8-bit B component in byte 2, and an 8-bit A component in byte 3
- enumerator POINTS_3D
3D point primitives, one coordinate (x, y, z) per primitive
- enumerator LINES_3D
3D line primitives, two coordinates (x0, y0, z0) and (x1, y1, z1) per primitive
- enumerator LINE_STRIP_3D
3D line strip primitive, a line primitive i is defined by each coordinate (xi, yi, zi) and the following (xi+1, yi+1, zi+1)
- enumerator TRIANGLES_3D
3D triangle primitive, three coordinates (x0, y0, z0), (x1, y1, z1) and (x2, y2, z2) per primitive
-
enum class DepthMapRenderMode
Depth map render mode.
Values:
- enumerator POINTS
render points
- enumerator LINES
render lines
- enumerator TRIANGLES
render triangles
Public Functions
- HOLOSCAN_OPERATOR_FORWARD_ARGS (HolovizOp) HolovizOp()=default
-
virtual void setup(OperatorSpec &spec) override
Define the operator specification.
- Parameters
spec – The reference to the operator specification.
- virtual void initialize() override
Initialize the operator.
This function is called when the fragment is initialized by Executor::initialize_fragment().
- virtual void start() override
Implement the startup logic of the operator.
This method is called multiple times over the lifecycle of the operator according to the order defined in the lifecycle, and used for heavy initialization tasks such as allocating memory resources.
-
virtual void compute(InputContext &op_input, OutputContext &op_output, ExecutionContext &context) override
Implement the compute method.
This method is called by the runtime multiple times. The runtime calls this method until the operator is stopped.
- Parameters
op_input – The input context of the operator.
op_output – The output context of the operator.
context – The execution context of the operator.
- virtual void stop() override
Implement the shutdown logic of the operator.
This method is called multiple times over the lifecycle of the operator according to the order defined in the lifecycle, and used for heavy deinitialization tasks such as deallocation of all resources previously assigned in start.
-
struct InputSpec
Input specification
Public Functions
- InputSpec() = default
- inline InputSpec(const std::string &tensor_name, InputType type)
- InputSpec(const std::string &tensor_name, const std::string &type_str)
-
explicit InputSpec(const std::string &yaml_description)
- Returns
an InputSpec from the YAML form output by description()
-
inline explicit operator bool() const noexcept
- Returns
true if the input spec is valid
-
std::string description() const
- Returns
a YAML string representation of the InputSpec
Public Members
- std::string tensor_name_
name of the tensor containing the input data
- float opacity_ = 1.f
layer opacity, 1.0 is fully opaque, 0.0 is fully transparent
- int32_t priority_ = 0
layer priority, determines the render order, layers with higher priority values are rendered on top of layers with lower priority values
- std::vector<float> color_ = {1.f, 1.f, 1.f, 1.f}
color of rendered geometry
- float line_width_ = 1.f
line width for geometry made of lines
- float point_size_ = 1.f
point size for geometry made of points
- std::vector<std::string> text_
array of text strings, used when type_ is TEXT.
- DepthMapRenderMode depth_map_render_mode_ = DepthMapRenderMode::POINTS
depth map render mode, used if type_ is DEPTH_MAP or DEPTH_MAP_COLOR.
- std::vector<View> views_
-
struct View
Layer view.
By default a layer will fill the whole window. When using a view the layer can be placed freely within the window.
Layers can also be placed in 3D space by specifying a 3D transformation matrix. Note that for geometry layers there is a default matrix which allows coordinates in the range of [0 … 1] instead of the Vulkan [-1 … 1] range. When specifying a matrix for a geometry layer, this default matrix is overwritten.
When multiple views are specified the layer is drawn multiple times using the specified layer views.
It’s possible to specify a negative term for height, which flips the image. When using a negative height, one should also adjust the y value to point to the lower left corner of the viewport instead of the upper left corner.
Public Members
- float offset_x_ = 0.f
- float offset_y_ = 0.f
offset of top-left corner of the view. Top left coordinate of the window area is (0, 0) bottom right coordinate is (1, 1).
- float width_ = 1.f
- float height_ = 1.f
width and height of the view in normalized range. 1.0 is full size.
- std::optional<std::array<float, 16>> matrix_
row major 4x4 transform matrix (optional, can be nullptr)