Visualization

Overview

Holoviz provides the functionality to composite real-time streams of frames with multiple different other layers like segmentation mask layers, geometry layers, and GUI layers.

For maximum performance, Holoviz makes use of Vulkan, which is already installed as part of the NVIDIA GPU driver.

Holoscan provides the Holoviz operator which is sufficient for many, even complex visualization tasks. The Holoviz operator is used by multiple Holoscan example applications.

Additionally, for more advanced use cases, the Holoviz module can be used to create application-specific visualization operators. The Holoviz module provides a C++ API and is also used by the Holoviz operator.

The term Holoviz is used for both the Holoviz operator and the Holoviz module below. Both the operator and the module roughly support the same feature set. Where applicable, information on how to use a feature with the operator and the module is provided. It’s explicitly mentioned below when features are not supported by the operator.

Layers

The core entity of Holoviz are layers. A layer is a two-dimensional image object. Multiple layers are composited to create the final output.

These layer types are supported by Holoviz:

• Image layer
• Geometry layer
• GUI layer

All layers have common attributes which define the look and also the way layers are finally composited.

The priority determines the rendering order of the layers. Before rendering, the layers are sorted by priority. The layers with the lowest priority are rendered first, so that the layer with the highest priority is rendered on top of all other layers. If layers have the same priority, then the render order of these layers is undefined.

The example below draws a transparent geometry layer on top of an image layer (geometry data and image data creation is omitted in the code). Although the geometry layer is specified first, it is drawn last because it has a higher priority (1) than the image layer (0).

1
  std::vector<ops::HolovizOp::InputSpec> input_specs;
2
   
3
  auto& geometry_spec =
4
      input_specs.emplace_back(ops::HolovizOp::InputSpec("point_tensor", ops::HolovizOp::InputType::POINTS));
5
  geometry_spec.priority_ = 1;
6
  geometry_spec.opacity_ = 0.5;
7
   
8
  auto& image_spec =
9
      input_specs.emplace_back(ops::HolovizOp::InputSpec("image_tensor", ops::HolovizOp::InputType::IMAGE));
10
  image_spec.priority_ = 0;
11
   
12
  auto visualizer = make_operator<ops::HolovizOp>("holoviz", Arg("tensors", input_specs));
13
   
14
  // the source provides two tensors named "point_tensor" and "image_tensor" at the "outputs" port.
15
  add_flow(source, visualizer, {{"outputs", "receivers"}});

Image Layers

1
  std::vector<ops::HolovizOp::InputSpec> input_specs;
2
   
3
  auto& image_spec =
4
      input_specs.emplace_back(ops::HolovizOp::InputSpec("image", ops::HolovizOp::InputType::IMAGE));
5
   
6
  auto visualizer = make_operator<ops::HolovizOp>("holoviz", Arg("tensors", input_specs));
7
   
8
  // the source provides an image named "image" at the "outputs" port.
9
  add_flow(source, visualizer, {{"output", "receivers"}});

Supported Image Formats

Geometry Layers

A geometry layer is used to draw 2d or 3d geometric primitives. 2d primitives are points, lines, line strips, rectangles, ovals or text and are defined with 2d coordinates (x, y). 3d primitives are points, lines, line strips or triangles and are defined with 3d coordinates (x, y, z).

Coordinates start with (0, 0) in the top left and end with (1, 1) in the bottom right for 2d primitives.

ImGui Layers

The Holoviz module supports user interface layers created with Dear ImGui.

Calls to the Dear ImGui API are allowed between viz::BeginImGuiLayer and viz::EndImGuiLayer, and are used to draw to the ImGui layer. The ImGui layer behaves like other layers and is rendered with the layer opacity and priority.

The code below creates a Dear ImGui window with a checkbox used to conditionally show a image layer.

1
namespace viz = holoscan::viz;
2
 
3
bool show_image_layer = false;
4
while (!viz::WindowShouldClose()) {
5
    viz::Begin();
6
 
7
    viz::BeginImGuiLayer();
8
 
9
    ImGui::Begin("Options");
10
    ImGui::Checkbox("Image layer", &show_image_layer);
11
    ImGui::End();
12
 
13
    viz::EndLayer();
14
 
15
    if (show_image_layer) {
16
        viz::BeginImageLayer();
17
        viz::ImageHost(...);
18
        viz::EndLayer();
19
    }
20
 
21
    viz::End();
22
}

ImGUI is a static library and has no stable API. Therefore, the application and Holoviz have to use the same ImGUI version. When the link target holoscan::viz::imgui is exported, make sure to link your application against that target.

Depth Map Layers

A depth map is a single channel 2d array where each element represents a depth value. The data is rendered as a 3D object using points, lines, or triangles. The color for the elements can also be specified.

Supported formats for the depth map:

• 8-bit unsigned normalized format that has a single 8-bit depth component
• 32-bit signed float format that has a single 32-bit depth component

Supported format for the depth color map:

• 32-bit unsigned normalized format that has an 8-bit R component in byte 0, an 8-bit G component in byte 1, an 8-bit B component in byte 2, and an 8-bit A component in byte 3

Depth maps are rendered in 3D and support camera movement.

1
  std::vector<ops::HolovizOp::InputSpec> input_specs;
2
   
3
  auto& depth_map_spec =
4
      input_specs.emplace_back(ops::HolovizOp::InputSpec("depth_map", ops::HolovizOp::InputType::DEPTH_MAP));
5
  depth_map_spec.depth_map_render_mode_ = ops::HolovizOp::DepthMapRenderMode::TRIANGLES;
6
   
7
  auto visualizer = make_operator<ops::HolovizOp>("holoviz",
8
      Arg("tensors", input_specs));
9
   
10
  // the source provides an depth map named "depth_map" at the "output" port.
11
  add_flow(source, visualizer, {{"output", "receivers"}});

Views

By default, a layer will fill the whole window. When using a view, the layer can be placed freely within the window.

Layers can also be placed in 3D space by specifying a 3D transformation matrix.

When multiple views are specified, the layer is drawn multiple times using the specified layer view.

It’s possible to specify a negative term for height, which flips the image. When using a negative height, you should also adjust the y value to point to the lower left corner of the viewport instead of the upper left corner.

Camera

When rendering 3d geometry using a geometry layer with 3d primitives or using a depth map layer the camera properties can either be set by the application or interactively changed by the user.

To interactively change the camera, use the mouse:

• Orbit (LMB)
• Pan (LMB + CTRL | MMB)
• Dolly (LMB + SHIFT | RMB | Mouse wheel)
• Look Around (LMB + ALT | LMB + CTRL + SHIFT)
• Zoom (Mouse wheel + SHIFT)

Using a Display in Exclusive Mode

Typically, Holoviz opens a normal window on the Linux desktop. In that case, the desktop compositor is combining the Holoviz image with all other elements on the desktop. To avoid this extra compositing step, Holoviz can render to a display directly.

Configure a Display for Exclusive Use

  sudo systemctl stop display-manager

$
  sudo systemctl start display-manager

Enable Exclusive Display in Holoviz

1
  auto visualizer = make_operator<ops::HolovizOp>("holoviz",
2
      Arg("use_exclusive_display", true), // required
3
      Arg("display_name", "DP-2"), // optional
4
      Arg("width", 2560), // optional
5
      Arg("height", 1440), // optional
6
      Arg("framerate", 240) // optional
7
      );

The name of the display can either be the EDID name as displayed in the NVIDIA Settings, or the output name provided by xrandr or hwinfo --monitor.

$
  Screen 0: minimum 8 x 8, current 4480 x 1440, maximum 32767 x 32767
$
  DP-0 disconnected (normal left inverted right x axis y axis)
$
  DP-1 disconnected (normal left inverted right x axis y axis)
$
  DP-2 connected primary 2560x1440+1920+0 (normal left inverted right x axis y axis) 600mm x 340mm
$
     2560x1440     59.98 + 239.97*  199.99   144.00   120.00    99.95
$
     1024x768      60.00
$
     800x600       60.32
$
     640x480       59.94
$
  USB-C-0 disconnected (normal left inverted right x axis y axis)

CUDA Streams

By default, Holoviz is using CUDA stream 0 for all CUDA operations. Using the default stream can affect concurrency of CUDA operations, see stream synchronization behavior for more information.

1
  const std::shared_ptr<holoscan::CudaStreamPool> cuda_stream_pool =
2
      make_resource<holoscan::CudaStreamPool>("cuda_stream", 0, 0, 0, 1, 5);
3
  auto visualizer =
4
      make_operator<holoscan::ops::HolovizOp>("visualizer",
5
          Arg("cuda_stream_pool") = cuda_stream_pool);

Reading the Frame Buffer

The rendered frame buffer can be read back. This is useful when doing offscreen rendering or running Holoviz in a headless environment.

1
  std::shared_ptr<holoscan::ops::HolovizOp> visualizer =
2
      make_operator<ops::HolovizOp>("visualizer",
3
          Arg("enable_render_buffer_output", true),
4
          Arg("enable_depth_buffer_output", true),
5
          Arg("allocator") = make_resource<holoscan::UnboundedAllocator>("allocator"),
6
          Arg("cuda_stream_pool") = cuda_stream_pool);
7
   
8
  add_flow(visualizer, destination, {{"render_buffer_output", "color_input"}});
9
  add_flow(visualizer, destination, {{"depth_buffer_output", "depth_input"}});

sRGB

The sRGB color space is supported for both images and the framebuffer. By default Holoviz is using a linear encoded framebuffer.

HDR

Holoviz supports selecting the framebuffer surface image format and color space. To enable HDR select a HDR color space.

Distributions supporting HDR

At the time of writing (08/2024) there is currently no official support for HDR on Linux. However there is experimental HDR support for Gnome version 44 and KDE Plasma 6.

KDE Plasma 6

Experimental HDR support is described in this blog post. Three steps are required to use HDR with Holoviz:

1. Enable HDR in the display configuration
2. Install the Vulkan HDR layer
3. Set the ENABLE_HDR_WSI environment variable to 1.

Run vulkaninfo to verify that HDR color spaces are reported

$
> vulkaninfo
$
...
$
GPU id : 0 (NVIDIA RTX 6000 Ada Generation):
$
        Surface type = VK_KHR_wayland_surface
$
        Formats: count = 11
$
                SurfaceFormat[0]:
$
                        format = FORMAT_A2B10G10R10_UNORM_PACK32
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[1]:
$
                        format = FORMAT_A2R10G10B10_UNORM_PACK32
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[2]:
$
                        format = FORMAT_R8G8B8A8_SRGB
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[3]:
$
                        format = FORMAT_R8G8B8A8_UNORM
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[4]:
$
                        format = FORMAT_B8G8R8A8_SRGB
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[5]:
$
                        format = FORMAT_B8G8R8A8_UNORM
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[6]:
$
                        format = FORMAT_R16G16B16A16_SFLOAT
$
                        colorSpace = COLOR_SPACE_SRGB_NONLINEAR_KHR
$
                SurfaceFormat[7]:
$
                        format = FORMAT_A2B10G10R10_UNORM_PACK32
$
                        colorSpace = COLOR_SPACE_HDR10_ST2084_EXT
$
                SurfaceFormat[8]:
$
                        format = FORMAT_A2R10G10B10_UNORM_PACK32
$
                        colorSpace = COLOR_SPACE_HDR10_ST2084_EXT
$
                SurfaceFormat[9]:
$
                        format = FORMAT_R16G16B16A16_SFLOAT
$
                        colorSpace = COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT
$
                SurfaceFormat[10]:
$
                        format = FORMAT_R16G16B16A16_SFLOAT
$
                        colorSpace = COLOR_SPACE_BT709_LINEAR_EXT
$
...

Gnome version 44

Gnome version 44 is part of Ubuntu 24.04.

Experimental HDR support had been added with this MR. To enable HDR, make sure Wayland is used (no HDR with X11), press Alt+F2, then type lg in the prompt to start the looking glass console. Enter global.compositor.backend.get_monitor_manager().experimental_hdr = 'on' to enable HDR, global.compositor.backend.get_monitor_manager().experimental_hdr = 'off' to disable HDR.

Gnome is not yet passing the HDR color space to Vulkan. The color space and other information can be queried using drm_info, check modes where HDR_OUTPUT_METADATA is not 0.

For the case below (Microstep MSI MPG343CQR display) the HDR color space is BT2020_RGB with SMPTE ST 2084 (PQ) EOTF. Max luminance is 446 nits.

$
> drm_info | grep -B 1 -A 11 HDR
$
...
$
│   │       ├───"Colorspace": enum {Default, BT2020_RGB, BT2020_YCC} = BT2020_RGB
$
│   │       ├───"HDR_OUTPUT_METADATA": blob = 114
$
│   │       │   ├───Type: Static Metadata Type 1
$
│   │       │   ├───EOTF: SMPTE ST 2084 (PQ)
$
│   │       │   ├───Display primaries:
$
│   │       │   │   ├───Red: (0.6768, 0.3096)
$
│   │       │   │   ├───Green: (0.2764, 0.6445)
$
│   │       │   │   └───Blue: (0.1514, 0.0693)
$
│   │       │   ├───White point: (0.3135, 0.3291)
$
│   │       │   ├───Max display mastering luminance: 446 cd/m²
$
│   │       │   ├───Min display mastering luminance: 0.0001 cd/m²
$
│   │       │   ├───Max content light level: 446 cd/m²
$
│   │       │   └───Max frame average light level: 446 cd/m²
$
-
$
...

Callbacks

Callbacks can be used to receive updates on key presses, mouse position and buttons, and window size.

1
  visualizer = make_operator<ops::HolovizOp>(
2
              "holoviz",
3
              Arg("key_callback",
4
                  ops::HolovizOp::KeyCallbackFunction([](ops::HolovizOp::Key key,
5
                                                         ops::HolovizOp::KeyAndButtonAction action,
6
                                                         ops::HolovizOp::KeyModifiers modifiers) -> void {
7
                  HOLOSCAN_LOG_INFO(
8
                    "key {} action {} modifiers {}", int(key), int(action), *(uint32_t*)(&modifiers));
9
              }))
10
          )
11
```Python
12
```python
13
  def callback(
14
      key: HolovizOp.Key, action: HolovizOp.KeyAndButtonAction, modifiers: HolovizOp.KeyModifiers
15
  ):
16
      print(key, action, modifiers)
17
   
18
  visualizer = HolovizOp(
19
              self,
20
              name="holoviz",
21
              key_callback=callback)

Scheduling Conditions

Holoviz provides specialized scheduling conditions that enable operators to synchronize their execution with display events. These conditions are particularly useful for rate-limiting and ensuring that frame generation is synchronized with the display’s refresh cycle, reducing latency and improving visual quality.

FirstPixelOut Condition

The C++ (holoscan::FirstPixelOutCondition) allows an operator to synchronize its execution with the time when the first pixel of the next display refresh cycle leaves the display engine for the display (FirstPixelOut). By synchronizing with FirstPixelOut, applications can ensure minimal tearing and optimal frame pacing.

The condition internally waits for the next FirstPixelOut signal from the display hardware. When FirstPixelOut occurs, the condition transitions to a ready state, allowing the operator to execute. After execution, the condition transitions back to a waiting state until the next FirstPixelOut.

Use cases:

• Synchronizing frame generation with display refresh rate
• Minimizing display tearing in real-time visualization
• Rate-limiting operators to match the display’s capabilities

Present Done Condition

The C++ (holoscan::PresentDoneCondition) allows an operator to synchronize its execution with the completion of frame presentation. This condition waits until a specific frame has been fully presented to the display before allowing the next operator execution.

The condition tracks presentation IDs, which increment with each frame presented. It blocks until a specific presentation ID is reached or a timeout occurs, ensuring that the application doesn’t generate frames faster than the display can present them.

Use cases:

• Preventing frame queue buildup by matching generation to presentation rate
• Reducing end-to-end latency in interactive applications
• Ensuring smooth frame pacing without dropped frames

Example Usage

The holoviz_conditions example demonstrates how to use these conditions to rate-limit a source operator:

1
#include <holoscan/operators/holoviz/conditions/present_done.hpp>
2
 
3
void compose() override {
4
  using namespace holoscan;
5
 
6
  auto visualizer = make_operator<ops::HolovizOp>("holoviz");
7
 
8
  auto source = make_operator<ops::SourceOp>(
9
      "source",
10
      // Run on every present completion
11
      make_condition<PresentDoneCondition>("present_limiter", visualizer));
12
 
13
  add_flow(source, visualizer, {{"output", "receivers"}});
14
}

Multiprocess Synchronization

The HoloViz operator supports multiprocess synchronization using a file-based FIFO mutex in the /tmp/ directory. This enables better predictability with the maximum end-to-end latency of the Holoscan SDK applications.

The support for this mutex is enabled by setting the environment variable HOLOSCAN_HOLOVIZ_MUTEX to 1.

$
export HOLOSCAN_HOLOVIZ_MUTEX=1

When the environmentvariable is not set, the HoloViz operator will not use the mutex, and there will be no attempt at explicit multiprocess synchronization.

When using the mutex, developers can optionally enable dropping frames for an upper bound on the latency of the HoloViz operator. Setting HoloViz operator’s multiprocess_framedrop_waittime_ms parameter to a non-zero value will drop frames when the mutex is not available for locking, even after waiting for the specified amount of time in ms. In this case, the number of dropped frames will be logged at the end of an application. Setting the parameter to 0, which is the default value, will not drop any frames.

Holoviz operator

Class Documentation

C++ (holoscan::ops::HolovizOp)

Python (holoscan.operators.HolovizOp).

Holoviz Operator Examples

There are multiple examples, both in Python and C++, showing how to use various features of the Holoviz operator.

Holoviz module

Concepts

The Holoviz module uses the concept of the immediate mode design pattern for its API, inspired by the Dear ImGui library. The difference to the retained mode, for which most APIs are designed, is that there are no objects created and stored by the application. This makes it fast and easy to make visualization changes in a Holoscan application.

Instances

The Holoviz module uses a thread-local instance object to store its internal state. The instance object is created when calling the Holoviz module is first called from a thread. All Holoviz module functions called from that thread use this instance.

When calling into the Holoviz module from other threads other than the thread from which the Holoviz module functions were first called, make sure to call viz::GetCurrent() and viz::SetCurrent() in the respective threads.

There are usage cases where multiple instances are needed; for example, to open multiple windows. Instances can be created by calling viz::Create(). Call viz::SetCurrent() to make the instance current before calling the Holoviz module function to be executed for the window the instance belongs to.

Getting Started

The code below creates a window and displays an image.

First, the Holoviz module needs to be initialized. This is done by calling viz::Init().

The elements to display are defined in the render loop; termination of the loop is checked with viz::WindowShouldClose.

The definition of the displayed content starts with viz::Begin, and ends with viz::End. viz::End starts the rendering and displays the rendered result.

Finally, the Holoviz module is shutdown with viz::Shutdown.

1
#include "holoviz/holoviz.hpp"
2
 
3
namespace viz = holoscan::viz;
4
 
5
viz::Init("Holoviz Example");
6
 
7
while (!viz::WindowShouldClose()) {
8
    viz::Begin();
9
    viz::BeginImageLayer();
10
    viz::ImageHost(width, height, viz::ImageFormat::R8G8B8A8_UNORM, image_data);
11
    viz::EndLayer();
12
    viz::End();
13
}
14
 
15
viz::Shutdown();

Result:

API

namespace_holoscan__viz

Holoviz Module Examples

There are multiple examples showing how to use various features of the Holoviz module.

Fullscreen vs Exclusive Display Mode

Holoviz offers two different modes for rendering to a display taking up the entire screen: fullscreen mode and exclusive display mode. While both modes render content to fill the entire display, they differ significantly in how they interact with the windowing system.

Fullscreen Mode

Fullscreen mode creates a borderless window that covers the entire display using GLFW (Graphics Library Framework). In this mode:

• Window Manager Integration: The window is still managed by the Linux window manager. Keyboard and mouse interactions are supported.
• Desktop Compositor: When the window covers the entire screen, the desktop compositor switches to flip mode, bypassing the compositing step. Therefore, the performance in fullscreen mode is better than in the windowed mode. However, the desktop compositor is still involved in the rendering path.
• Simpler Setup: No special display configuration is required. The application can switch to full-screen mode at runtime.
• G-SYNC Support: G-SYNC is supported in fullscreen mode once it is enabled from NVIDIA Control Panel.

To enable fullscreen mode, set the fullscreen parameter to true:

1
  auto visualizer = make_operator<ops::HolovizOp>("holoviz",
2
      Arg("fullscreen", true)
3
      );

Exclusive Display Mode

Exclusive display mode renders directly to a display using Vulkan’s VK_KHR_display extension, completely bypassing the window manager and desktop compositor. This mode takes up the entire screen, just like the fullscreen mode. However, there are a few differences with the fullscreen mode. In this mode:

• Direct Rendering: The GPU writes pixels to a memory area, and the display controller reads from this memory and sends the pixels to the display. Vulkan’s present call switches which memory region the display controller reads from, bypassing the window manager and compositor.
• Performance: Eliminating the involvement of the window manager anddesktop compositor altogether, this mode provides maximum rendering performance and minimal latency.
• Display Control: Takes exclusive control of the specified display, preventing other applications from using it. Keyboard and mouse interactions are not supported.
• Setup Required: Requires the display to be disabled in the X server configuration or the display manager to be stopped (see Configure a Display for Exclusive Use).
• Vulkan Extensions: Uses Vulkan’s direct display extensions.
• G-SYNC Support: G-SYNC is currently not supported in exclusive display mode.

To enable exclusive display mode, set the use_exclusive_display parameter to true (as shown in Enable Exclusive Display in Holoviz).