Visualization Module

Holoviz is a module of the Holoscan SDK for visualizing data. Holoviz composites real time streams of frames with multiple different other layers like segmentation mask layers, geometry layers and GUI layers.

For maximum performance Holoviz makes use of Vulkan, which is already installed as part of the Nvidia GPU driver.

Holoviz uses the concept of the immediate mode design pattern for its API, inspired by the Dear ImGui library. The difference to the retained mode, for which most APIs are designed for, is, that there are no objects created and stored by the application. This makes it fast and easy to make visualization changes in a Holoscan application.

The code below creates a window and displays an image.

First Holoviz needs to be initialized. This is done by calling viz::Init().

The elements to display are defined in the render loop, termination of the loop is checked with viz::WindowShouldClose().

The definition of the displayed content starts with viz::Begin() and ends with viz::End(). viz::End() starts the rendering and displays the rendered result.

Finally Holoviz is shutdown with viz::Shutdown().

            

            
#include "holoviz/holoviz.hpp"

namespace viz = holoscan::viz;

viz::Init("Holoviz Example");

while (!viz::WindowShouldClose()) {
    viz::Begin();
    viz::BeginImageLayer();
    viz::ImageHost(width, height, viz::ImageFormat::R8G8B8A8_UNORM, image_data);
    viz::EndLayer();
    viz::End();
}

viz::Shutdown();
        

Result:

Fig. 19 Holoviz example app

The core entity of Holoviz are layers. A layer is a two-dimensional image object, multiple layers are composited to form the final output.

These layer types are supported by Holoviz:

• image layer

• geometry layer

• GUI layer

The definition of a layer is started by calling one of the layer begin functions viz::BeginImageLayer(), viz::BeginGeometryLayer() or viz::BeginImGuiLayer(). The layer definition ends with viz::EndLayer().

All layers have common attributes, these are priority and opacity. The start of a layer definition is resetting these values to their defaults.

The priority determines the rendering order of the layers. Before rendering the layers they are sorted by priority, the layers with the lowest priority are rendered first so that the layer with the highest priority is rendered on top of all other layers. If layers have the same priority then the render order of these layers is undefined. Priority is set by calling viz::LayerPriority().

Opacity is used to blend transparent layers over other layers. Opacity is set by calling viz::LayerOpacity().

The code below draws a transparent geometry layer on top of an image layer (layer details are omitted). Although the geometry layer is specified first, it is drawn last because it has a higher priority (1) than the image layer (0). As mentioned the start of a layer is resetting layer attributes, so for the image layer, there is no need to set the opacity to 1.0 since the default is already 1.0.

            

            
namespace viz = holoscan::viz;

viz::Begin();

viz::BeginGeometryLayer();
viz::LayerPriority(1);
viz::LayerOpacity(0.5);
/// details omitted
viz::EndLayer();

viz::BeginImageLayer();
viz::LayerPriority(0);
/// details omitted
viz::EndLayer();

viz::End();
        

Image Layers

The function viz::BeginImageLayer() starts an image layer. An image layer displays a rectangular 2D image.

The image data is defined by calling viz::ImageCudaDevice() and viz::ImageHost(). Various input formats are supported, see viz::ImageFormat. For single channel image formats image colors can be looked up by defining a lookup table with viz::LUT().

Geometry Layers

The function viz::BeginGeometryLayer() starts a geometry layer. A geometry layer is used to draw geometric primitives such as points, lines, rectangles, ovals or text. See viz::PrimitiveTopology for supported geometry primitive topologies. Coordinates start with (0, 0) in the top left and end with (1, 1) in the bottom right.

There are functions to set attributes for geometric primitives like color (viz::Color()), line width (viz::LineWidth()) and point size (viz::PointSize()).

The code below draws a red rectangle and a green text.

            

            
namespace viz = holoscan::viz;

viz::BeginGeometryLayer();

// draw a red rectangle
viz::Color(1.f, 0.f, 0.f, 0.f);
const float data[]{0.1f, 0.1f, 0.9f, 0.9f};
viz::Primitive(viz::PrimitiveTopology::RECTANGLE_LIST, 1, sizeof(data) / sizeof(data[0]), data);

// draw green text
viz::Color(0.f, 1.f, 0.f, 0.f);
viz::Text(0.5f, 0.5f, 0.2f, "Text");

viz::EndLayer();
        

ImGui Layers

Holoviz support user interface layers created with Dear ImGui.

If using Dear ImGui, create a context and pass it to Holoviz using viz::ImGuiSetCurrentContext(), do this before calling viz::Init(). Background: the Dear ImGui context is a global variable. Global variables are not shared across so/DLL boundaries. Therefore the app needs to create the Dear ImGui context first and then provide the pointer to Holoviz like this:

            

            
ImGui::CreateContext();
holoscan::viz::ImGuiSetCurrentContext(ImGui::GetCurrentContext());
        

Calls to the Dear ImGui API are allowed between viz::BeginImGuiLayer() and viz::EndImGuiLayer() are used to draw to the ImGui layer. The ImGui layer behaves like other layers and is rendered with the layer opacity and priority.

The code below creates a Dear ImGui window with a checkbox used to conditionally show a image layer.

            

            
namespace viz = holoscan::viz;

bool show_image_layer = false;
while (!viz::WindowShouldClose()) {
    viz::Begin();

    viz::BeginImGuiLayer();

    ImGui::Begin("Options");
    ImGui::Checkbox("Image layer", &show_image_layer);
    ImGui::End();

    viz::EndLayer();

    if (show_image_layer) {
        viz::BeginImageLayer();
        viz::ImageHost(...);
        viz::EndLayer();
    }

    viz::End();
}
        

Depth Map Layers

A depth map is single channel 2d array where each element represents a depth value. The data is specified with viz::DepthMap() and rendered as a 3d object using points, lines or triangles. The color for the elements can also be specified.

Supported format for the depth map:

• 8-bit unsigned normalized format that has a single 8-bit depth component

Supported format for the depth color map:

• 32-bit unsigned normalized format that has an 8-bit R component in byte 0, an 8-bit G component in byte 1, an 8-bit B component in byte 2, and an 8-bit A component in byte 3

Depth maps are rendered in 3D and support camera movement.

The camera is operated using the mouse.

• Orbit (LMB)

• Pan (LMB + CTRL | MMB)

• Dolly (LMB + SHIFT | RMB | Mouse wheel)

• Look Around (LMB + ALT | LMB + CTRL + SHIFT)

• Zoom (Mouse wheel + SHIFT)

Usually Holoviz opens a normal window on the Linux desktop. In that case the desktop compositor is combining the Holoviz image with all other elements on the desktop. To avoid this extra compositing step, Holoviz can render to a display directly.

Configure a display for exclusive use

            

            
sudo systemctl stop display-manager
        

            

            
sudo systemctl start display-manager
        

Enable exclusive display in Holoviz

Provide the name of the display and desired display mode properties to viz::Init().

For example, here are parameters to pass to the Holoviz Operator:

            

            
  # required
  use_exclusive_display: true
  # optional
  display_name: DP-2
  width: 2560
  height: 1440
  framerate: 240
        

The name of the display can either be the EDID name as displayed in the NVIDIA Settings, or the output name used by xrandr. If the name is nullptr then the first display is selected.

            

            
Screen 0: minimum 8 x 8, current 4480 x 1440, maximum 32767 x 32767
DP-0 disconnected (normal left inverted right x axis y axis)
DP-1 disconnected (normal left inverted right x axis y axis)
DP-2 connected primary 2560x1440+1920+0 (normal left inverted right x axis y axis) 600mm x 340mm
   2560x1440     59.98 + 239.97*  199.99   144.00   120.00    99.95
   1024x768      60.00
   800x600       60.32
   640x480       59.94
USB-C-0 disconnected (normal left inverted right x axis y axis)
        

When providing Cuda resources to Holoviz through e.g. viz::ImageCudaDevice() Holoviz is using Cuda operations to use that memory. The Cuda stream used by this operations can be set by calling viz::SetCudaStream(). The stream can be changed at any time.

The rendered framebuffer can be read back using viz::ReadFramebuffer().