Clara Holoscan SDK is a collection of documentation, drivers, tools, and reference applications that has been designed by NVIDIA to help developers design and build end-to-end streaming workflows and hardware platforms for medical imaging applications. From building a streaming inference application using DeepStream, to minimizing end-to-end latency using latency measurement tools and GPUDirect RDMA-optimized drivers, to deployment on a Clara AGX embedded device; the Clara Holoscan SDK takes an incremetal approach to introduce concepts and NVIDIA technology in a way that can be applied irrespective of previous development experience in the field of medical imaging.
While the ultimate goal of the Clara Holoscan SDK is to deploy applications on a Clara AGX embedded device, the SDK includes x86 Linux versions of the sample applications in order to provide a cross-platform development environment that can be used even without access to a Clara AGX device. In any case, an NVIDIA GPU will be required in order to utilize DeepStream, TensorRT, CUDA, and any other NVIDIA software components that are used by the Clara Holoscan SDK (see Prerequisites).
The Clara Holoscan SDK has been divided into a number of separate installation packages, as follows.
DeepStream Sample Applications
Building a powerful end-to-end inference application using DeepStream and TensorRT requires the use of a number of different software libraries that can be overwhelming if starting with no knowledge of these libraries. To help alleviate this problem, the Holoscan SDK includes three different sample applications that build on top of one another to incrementally introduce new components and features.
All of the sample applications included with the Holoscan SDK are written in C and/or C++.
GStreamer Sample
The GStreamer sample application is the most basic sample which demonstrates the use of the GStreamer framework in order to simply capture image frames from a camera and then render the captured video stream onto the display. This sample does not make use of DeepStream, but since DeepStream functionality is exposed to applications via GStreamer framework plugins, a basic understanding of GStreamer is essential to write DeepStream applications.
The GStreamer sample is available for the Clara AGX and Linux host, and are
located at /opt/nvidia/clara-holoscan-sdk/clara-holoscan-deepstream-sample/GStreamerJetsonSample
on AGX, and /opt/nvidia/clara-holoscan-sdk/clara-holoscan-deepstream-sample/GStreamerLinuxHostSample
on the host machine.
DeepStream Sample
The DeepStream sample application builds on top of the GStreamer sample by
including the DeepStream inference plugin into the pipeline such that the
stomach and intestines regions of a sample model are
identified by bounding boxes rendered on top of the video stream. Alternatively,
the sample model can be replaced with a user-provided model (by changing a
configuration file used by the sample) to identify regions specified by the
custom model.
This sample also expands on the GStreamer sample by providing multi-camera support, as well as the ability to send the captured and annotated video across a UDP H.264 video stream such that it can be viewed by another remote system.
The DeepStream sample is available for Clara AGX and the Linux host, and are
located at /opt/nvidia/clara-holoscan-sdk/clara-holoscan-deepstream-sample/DeepstreamJetsonAGXSample
on AGX, and /opt/nvidia/clara-holoscan-sdk/clara-holoscan-deepstream-sample/DeepstreamLinuxHostSample
on the host machine.
The DeepStream sample is explored in more detail here: DeepStream Sample Application.
Endoscopy Sample
The Endoscopy sample builds on top of the DeepStream sample by adding runtime controls that are specific to the model and inferencing that is performed by DeepStream, such as being able to enable or disable the annotation of individual inference regions that are provided by the model. The UI is also modified to provide a more streamlined experience that might be desirable for an application that is deployed into medical imaging environments.
The Endoscopy sample is available only for Clara AGX, and is located here:
/opt/nvidia/clara-holoscan-sdk/clara-holoscan-deepstream-sample/EndoscopyJetsonAGXSample.
The Endoscopy sample is explored in more detail here: Endoscopy Sample Application.
DeepStream GPU Video Source Plugin
To help developers get started with writing GPU-enabled GStreamer plugins that interface with DeepStream and/or use GPUDirect RDMA for efficient data transfer between the GPU and other peripheral devices, the Clara Holoscan SDK includes the DeepStream GPU Video Source Plugin. This sample plugin produces a stream of GPU buffers that are allocated, filled with a test pattern using CUDA, and then output by the plugin such that they can be consumed by other DeepStream or GPU-enabled components. Passing GPU buffers through the pipeline means that data can flow between components directly in GPU memory, avoiding any copies to or from system memory throughout the pipeline.
Video Pipeline Latency Tool
To help developers make sense of the overall end-to-end latency that could be added to a video stream by augmenting it through a GPU-powered Holoscan platform such as the Clara AGX Developer Kit, the Holoscan SDK includes a Video Pipeline Latency Measurement Tool. This tool can be used to measure and estimate the total end-to-end latency of a video streaming application including the video capture, processing, and output using various hardware and software components that are supported by Clara Holoscan platforms. The measurements taken by this tool can then be displayed with a comprehensive and easy-to-read visualization of the data.
