Overview

NVIDIA Holoscan is the AI sensor processing platform that combines hardware systems for low-latency sensor and network connectivity, optimized libraries for data processing and AI, and core microservices to run streaming, imaging, and other applications, from embedded to edge to cloud. It can be used to build streaming AI pipelines for a variety of domains, including Medical Devices, High Performance Computing at the Edge, Industrial Inspection and more.

Visit the NGC demo website for a live demonstration of some of Holoscan capabilities.

Extensions

Many aspects of the Holoscan SDK are currently implemented within extensions. The extensions packaged in the SDK cover tasks such as IO, machine learning inference, image processing, and visualization. They rely on a set of Core Technologies.

This guide will provide more information on the existing extensions, and how to create your own.

Applications

This SDK includes multiple sample applications to show how users can implement their own end-to-end inference pipeline for streaming use cases, as well as additional “bring your own model” (BYOM) abilities which are modality agnostic. This guide provides detailed information on the inner-workings of those applications, and how to create your own.

See below for some information regarding the sample applications:

Endoscopy Tool Tracking

Based on a LSTM (long-short term memory) stateful model, these applications demonstrate the use of custom components for tool tracking, including composition and rendering of text, tool position, and mask (as heatmap) combined with the original video stream.

Ultrasound Segmentation

Full workflow including a generic visualization of segmentation results from a spinal scoliosis segmentation model of ultrasound videos. The model used is stateless, so this workflow could be configured to adapt to any vanilla DNN model. These applications comes with the support of an AJA capture card or replay from a video file included in the sample application container.

Colonoscopy Polyp Segmentation

As an example of the BYOM ability mentioned above, we show how the same code used for ultrasound segmentation may be used for a polyp segmentation application.

This model was trained on the Kvasir-SEG dataset ¹, using the ColonSegNet model architecture ².

Refer to the sample data resource on NGC for more information related to the model and video.

Hi-Speed Endoscopy

The example app showcases how high resolution cameras can be used to capture the scene, post-processed on GPU and displayed at high frame rate. This app requires Emergent Vision Technologies camera and a display with high refresh rate to keep up with the camera’s framerate.

Multi AI Ultrasound

Demonstrates how to run multiple inference pipelines in a single application by leveraging the Holoscan Inference module, a framework that facilitates designing and executing inference applications in the Holoscan SDK. The Multi AI operators (inference and postprocessor) use APIs from the Holoscan Inference module to extract data, initialize and execute the inference workflow, process, and transmit data for visualization. The applications uses models and echocardiogram data from iCardio.ai. The models include:

• a Plax chamber model, that identifies four critical linear measurements of the heart

• a Viewpoint Classifier model, that determines confidence of each frame to known 28 cardiac anatomical view as defined by the guidelines of the American Society of Echocardiography

• an Aortic Stenosis Classification model, that provides a score which determines likeability for the presence of aortic stenosis

C++ And Python APIs

The Holoscan SDK also includes C++ and Python APIs for the creation of applications. These APIs are designed to be user-friendly and flexible to use.

Please see the Using the SDK section for more information.

Video Pipeline Latency Tool

To help developers make sense of the overall end-to-end latency that could be added to a video stream by augmenting it through a GPU-powered Holoscan platform such as the NVIDIA IGX Orin Developer Kit, the Holoscan SDK includes a Video Pipeline Latency Measurement Tool. This tool can be used to measure and estimate the total end-to-end latency of a video streaming application including the video capture, processing, and output using various hardware and software components that are supported by the Holoscan Developer Kits. The measurements taken by this tool can then be displayed with a comprehensive and easy-to-read visualization of the data.

High-level changes are described in the Using the SDK section. More detailed changes can be found in the Holoscan SDK release notes on Github.

---