NVIDIA Clara Holoscan is the AI computing platform for medical devices, consisting of Clara Developer Kits and the Clara Holoscan SDK. Clara Holoscan allows medical device developers to create the next-generation of AI-enabled medical devices.
The Clara Holoscan SDK version 0.3.0 provides the foundation to run streaming applications on Clara Developer Kits, enabling real-time AI inference and fast IO. These capabilities are showcased within the reference extensions and applications described below.
Extensions
The core of the Clara Holoscan SDK is implemented within extensions. The extensions packaged in the SDK cover tasks such as IO, machine learning inference, image processing, and visualization. They rely on a set of Core Technologies.
This guide will provide more information on the existing extensions, and how to create your own.
Applications
This SDK includes two core sample applications to show how users can implement their own end-to-end inference pipeline for streaming use cases, as well as an additional “bring your own model” (BYOM) segmentation ability which is modality agnostic. This guide provides detailed information on the inner-workings of those applications, and how to create your own.
See below for some information regarding the sample applications:
Endoscopy Tool Tracking
Leveraging a long-short term memory (LSTM) stateful model, this application demonstrates the use of custom components for surgical tool tracking and classification, as well as composition and rendering of text, tool position, and mask (as heatmap) overlayed on the original frames. This guide provides more details on the inner-workings of the Endoscopy Tool Tracking application.
The convolutional LSTM model and sample surgical video data were kindly provided by Research Group Camma, IHU Strasbourg & University of Strasbourg:
Nwoye, C.I., Mutter, D., Marescaux, J. and Padoy, N., 2019. Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. International journal of computer assisted radiology and surgery, 14(6), pp.1059-1067
Refer to the sample data resource on NGC for more information related to the model and video.
Ultrasound Segmentation
Generic visualization of segmentation results based on a spinal scoliosis segmentation model of ultrasound videos. The model used is stateless, so this workflow could be configured to adapt to any vanilla DNN model. This guide will provide more details on the inner-workings of the Ultrasound Segmentation application and how to adjust it to use your own data.
The model is from a King’s College London research project, created by Richard Brown and released under the Apache 2.0 license.
The ultrasound dataset is released under the CC BY 4.0 license. When using this data, please cite the following paper:
Ungi et al., “Automatic Spine Ultrasound Segmentation for Scoliosis Visualization and Measurement,” in IEEE Transactions on Biomedical Engineering, vol. 67, no. 11, pp. 3234-3241, Nov. 2020, doi: 10.1109/TBME.2020.2980540.
Refer to the sample data resource on NGC for more information related to the model and video.
Colonoscopy Polyp Segmentation
As an example of the BYOM ability mentioned above, we show how the same code used for ultrasound segmentation may be used for a polyp segmentation application.
This model was trained on the Kvasir-SEG dataset [1], using the ColonSegNet model architecture [2].
Refer to the sample data resource on NGC for more information related to the model and video.
Video Pipeline Latency Tool
To help developers make sense of the overall end-to-end latency that could be added to a video stream by augmenting it through a GPU-powered Holoscan platform such as the NVIDIA IGX Orin Developer Kit, the Holoscan SDK includes a Video Pipeline Latency Measurement Tool. This tool can be used to measure and estimate the total end-to-end latency of a video streaming application including the video capture, processing, and output using various hardware and software components that are supported by Clara Holoscan platforms. The measurements taken by this tool can then be displayed with a comprehensive and easy-to-read visualization of the data.
The following table outlines the component versions that have been upgraded or removed in version 0.3.0:
Component |
Holoscan 0.3.0 |
Holoscan 0.2.0 |
|---|---|---|
Jetpack |
Holopack 1.1 |
JP5 HP1 |
GXF[3] |
2.4.3 |
2.4.2 |
Holoscan C++ API
The most significant change in Holoscan 0.3.0 is the addition of a new C++ API for the creation of GXF extensions, giving developers an additional pathway to building their desired applications.
Jha, Debesh, Pia H. Smedsrud, Michael A. Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D. Johansen, “Kvasir-seg: A segmented polyp dataset” Proceedings of the International Conference on Multimedia Modeling, pp. 451-462, 2020.
Jha D, Ali S, Tomar NK, Johansen HD, Johansen D, Rittscher J, Riegler MA, Halvorsen P. Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning. IEEE Access. 2021 Mar 4;9:40496-40510. doi: 10.1109/ACCESS.2021.3063716. PMID: 33747684; PMCID: PMC7968127.
NVIDIA Graph eXecution Framework (GXF)