BYOM

The Holoscan platform is optimized for performing AI inferencing workflows. This section shows how the user can easily modify the bring_your_own_model example to create their own AI applications.

In this example we’ll cover:

• The usage of FormatConverterOp, InferenceOp, SegmentationPostprocessorOp operators to add AI inference into the workflow.
• How to modify the existing code in this example to create an ultrasound segmentation application to visualize the results from a spinal scoliosis segmentation model.

Operators and Workflow

Here is the diagram of the operators and workflow used in the byom.py example.

The example code already contains the plumbing required to create the pipeline above where the video is loaded by VideoStreamReplayer and passed to two branches. The first branch goes directly to Holoviz to display the original video. The second branch in this workflow goes through AI inferencing and can be used to generate overlays such as bounding boxes, segmentation masks, or text to add additional information.

This second branch has three operators we haven’t yet encountered.

• Format Converter: The input video stream goes through a preprocessing stage to convert the tensors to the appropriate shape/format before being fed into the AI model. It is used here to convert the datatype of the image from uint8 to float32 and resized to match the model’s expectations.

• Inference: This operator performs AI inferencing on the input video stream with the provided model. It supports inferencing of multiple input video streams and models.

• Segmentation Postprocessor: This postprocessing stage takes the output of inference, either with the final softmax layer (multiclass) or sigmoid (2-class), and emits a tensor with uint8 values that contain the highest probability class index. The output of the segmentation postprocessor is then fed into the Holoviz visualizer to create the overlay.

Prerequisites

To follow along this example, you can download the ultrasound dataset with the following commands:

1
./scripts/download_ngc_data --url nvidia/clara-holoscan/holoscan_ultrasound_sample_data:20220608

You can also follow along using your own dataset by adjusting the operator parameters based on your input video and model, and converting your video and model to a format that is understood by Holoscan.

Input Video

The video stream replayer supports reading video files that are encoded as GXF entities. These files are provided with the ultrasound dataset as the ultrasound_256x256.gxf_entities and ultrasound_256x256.gxf_index files.

Input model

Currently, the inference operators in Holoscan are able to load ONNX models, or TensorRT engine files built for the GPU architecture on which you will be running the model. TensorRT engines are automatically generated from ONNX by the operators when the applications run.

If you are converting your model from PyTorch to ONNX, chances are your input is NCHW and will need to be converted to NHWC. We provide an example transformation script named graph_surgeon.py, installed in /opt/nvidia/holoscan/bin or available on GitHub. You may need to modify the dimensions as needed before modifying your model.

Understanding the Application Code

Before modifying the application, let’s look at the existing code to get a better understanding of how it works.

1
  import os
2
  from argparse import ArgumentParser
3
   
4
  from holoscan.core import Application
5
   
6
  from holoscan.operators import (
7
      FormatConverterOp,
8
      HolovizOp,
9
      InferenceOp,
10
      SegmentationPostprocessorOp,
11
      VideoStreamReplayerOp,
12
  )
13
  from holoscan.resources import UnboundedAllocator
14
   
15
   
16
  class BYOMApp(Application):
17
      def __init__(self, data):
18
          """Initialize the application
19
   
20
          Parameters
21
          ----------
22
          data : Location to the data
23
          """
24
   
25
          super().__init__()
26
   
27
          # set name
28
          self.name = "BYOM App"
29
   
30
          if data == "none":
31
              data = os.environ.get("HOLOSCAN_INPUT_PATH", "../data")
32
   
33
          self.sample_data_path = data
34
   
35
          self.model_path = os.path.join(os.path.dirname(__file__), "../model")
36
          self.model_path_map = {
37
              "byom_model": os.path.join(self.model_path, "identity_model.onnx"),
38
          }
39
   
40
          self.video_dir = os.path.join(self.sample_data_path, "racerx")
41
          if not os.path.exists(self.video_dir):
42
              raise ValueError(f"Could not find video data: {self.video_dir=}")

Next, we look at the operators and their parameters defined in the application YAML file.

1
  def compose(self):
2
      host_allocator = UnboundedAllocator(self, name="host_allocator")
3
   
4
      source = VideoStreamReplayerOp(
5
          self, name="replayer", directory=self.video_dir, **self.kwargs("replayer")
6
      )
7
   
8
      preprocessor = FormatConverterOp(
9
          self, name="preprocessor", pool=host_allocator, **self.kwargs("preprocessor")
10
      )
11
   
12
      inference = InferenceOp(
13
          self,
14
          name="inference",
15
          allocator=host_allocator,
16
          model_path_map=self.model_path_map,
17
          **self.kwargs("inference"),
18
      )
19
   
20
      postprocessor = SegmentationPostprocessorOp(
21
          self, name="postprocessor", allocator=host_allocator, **self.kwargs("postprocessor")
22
      )
23
   
24
      viz = HolovizOp(self, name="viz", **self.kwargs("viz"))

Finally, we define the application and workflow.

1
          # Define the workflow
2
          self.add_flow(source, viz, {("output", "receivers")})
3
          self.add_flow(source, preprocessor, {("output", "source_video")})
4
          self.add_flow(preprocessor, inference, {("tensor", "receivers")})
5
          self.add_flow(inference, postprocessor, {("transmitter", "in_tensor")})
6
          self.add_flow(postprocessor, viz, {("out_tensor", "receivers")})
7
   
8
   
9
  def main(config_file, data):
10
      app = BYOMApp(data=data)
11
      # if the --config command line argument was provided, it will override this config_file
12
      app.config(config_file)
13
      app.run()
14
   
15
   
16
  if __name__ == "__main__":
17
      # Parse args
18
      parser = ArgumentParser(description="BYOM demo application.")
19
      parser.add_argument(
20
          "-d",
21
          "--data",
22
          default="none",
23
          help=("Set the data path"),
24
      )
25
   
26
      args = parser.parse_args()
27
      config_file = os.path.join(os.path.dirname(__file__), "byom.yaml")
28
      main(config_file=config_file, data=args.data)

Modifying the Application for Ultrasound Segmentation

To create the ultrasound segmentation application, we need to swap out the input video and model to use the ultrasound files, and adjust the parameters to ensure the input video is resized correctly to the model’s expectations.

We will need to modify the Python and YAML files to change our application to the ultrasound segmentation application.

1
  class BYOMApp(Application):
2
      def __init__(self, data):
3
          """Initialize the application
4
   
5
          Parameters
6
          ----------
7
          data : Location to the data
8
          """
9
   
10
          super().__init__()
11
   
12
          # set name
13
          self.name = "BYOM App"
14
   
15
          if data == "none":
16
              data = os.environ.get("HOLOSCAN_INPUT_PATH", "../data")
17
   
18
          self.sample_data_path = data
19
   
20
          self.model_path = os.path.join(self.sample_data_path, "ultrasound_segmentation", "model")
21
          self.model_path_map = {
22
              "byom_model": os.path.join(self.model_path, "us_unet_256x256_nhwc.onnx"),
23
         }
24
   
25
          self.video_dir = os.path.join(self.sample_data_path, "ultrasound_segmentation", "video")
26
          if not os.path.exists(self.video_dir):
27
              raise ValueError(f"Could not find video data: {self.video_dir=}")

The above changes are enough to update the BYOM example to the ultrasound segmentation application.

In general, when deploying your own AI models, you will need to consider the operators in the second branch. This example uses a pretty typical AI workflow:

• Input: This could be a video on disk, an input stream from a capture device, or other data stream.
• Preprocessing: You may need to preprocess the input stream to convert tensors into the shape and format that is expected by your AI model (e.g., converting datatype and resizing).
• Inference: Your model will need to be in ONNX or TensorRT format.
• Postprocessing: An operator that postprocesses the output of the model to a format that can be readily used by downstream operators.
• Output: The postprocessed stream can be displayed or used by other downstream operators.

The Holoscan SDK comes with a number of built-in operators that you can use to configure your own workflow. If needed, you can write your own custom operators or visit Holohub for additional implementations and ideas for operators.

Running the Application

After modifying the application as instructed above, running the application should bring up the ultrasound video with a segmentation mask overlay similar to the image below.

[error] Error in Inference Manager ... TRT Inference: failed to build TRT engine file.

$
sudo chmod a+w /opt/nvidia/holoscan/examples/bring_your_own_model/model

Customizing the Inference Operator

The built-in InferenceOp operator provides the functionality of the holoinfer. This operator has a receivers port that can connect to any number of upstream ports to allow for multiai inferencing, and one transmitter port to send results downstream. Below is a description of some of the operator’s parameters and a general guidance on how to use them.

• backend: If the input models are in tensorrt engine file format, select trt as the backend. If the input models are in onnx format select either trt or onnx as the backend.
• allocator: Can be passed to this operator to specify how the output tensors are allocated.
• model_path_map: Contains dictionary keys with unique strings that refer to each model. The values are set to the path to the model files on disk. All models must be either in onnx or in tensorrt engine file format. The Holoscan Inference Module will do the onnx to tensorrt model conversion if the TensorRT engine files do not exist.
• pre_processor_map: This dictionary should contain the same keys as model_path_map, mapping to the output tensor name for each model.
• inference_map: This dictionary should contain the same keys as model_path_map, mapping to the output tensor name for each model.
• enable_fp16: Boolean variable indicating if half-precision should be used to speed up inferencing. The default value is False, and uses single-precision (32-bit fp) values.
• input_on_cuda: Indicates whether input tensors are on device or host.
• output_on_cuda: Indicates whether output tensors are on device or host.
• transmit_on_cuda: If True, it means the data transmission from the inference will be on Device, otherwise it means the data transmission from the inference will be on Host.

Common Pitfalls Deploying New Models

Color Channel Order

It is important to know what channel order your model expects. This may be indicated by the training data, pre-training transformations performed at training, or the expected inference format used in your application.

For example, if your inference data is RGB, but your model expects BGR, you will need to add the following to your segmentation_preprocessor in the yaml file: out_channel_order: [2,1,0].

Normalizing Your Data

Similarly, default scaling for streaming data is [0,1], but dependent on how your model was trained, you may be expecting [0,255].

For the above case, you would add the following to your segmentation_preprocessor in the YAML file:

scale_min: 0.0 scale_max: 255.0

Network Output Type

Models often have different output types such as Sigmoid, Softmax, or perhaps something else, and you may need to examine the last few layers of your model to determine which applies to your case.

As in the case of our ultrasound segmentation example above, we added the following in our YAML file: network_output_type: softmax