9.5. Clara Deploy Base Inference Operator V2

This asset requires the Clara Deploy SDK. Follow the instructions on the Clara Bootstrap page to install the Clara Deploy SDK.

9.5.1. Overview

The NVIDIA Clara Train SDK and MMAR provide pre-trained models unique to medical imaging, with additional capabilities such as integration with the AI-assisted Annotation SDK for increasing the annotation speed of medical images. This allows the access to AI-assisted labeling [Reference].

To accelerate the deployment of Clara Train pre-trained models using Clara Deploy SDK, this containerized AI inference application was developed as a base container, which can be customized for deploying a specific pre-trained model. The customized container can then be used as the AI inference operator in Clara Deploy pipelines.

Customizing this base container requires the inference or validation configuration file used during model training with Clara Train. In addition, the trained model must have been exported using a format compatible with TRITON (formerly TRTIS), the TensorRT Inference Server. Steps on how to create model specific containers are provided in the following sections.

This base inference application uses the same set of transform functions and the same scanning window inference logic as Clara Train SDK 3.0. The output writer, however, is specific to Clara Deploy due to the need to support registration of Clara Deploy pipeline results.

9.5.1.1. Version information

This base inference application is targeted to run in the following environment:

  • Ubuntu 18.04
  • Python 3.6
  • NVIDIA TensorRT Inference Server Release 1.5.0, container version 19.08

9.5.2. Inputs

This application, in the form of a Docker container, expects an input folder (/input by default), which can be mapped to the host volume when the Docker container is started. This folder should contain a volume image file in NIfTI or MetaImage format. Furthermore, the volume image should be constructed from a single series of a DICOM study, typically the axial series with the data type of the original primary.

9.5.3. Outputs

This application saves the segmentation results to an output folder (/output by default), which also can be mapped to a folder on the host volume. After the application completes successfully, a segmentation volume image in MetaImage format is saved in the output folder.

The name of the output file is the same as that of the input file, due to certain limitations of the downstream operator in Clara Deploy SDK.

This container also publishes data for the Clara Deploy Render Server in the /publish folder by default. The original volume image, segmented volume image, along with config files for the Render Server, are saved in this folder.

9.5.4. AI Model

For testing, this base application uses a model trained using the NVIDIA Clara Train SDK V3.0 for lung segmentation, namely segmentation_ct_lung_v1. It is converted from a TensorFlow Checkpoint model to tensorflow_graphdef using the Clara Train SDK model export tool. The input tensor is of the shape 320 x 320 x 64 with a single channel, and the output is of the same shape with two channels.

The key model attributes (e.g. the model name) must be present in the config_inference.json file and is consumed by this application at runtime.

9.5.4.1. NVIDIA Triton Inference Server

This application performs inference on Triton (formerly known as TRTIS), the NVIDIA Triton Inference Server, which provides an inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. You can read more on Triton here.

9.5.5. Directory Structure

The directories in the container are shown below.

/
├── app_base_inference_v2
    ├── ai4med
    ├── config
    │   ├── config_render.json
    │   ├── config_inference.json
    │   └── __init__.py
    ├── inferers
    ├── model_loaders
    ├── ngc
    ├── public
    │   └── docs
    │       └── README.md
    ├── utils
    ├── writers
    │   ├── __init__.py
    |   ├── classification_result_writer.py
    │   ├── mhd_writer.py
    │   └── writer.py
    ├── app.py
    ├── Dockerfile
    ├── executor.py
    ├── logging_config.json
    ├── main.py
    └── requirements.txt
/input
/output
/publish
/logs

The app_base_inference_v2 contains the base application source code:

  • The ai4med directory contains compiled modules from Clara Train SDK.
  • The config directory contains model-specific configuration files, which need to be replaced when building a customized container for a specific model.
    • The config_inference.json file contains the configuration sections for pre- and post-transforms, as well as the model loader, referer, and writer.
    • The config_render.json contains the configuration for the Clara Deploy Render Server.
  • The inferers and model_loaders directories contain implementation of Triton API client for inference as well as model status.
  • The public directory contains the user documentation.
  • The utils directory contains utilities for loading modules and creating application objects.
  • The Writers directory contains the specialized output writer required by Clara Deploy SDK, which saves the segmentation result to a volume image file as MetaImage.

The model name must be correctly specified in the inferer attribute in the config_inference.json file, as shown in the following example:

"inferer":
{
    "name": "TRTISScanWindowInferer",
    "args": {
        "model_name": "segmentation_ct_lung_v1",
        "ip": "localhost",
        "port": 8000,
        "protocol": "HTTP"
        "output_type": "RAW"
    }
}

9.5.6. Executing Locally

To see the internals of the container, and to manually run the application, follow these steps. Please note that the Triton server with the required model must be accessible from within this container–otherwise, a failure will occur.

  1. Start the container in interactive mode. See the next section on how to run the container, and replace the docker run command with docker run -it --entrypoint /bin/bash
  2. Once in the Docker terminal, ensure the current directory is /.
  3. Execute the command python ./app_base_inference_v2/main.py".
  4. Once finished, type exit.

9.5.7. Executing in Docker

9.5.7.1. Prerequisites

  1. Use the docker images command to check that the Docker image of Triton has been imported into the local Docker repository. Look for the image name TRTIS and the correct tag for the release (e.g. 19.08-py3). The Docker image can also be pulled from NVIDIA if not present locally.
  2. Ensure that the model folder, including the config.pbtxt, is present on the Clara Deploy host. Verify it using the following steps:
    • Log on to the Clara Deploy host.
    • Check for the folder segmentation_ct_lung_v1 under the directory /clara/common/models.

9.5.7.2. Step 1

Change to your working directory (e.g. test).

9.5.7.3. Step 2

Create, if they do not exist, the following directories under your working directory:

  • input containing the input image file.
  • output for the AI inference output.
  • publish for publishing data for the Render Server.
  • logs for the log files.
  • models for models, and copy over segmentation_ct_lung_v1 folder.

9.5.7.4. Step 3

Note: If this base inference application container has already been pulled from NGC, tag the container:

docker tag <pulled base container> app_base_inference_v2:latest

In your working directory, create a shell script( e.g. run_base_docker.sh) and copy the content below. Comment out or remove the command in the script that builds the container if it has been pulled from NGC.

#!/bin/bash

# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
#
# NVIDIA CORPORATION and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.

# Clara Core would launch the container with the following environment variables internally,
# to provide runtime information.
export NVIDIA_CLARA_TRTISURI="localhost:8000"

APP_NAME="app_base_inference_v2"
TRTIS_IMAGE="nvcr.io/nvidia/tensorrtserver:19.08-py3"

MODEL_NAME="segmentation_ct_lung_v1"
NETWORK_NAME="container-demo"

# Install prerequisites.
# Note: Remove this command if the image has been pulled from NGC.
. envsetup.sh

# Create network
docker network create ${NETWORK_NAME}

# Run TRTIS(name: trtis), maping ./models/${MODEL_NAME} to /models/${MODEL_NAME}
# (localhost:8000 will be used)
nvidia-docker run --name trtis --network ${NETWORK_NAME} -d --rm --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
    -p 8000:8000 \
    -v $(pwd)/models/${MODEL_NAME}:/models/${MODEL_NAME} ${TRTIS_IMAGE} \
    trtserver --model-store=/models

# Build Dockerfile.
# Note: Remove this command if the image has been pulled from NGC.
docker build -t ${APP_NAME} -f ${APP_NAME}/Dockerfile .

# Wait until TRTIS is ready
trtis_local_uri=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' trtis)
echo -n "Wait until TRTIS ${trtis_local_uri} is ready..."
while [ $(curl -s ${trtis_local_uri}:8000/api/status | grep -c SERVER_READY) -eq 0 ]; do
    sleep 1
    echo -n "."
done
echo "done"

export NVIDIA_CLARA_TRTISURI="${trtis_local_uri}:8000"

# Run ${APP_NAME} container.
# Like below, Clara Core would launch the app container with the following environment variables internally,
# to provide input/output path information.
# (They are subject to change. Do not use the environment variables directly in your application!)
docker run --name ${APP_NAME} --network ${NETWORK_NAME} -t --rm \
    -v $(pwd)/input:/input \
    -v $(pwd)/output:/output \
    -v $(pwd)/logs:/logs \
    -v $(pwd)/publish:/publish \
    -e NVIDIA_CLARA_TRTISURI \
    -e DEBUG_VSCODE \
    -e DEBUG_VSCODE_PORT \
    -e NVIDIA_CLARA_NOSYNCLOCK=TRUE \
    ${APP_NAME}

echo "${APP_NAME} has finished."

# Stop TRTIS container
echo "Stopping TRTIS"
docker stop trtis > /dev/null

# Remove network
docker network remove ${NETWORK_NAME} > /dev/null

9.5.7.5. Step 4

Execute the created script:

./run_base_docker.sh

Wait for the application container to finish.

9.5.7.6. Step 5

Check for the following output files.

In the output folder (whose contents will be consumed by DICOM object writers in a Clara Deploy pipeline):

  • Segmentation image file, both .mhd and .raw for MetaImage format. File name may appear to be the same as the input

In the publish folder (whose contents will be registered for Render Server in a Clara Deploy pipeline):

  • The original volume image (e.g. image.mhd and image.raw)
  • The segmentation volume image (e.g. image.out.mhd and image.out.raw)
  • The rendering configuration file (config_render.json)
  • A metadata file describing the other files (config.meta)

9.5.8. Creating Model Specific Application

This section describes how to use the base application container to build a model-specific container to deploy Clara pre-trained models.

9.5.8.1. Prerequisites

First, prepare data files using Clara Train SDK:

  • With the Export tool, export the trained model to a platform compatible with TRTIS (e.g. tensorflow_graphdef). The server side configuration file, config.pbtxt, must also be generated. For details, please refer to the Triton and Clara Train SDK documentation.
  • The validation and inference pipeline configuration file must be available.
  • A test dataset of the volume image, in NIfTI or MetaImage format, is available for testing the container directly.
  • A test dataset of the DICOM studies is available for testing the Clara Deploy pipeline created with the customized application as its inference operator.

9.5.8.2. Steps

9.5.8.2.1. Step 1

Pull the base application container into the local Docker registry, if not already present.

9.5.8.2.2. Step 2

Create a Python project, e.g. my_custom_app, with the folder structure shown below:

my_custom_app
├── config
│   ├── config_inference.json
│   └── config_render.json
├── Dockerfile
└── public
    └── docs
        └── README.md

where the config_render.json contains the transfer functions for the redenring, and config_inference.json can be copied from the configuration file used during training validation and modified in the next step.

9.5.8.2.3. Step 3

Open the config_inference.json file. Keep the pre_transforms and post_transforms sections as is, but change the name of the inferer and model_loader sections as shown below. The model_name must be changed to the model used in the inference:

"inferer":
{
    "name": "TRTISScanWindowInferer",
    "args": {
        "model_name": "segmentation_ct_lung_v1",
        "ip": "localhost",
        "port": 8000,
        "protocol": "HTTP"
    }
},

"model_loader":
{
    "name": "TRTISModelLoader",
    "args": {
        "model_spec_file_name": "{PBTXT_PATH}"
    }
}

9.5.8.2.4. Step 4

Open the Dockerfile and update it with the content shown below.

Note: Update the actual app_base_inference_v2 container name and tag if they are different in your environment.

# Build upon the named base container; version tag can be used if known.
FROM app_base_inference_v2:latest

# This is a well known folder in the base container. Please do not change it.
ENV BASE_NAME="app_base_inference_v2"

# This is the name of the folder containing the config files; same as the app name.
ENV MY_APP_NAME="my_custom_app"

WORKDIR /app

# Copy configuration files to overwrite base defaults
COPY ./$MY_APP_NAME/config/* ./$BASE_NAME/config/

9.5.8.2.5. Step 5

Build the customized container with the following command, or run the command using the shell script.

APP_NAME="my_custom_app"
docker build -t ${APP_NAME} -f ${APP_NAME}/Dockerfile .

9.5.9. License

An End User License Agreement is included with the product. By pulling and using the Clara Deploy asset on NGC, you accept the terms and conditions of these licenses.

9.5.10. Suggested Reading

Release Notes, the Getting Started Guide, and the SDK itself are available at the NVIDIA Developer forum: (https://developer.nvidia.com/clara).

For answers to any questions you may have about this release, visit the NVIDIA Devtalk forum: (https://devtalk.nvidia.com/default/board/362/clara-sdk/).