Abstract

The AI Assisted Annotation SDK Getting Started Guide describes how to use deep learning techniques to approximate points as input to return an auto-annotated set of slices of medical images.

1. Overview

The NVIDIA Clara Train SDK with AI-Assisted Annotation uses deep learning techniques to take points of interest drawn by radiologists to approximate points as input along with the 3D volume data to return an auto-annotated set of slices. The auto-annotation step is achieved using NVIDIA’s pre-trained deep learning models for different organs. Neither application developers nor radiologists need to have knowledge of deep learning to benefit from NVIDIA’s deep learning expertise out of the box.

The Clara Train SDK works in conjunction with the Transfer Learning Toolkit for medical imaging. If you are a developer or an engineer developing medical image analysis applications for healthcare providers, this guide can help you to get started with integrating the AI-Assisted Annotation SDK into your existing custom applications or into existing medical imaging applications such as MITK, or ITK-Snap, without any prior deep learning knowledge.

2. Requirements and installation of the SDK

Using the Clara Train SDK requires the following:

Hardware Requirements

Recommended

  • 1 GPU or more
  • 16 GB GPU memory
  • 8 core CPU
  • 32 GB system RAM
  • 80 GB free disk space

Software Requirements

Installation Prerequisites

Access registration

Get an NGC API Key

  • NVIDIA GPU Cloud account and API key - https://ngc.nvidia.com/
    1. Go to NGC and search for Clara Train container in the Catalog tab. This message is displayed, Sign in to access the PULL feature of this repository.
    2. Enter your email address and click Next or click Create an Account.
    3. Click Sign In.
    4. Click the Clara Train SDK tile.
    Note: Save the API key in a secure location. You will need it to use the AI assisted annotation SDK.

Download the docker container

  • Execute docker login nvcr.io from the command line and enter your username and password.
    • Username: $oauthtoken
    • Password: API_KEY
  • dockerImage=nvcr.io/nvidia/clara-train-sdk:v1.0-py3
  • docker pull ${dockerImage}

Running the container

Once downloaded, run the docker using this command:

docker run -it --rm --ipc=host --net=host --runtime=nvidia --mount type=bind,source=/your/dataset/location,target=/workspace/data $dockerImage /bin/bash
Note: If you are on a network that uses a proxy server to connect to the Internet, you can provide proxy server details when launching the container.
 docker run --runtime=nvidia -it --rm -e HTTPS_PROXY=https_proxy_server_ip:https_proxy_server_port -e HTTP_PROXY=http_proxy_server_ip:http_proxy_server_port $dockerImage /bin/bash

The docker, by default, starts in the /opt/nvidia folder. To access local directories from within the docker, they have to be mounted in the docker. To mount a directory, use the -v <source_dir>:<mount_dir> option. For more information, see Bind Mounts. Here is an example:

docker run --runtime=nvidia -it --rm -v /home/<username>/tlt-experiments:/workspace/tlt-experiments $dockerImage /bin/bash

This mounts the /home/<username>/tlt-experiments directory in your disk to /workspace/tlt-experiments in docker.

Using the annotation server

This chapter describes tasks and information needed to use the Annotation server.

Annotation server setup

Ways of running the annotation server

Use the proper account credentials, and the container image name, to pull and run the container for the annotation server as follows:

Run the annotation server with Default options (models will be saved inside docker hence will not persist).

docker run $NVIDIA_RUNTIME \
       -it --rm -p 5000:5000 \
       $DOCKER_IMAGE \
       start_aas.sh

Run the annotation Server with Advanced options (e.g. mount workspace from host machine to persist models/logs/configs).

export AIAA_SERVER_PORT=5000
export LOCAL_WORKSPACE=/var/nvidia/aiaa
export REMOTE_WORKSPACE=/workspace
docker run $NVIDIA_RUNTIME \
       -it --rm -p $AIAA_SERVER_PORT:5000 \
       -v $LOCAL_WORKSPACE:$REMOTE_WORKSPACE \
       $DOCKER_IMAGE \
       start_aas.sh \
             --workspace $REMOTE_WORKSPACE \
             --port $AIAA_SERVER_PORT

Workspace

The annotation server uses workspace directory (if not specified default shall be insider docker at the path: /var/nvidia/aiaa) for saving configs, models (Frozen), logs etc.

You can shutdown and run the docker multiple times, if you mount an external workspace path while running the docker. See the advanced options while running the docker.

Following are files/folders from workspace directory.

Name Description
aiaa_server_config.json The annotation server supports automatic loading of models and corresponding configs are saved in this file.
aiaa_server_dashboard.db The annotation Web/API activities are monitored through Flask Dashboard and corresponding data is saved here.
downloads Temporary downloads from NGC happens here and temporary data is removed after successful import of model into the annotation Server.
logs The annotation server logs are stored over here.
models All serving models in Frozen Format are stored here.

Server logs

Once the server is up and running, you can watch or pull the server logs in a browser through http://127.0.0.1:5000/logs?lines=100 to fetch recent 100 lines.

Use http://127.0.0.1:5000/logs?lines=-1 to fetch everything from the current log file.

Integrating into your own application

Once the annotation server is setup and started with specific organ models, client component of the SDK is delivered as an open source reference implementation to demonstrate the integration process. Client code and libraries are provided for both C++ and Python languages on NVIDIA Github page here: https://github.com/NVIDIA/ai-assisted-annotation-client

Loading models in the Annotation server

These admin utilities (through curl) are supported in the annotation server to manage annotation and segmentation models or via a browser interface at e.g. http://127.0.0.1:5000/docs.

Note: Run admin utilities from the docker-host machine addressing as 0.0.0.0 or 127.0.0.1 or localhost in the request URI.

Model configuration

The annotation server needs a configuration for each model which needs to be imported either from NGC or see section 4.3, Bring your own model, for information on using your own model.
  • NGC - this config is expected to be part of MMAR archive. It is in MMAR/configs/config-aiaa.json otherwise the annotation server will make an attempt to use default config for some pre-trained models.
  • User provided models - Upload this configuration along with the model (in CKPT/Frozen format) as part of multi-part/data format.

Segmentation model

Name Type Description Example
version string version of model 1.0
type string Either annotation or segmentation annotation
labels array[string] labels for indicating the purpose of the segmentation or annotation [spleen]
description string Long description of model 3D annotation model for spleen
threshold double Threshold of probability for inference 0.5
input_nodes dict Dictionary of input tensor names

{ "image": "NV_MODEL

_INPUT" }

output_nodes dict Dictionary of output tensor names

{"model":

"NV_MODEL

_OUTPUT"}

pre_transforms array[Transform] Array of Transforms are applied before inference

[ { "name": "tlt2.transforms.

VolumeTo4DArray", "args": {

"fields": List [ "image", "label" ] } } ]

post_transforms array[Transform] Array of Transforms are applied after inference

[ { "name": "tlt2.transforms.

SplitAcrossChannels",

"args":

{ "applied_key": "model",

"channel_names": List [

"background", "prediction" ] } } ]

channel_first boolean Channel First should be used for inference true
roi array[double] Cropped size [x,y,z] to be used in case of annotation models [128,128,128]

Annotation model

Name Type Description Example
sigma double Sigma value; reserved for future use 3.0
padding double Padding size used by clients to prepare the input for the annotation model 20.0

Transform

Name Type Description Example
name string Name of Transform. If you are using any transforms from TLT2 library/framwork, you can use "tlt2.transforms.xyz" or direct transforms name.

tlt2.transforms.

or

SplitAcrossChannels

args dict Dictionary which will be used as input to the transformer { "applied_key": "model", "channel_names": List [ "background", "prediction" ] }

Annotation_Spleen

Here's an example of model configuration for a spleen annotation model:

{
  "version": "1",
  "type": "annotation",
  "labels": [
    "spleen"
  ],
  "description": "A pre-trained model for volumetric (3D) segmentation of the spleen from CT image. It is trained using the runner-up awarded pipeline of the \"Medical Segmentation Decathlon Challenge 2018\" with 32 training images and 9 validation images.",
  "format": "CKPT",
  "threshold": 0.5,
  "roi": [
    128,
    128,
    128
  ],
  "sigma": 3.0,
  "padding": 20.0,
  "input_nodes": {
    "image": "NV_MODEL_INPUT"
  },
  "output_nodes": {
    "model": "NV_MODEL_OUTPUT"
  },
  "pre_transforms": [
    {
      "name": "VolumeTo4DArray",
      "args": {
        "fields": [
          "image",
          "label"
        ]
      }
    },
    {
      "name": "ScaleIntensityRange",
      "args": {
        "field": "image",
        "a_min": -1024,
        "a_max": 1024,
        "b_min": -1.0,
        "b_max": 1.0,
        "clip": true
      }
    },
    {
      "name": "AddExtremePointsChannel",
      "args": {
        "image_field": "image",
        "label_field": "label",
        "sigma": 3,
        "pert": 0
      }
    }
  ],
  "post_transforms": [
    {
      "name": "SplitAcrossChannels",
      "args": {
        "applied_key": "model",
        "channel_names": [
          "background",
          "prediction"
        ]
      }
    }
  ]
}

Pulling models from the NGC

Make sure you have access to the models and download them from NGC.

Start the annotation server

Run the annotation server included in the docker container with default options
docker run $NVIDIA_RUNTIME \-it --rm -p 5000:5000 \$DOCKER_IMAGE \start_aas.sh

Once the server is up you can start pulling existing segmentation or annotation models from NGC.

To use Method 1 you should VNC into the server to http://127.0.0.1. If you are in the docker command line you should use Method 2.

Method 1

Open in browser http://127.0.0.1:5000/docs/.

Try API "/admin/model/load" and provide Version and complete Model Path which typically includes {org}/{team}/{model}.

Method 2

curl -X PUT "http://0.0.0.0:5000/admin/model/annotation_ct_spleen" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d '{"path":"nvidia/med/annotation_ct_spleen","version":"1"}'

This assumes config_aiaa.json is present in MMAR package under ROOT/ or ROOT/config/

  1. If you are pulling MMAR model from NGC, config/config_aiaa.json is present
  2. Corresponding <model_name>.json is present in docker at /opt/nvidia/medical/tlt2/src/apps/aas/configs/models
Note:

If the required config_aiaa.json is missing in MMAR package, then you have to use the BYOM method described in the next section to push the new model to the annotation server.

Bring your own model

This release enables you to use your own models in the annotation server, if you do not want to use the models developed by Nvidia. Follow the instructions in this section to use your own models.

Train your model

  1. Train your own model using Tensorflow. For example you can use V-Net based segmentation models to train.
  2. Save your model using Tensorflow CheckPoint format (currently only the CheckPoint format is supported on the Annotation Server).
  3. If you have TensorRT version of the model, mode.trt.pb, you can provide it directly instead of in the CheckPoint format.

Prepare your model for deployment

  1. Prepare corresponding config-aiaa.json (ModelConfig) with respect to the annotation server. For more on ModelConfig refer: http://127.0.0.1:5000/#tag/Admin-(model)/paths/~1admin~1model~1{model}/put.
  2. Deploy required python libraries (to support own pre-transforms/post-transforms) into annotation server manually (through bash).
  3. Prepare a zip which includes the model check point and related files.

Deploy your model

  1. You can use CURL or API tryout interface through browser to upload model.zip and config-aiaa.json to the annotation server.
  2. The best approach is to "Try-Out" upload options through the annotation server in browser mode by visiting http://127.0.0.1:5000/docs/.
Here are three examples:
curl -X PUT "http://127.0.0.1:5000/admin/model/byom_segmentation_spleen" \
     -F "config=@config-aiaa.json;type=application/json" \
     -F "data=@model.zip"
# If you have MMAR archive
curl -X PUT "http://127.0.0.1:5000/admin/model/segmentation_ct_spleen" \
     -F "config=@config-aiaa.json;type=application/json"
     -F "data=@segmentation_ct_spleen.with_models.tgz"
 
# If you have MMAR archive (if model config is skipped, it will search config_aiaa.json inside Archive)
curl -X PUT "http://127.0.0.1:5000/admin/model/segmentation_ct_spleen" \
     -F "data=@segmentation_ct_spleen.with_models.tgz"

Refresh the model configuration

If you want to update the configuration without updating the model, you can do this by refreshing the configs and reloading an existing model with new configs.

Here's an example:

curl -X PATCH "http://127.0.0.1:5000/admin/model/byom_segmentation_spleen" \
     -H "Content-Type: application/json" \
     -d @config-aiaa.json

Delete model

Here's an example of deleting a model:

curl -X DELETE "http://127.0.0.1:5000/admin/model/byom_segmentation_spleen"

Verify that model is loaded in the server

The annotation server responds to models API to list all models currently loaded. In your shell, run this command:

curl http://$MACHINE_IP:5000/v1/models

You should see results similar to this:

[{"sigma": 3.0, "internal name": "annotation_spleen", "labels": ["spleen"], "type": "annotation", "description": "A pre-trained model for volumetric (3D) segmentation of the spleen from CT image. It is trained using the runner-up awarded pipeline of the Medical Segmentation Decathlon Challenge 2018 with 32 training images and 9 validation images.", "roi": [128, 128, 128], "padding": 20.0, "name": "annotation_spleen", "version": "1"}]

Training and deploying annotation models

Preparing data for segmentation

See Working with Classification and Segmentation Models in the Clara Train SDK: Transfer Learning Getting Started Guide for details on preparing data for use with transfer learning for classification tasks.

Next steps

For Training and Deploying new model using your own data, please see, Medical Model Archive in the Clara Train SDK: Transfer Learning Getting Started Guide.

  • Training the model: Run train.sh to train the model.
  • Exporting the model to a Tensorrt Optimized Model Inference: Run export.sh to export the model.

  • Running inference: Run infer.sh to run inference on the model.
  • Evaluating the model: Run evaluate.sh to evaluate the model a list of images based on a set of ground truth labels.

Appendix

Available annotation models

Here are the list of pre-trained annotation models available on NGC. Configuration files for using the models in the annotation server are inside MMAR archives on NGC.

Annotation models were trained with our AIAA 3D model (dextr3D) using a training approach similar to [1] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture either is derived from [2] and initialized with ImageNet pre-trained weights or using our model architecture developed for brain lesion segmentation [3]. During training, point clicks are "simulated" from the ground truth."

Note:

[1] Maninis, Kevis-Kokitsi, et al. "Deep extreme cut: From extreme points to object segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. https://arxiv.org/abs/1711.09081.

[2] Liu, Siqi, et al. "3d anisotropic hybrid network: Transferring convolutional features from 2d images to 3d anisotropic volumes." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. https://arxiv.org/abs/1711.08580.

[3] Myronenko, Andriy. "3D MRI brain tumor segmentation using autoencoder regularization." International MICCAI Brainlesion Workshop. Springer, Cham, 2018. https://arxiv.org/abs/1810.11654.

Annotation model Description
Brain tumor annotation  
  • annotation_mri_brain_tumors_t1ce_tc

A pre-trained model for volumetric (3D) annotation of brain tumors from T1c MRIs.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is detailed in [1]. During training point clicks are "simulated" from the ground truth labels.

The model was trained to segment the "tumor core" (TC) based on T1c (T1 contrast). It utilized an approach similar to what was described in "3D MRI brain tumor annotation using autoencoder regularization," a winning method in Multimodal Brain Tumor annotation Challenge (BraTS) 2018. The labelled dataset from BraTS 2018 was paritioned, based on our own split, into 256 training data and 29 validation data, as shown in config/anno_brats18_datalist_t1ce.json.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at https://www.med.upenn.edu/sbia/brats2018/data.html.

The provided training configuration required 16GB GPU memory.

Model Input Shape: 128 x 128 x 128

Training Script: train.sh

Training algorithm based on [1]; For details of model architecture, see [3].

Model input and output:

  • Input: 1 channel T1c MRIs scan images
  • Output: 2 channels for background & foreground

This Dice score on the validation data achieved by this model is 0.882.

  • annotation_mri_brain_tumors_t2_wt

A pre-trained model for volumetric (3D) annotation of brain whole tumors from T2 MRIs.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is detailed in [1]. During training point clicks are "simulated" from the ground truth labels.

This model resembles "annotation_mri_brain_tumors_t1ce_tc" model, except the input in another MRI channel is used as input (T2) and the output is the "whole tumor" region (WT). Both utilized an approach similar to what was described in "3D MRI brain tumor annotation using autoencoder regularization," a winning method in Multimodal Brain Tumor annotation Challenge (BraTS) 2018. The labelled dataset from BraTS 2018 was paritioned, based on our own split, into 256 training data and 29 validation data for this training task, as shown in config/anno_brats18_datalist_t2.json.

For more detailed description of tumor regions,please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at https://www.med.upenn.edu/sbia/brats2018/data.html.

The provided training configuration required 16GB GPU memory.

Model Input Shape: 128 x 128 x 128

Training Script: train.sh

Training algorithm based on [1]; For details of model architecture, see [3].

  • Model input and output:

  • Input: 1 channel T2 MRI
  • Output: 2 channels for background & foreground

This Dice score on the validation data achieved by this model is 0.893.

Liver annotation  
  • annotation_ct_liver

A pre-trained model for volumetric (3D) annotation of the liver in portal venous phase CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model was trained using the Liver dataset, as part of "Medical Segmentation Decathlon Challenge 2018". It consists of 131 labelled data and 70 unlabelled data. The labelled data was partitioned, based on our own split, into 104 training images and 27 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of "Medical Segmentation Decathlon Challenge 2018," see http://medicaldecathlon.com/.

The training dataset is Task03_Liver.tar from the link above. The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}
Note: to match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The provided training configuration required 16GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Model Input Shape: 128 x 128 x 128

Training Script: train.sh

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel CT image
  • Output: 2 channels for background & foreground

This Dice score on the validation data achieved by this model is 0.956.

Liver tumor annotation  
  • annotation_ct_liver_tumor

A pre-trained model for volumetric (3D) annotation of the liver tumor in portal venous phase CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model was trained with both the Liver and Hepatic Vessel dataset, as parts of "Medical Segmentation Decathlon Challenge 2018". Only data including labelled tumor were selected into our dataset. The dataset was then partitioned, based on our own split, into 254 training images and 83 validation images for this training task, as shown in config/anno_liver_tumor_dataset.json.

For more detailed description of "Medical Segmentation Decathlon Challenge 2018," please see http://medicaldecathlon.com/.

The training dataset is Task08_HepaticVessel.tar and Task03_Liver.tar from the link above.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}
Note: to match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The provided training configuration required 16GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Model Input Shape: 128 x 128 x 128

Training Script: train.sh

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel CT image
  • Output: 2 channels for background & foreground

This Dice score on the validation data achieved by this model is 0.788.

Hippocampus annotation  
  • annotation_mri_hippocampus

A pre-trained model for volumetric (3D) segmentation of the hippocampus head and body from mono-modal MRI image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model is trained with 208 training images and 52 validation images.

Training Data Source: Task04_Hippocampus.tar from http://medicaldecathlon.com/ The data was converted to resolution 1mm x 1mm x 1mm for training, using the following command:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

The training was performed with command train.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: 128 x 128 x 128

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel MRI image
  • Output 2 channels:
    • Label 1: hippocampus
    • Label 0: everything else.

This model achieve the following Dice score on the validation data (our own split from the training dataset): Hippocampus: 0.873 (mean_dice1: 0.886 mean_dice2: 0.865)

Lung tumor annotation  
  • annotation_ct_lung_tumor

A pre-trained model for volumetric (3D) annotation of the lung tumors from CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model is trained with 50 training image pairs and 13 validation images.

The training dataset is Task06_Lung.tar from http://medicaldecathlon.com/.

The data was converted to resolution 1mm x 1mm x 1mm for training.

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}
Note: To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The training was performed with command train.sh, which required 16GB-memory GPUs.

Training Graph Input Shape: 128 x 128 x128

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel CT image
  • Output: 2 channels for background & foreground

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Lung tumor: 0.79
Prostate annotation  
  • annotation_mri_prostate_cg_and_pz

A pre-trained model for volumetric (3D) segmentation of the prostate central gland and peripheral zone from the multimodal MR (T2, ADC).

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

Training Data Source: Task05_Prostate.tar from http://medicaldecathlon.com/ The data was converted to resolution 1mm x 1mm x 1mm for training using the following command:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}.

The training was performed with command train.sh, which required 16GB-memory GPUs.

Training Graph Input Shape: 128 x 128 x 128

Training algorithm based on [1]; For details of model architecture, see [2].

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Prostate central gland and peripheral zone: 0.74
Model input and output:
  • Input: 2 channel MRI image
  • Output 2 channels:
    • Label 1: prostate
    • Label 0: everything else

This model achieve the following Dice score on the validation data (our own split from the training dataset): Prostate: 0.743 (mean_dice1: 0.506 mean_dice2: 0.883)

Left atrium annotation  
  • annotation_mri_left_atrium

A pre-trained model for volumetric (3D) annotation of the left atrium from MRI image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model is trained with 16 training image pairs and 4 validation images.

The training dataset is Task02_Heart.tar from http://medicaldecathlon.com/.

The data was converted to resolution 1mm x 1mm x 1mm for training.

 tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}
Note: To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The training was performed with command train.sh, which required 16GB-memory GPUs.

Training Graph Input Shape: 128 x 128 x128

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel MRI image
  • Output: 2 channels for background & foreground

This model achieves the following Dice score on the validation data (our own split from the training dataset): Left Atrium: 0.92

Pancreas  
  • annotation_ct_pancreas

A pre-trained model for volumetric (3D)annotation of the colon from CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model was trained with the Pancreas dataset, as part of "Medical Segmentation Decathlon Challenge 2018". It consists of 281 labelled data and 139 unlabelled data. The labelled data were partitioned, based on our own split, into 224 training images and 57 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of "Medical Segmentation Decathlon Challenge 2018, see http://medicaldecathlon.com/.

The training dataset is Task07_Pancreas.tar from the link above.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}
Note: To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The provided training configuration required 16GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Model Input Shape: 128 x 128 x 128

Training Script: train.sh

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel CTimage
  • Output: 2 channels for background & foreground

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Pancreas (including tumor): 0.840
Colon tumor annotation  
  • annotation_ct_colon_tumor

A pre-trained model for volumetric (3D) annotation of the colon from CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model is trained with 100 training image pairs and 26 validation images.

The training dataset is Task10_Colon.tar from http://medicaldecathlon.com/.

The data was converted to resolution 1mm x 1mm x 1mm for training.

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}

Note: To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The training was performed with command train.sh, which required 16GB-memory GPUs.

Training Graph Input Shape: 28 x 128 x128

Model input and output:
  • Input: 1 channel CT image
  • Output: 2 channels for background & foreground

This model achieves the following Dice score on the validation data (our own split from the training dataset): colon: 0.68

Hepatic vessel annotation  
  • annotation_ct_hepatic_vessel

A pre-trained model for volumetric (3D) annotation of the hepatic vessel CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

This model was trained on the Hepatic vessel dataset, as part of "Medical Segmentation Decathlon Challenge 2018". It consists of 303 labelled data and 140 unlabelled data. The labelled data was partitioned, based on our own split, into 242 training images and 61 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of "Medical Segmentation Decathlon Challenge 2018," see http://medicaldecathlon.com/.

The training dataset is Task08_HepaticVessel.tar from the link above.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}
Note: To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The provided training configuration required 16GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Training Script: train.sh

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel CTimage
  • Output: 2 channels for background & foreground

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Hepatic vessel: 0.564
Spleen annotation  
  • annotation_ct_spleen

A pre-trained model for volumetric (3D) annotation of the spleen from CT image.

This model was trained using our AIAA 3D model (dextr3D) using a training approach similar to [6] where the user input (extreme point clicks) is modeled as 3D Gaussians in an additional input channel to the network. The network architecture is derived from [3] and initialized with ImageNet pre-trained weights. During training point clicks are "simulated" from the ground truth labels.

The training dataset is Task09_Spleen.tar from http://medicaldecathlon.com/.

The data was converted to 1mm resolution before training:

 tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}
Note: To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR's config folder.

The training was performed with command train.sh, which required 16GB-memory GPUs.

Training Graph Input Shape: 128 x 128 x128

Training algorithm based on [1]; For details of model architecture, see [2].

Model input and output:
  • Input: 1 channel CTimage
  • Output: 2 channels for background & foreground
  • Training algorithm based on [1]; For details of model architecture, see [2].

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Spleen: 0.96

[1] Myronenko, Andriy. "3D MRI brain tumor segmentation using autoencoder regularization." International MICCAI Brainlesion Workshop. Springer, Cham, 2018.https://arxiv.org/abs/1810.11654.

[2] Xia, Yingda, et al. "3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training." arXiv preprint arXiv:1811.12506 (2018).https://arxiv.org/abs/1811.12506.

[3] Liu, Siqi, et al. "3d anisotropic hybrid network: Transferring convolutional features from 2d images to 3d anisotropic volumes." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018.https://arxiv.org/abs/1711.08580.

[4] Huang, Gao, et al. "Densely connected convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.https://arxiv.org/abs/1608.06993.

[5] Wang, Xiaosong, et al. "Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.https://arxiv.org/abs/1705.02315.

[6] Maninis, Kevis-Kokitsi, et al. "Deep extreme cut: From extreme points to object segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.https://arxiv.org/abs/1711.09081.

Data transforms and augmentations

Here is a list of built-in data transformation functions. If you need additional transformation functions, please contact us at the TLT user forum: http://devtalk.nvidia.com.

Transforms Description

LoadNifty

Load NIfTI data. The value of each key (specified by fields) in input "dict" can be a string (a path to a single NIfTI file) or a list of strings (several paths to multiple NIfTI files, if there are several channels saved as separate files).

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

  • Returns:

    - Each field of "dict" is substituted by a 4D numpy array.

VolumeTo4dArray

Transforms the value of each key (specified by fields) in input "dict" from 3D to 4D numpy array by expanding one channel, if needed.

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

  • Returns:

    - Each field of "dict" is substituted by a 4D numpy array.

ScaleIntensityRange

Randomly shift the intensity level of the numpy array.

  • init_args:

    - field: string

    one key_value to apply, e.g. "image".

    - magnitude: float

    quantity scale of shift, has to be greater than zero.

ScaleIntensityOscillation

Randomly shift scale level for image.
  • Args:
    • field: key string, e.g. "image".
    • magnitude: quantity of scale shift, has to be greater than zero.
  • Returns
    • Data with an offset on intensity scale.

data with an offset on intensity scale.

CropSubVolumePosNegRatioCtrlEpoch

Randomly crop the foreground and background ROIs from both the image and mask for training. The sampling ratio between positive and negative samples is adjusted with the epoch number.

  • init_args:

    - image_field: string

    one key_value to apply, e.g. "image".

    - label_field: string

    one key_value to apply, e.g. "label".

    - size: list of ints

    cropped ROI size, e.g., [96, 96, 96].

    - ratio_start: float

    positive/negative ratio when training start.

    - ratio_end: float

    positive/negative ratio when training end.

    - ratio_step: float

    changing of positive/negative ratio after each step.

    - num_epochs: int

    epochs of one step.

  • Returns:

    - Updated dictionary with cropped ROI image and mask

transforms_fastaug.

TransformVolumeCropROIFast

PosNegRatio

Fast 3D data augmentation method (CPU based) by combining 3D morphological transforms (rotation, elastic deformation, and scaling) and ROI cropping. The sampling ratio is specified by pos/neg.

  • init_args:

    - applied_keys: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - size: list of int

    cropped ROI size, e.g., [96, 96, 96].

    - deform: boolean

    whether to apply 3D deformation.

    - rotation: boolean

    whether to apply 3D rotation.

    - rotation_degree: float

    the degree of rotation, e.g., 15 means randomly rotate the image/label in a range [-15, +15].

    - scale: boolean

    whether to apply 3D scaling.

    - scale_factor: float

    the percentage of scaling, e.g., 0.1 means randomly scaling the image/label in a range [-0.1, +0.1].

    - pos: float

    the factor controlling the ratio of positive ROI sampling.

    - neg: float

    the factor controlling the ratio of negative ROI sampling.

  • Returns:

    - Updated dictionary with cropped ROI image and mask after data augmentation.

AdjustContrast

Randomly adjust the contrast of the field in input "dict".

  • init_args:

    - field: string

    one key_value to apply, e.g. "image".

AddGaussianNoise

Randomly add Gaussian noise to the field in input "dict".

  • init_args:

    - field: string

    one key_value to apply, e.g. "image".

LoadPng

Load png image and the label. The value of "image" must be a string (a path to a single png file) while the value of the "label" must be a list of labels.

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

  • Returns:

    - "image" of "dict" is substituted by a 3D numpy array while the "label" of "dict" is substituted by a numpy list

CropRandomSubImageInRange

Randomly crop 2D image. The crop size is randomly selected between lower_size and image size.

  • init_args:

    - lower_size: int or float

    lower limit of crop size, if float, then must be fraction <1

    - max_displacement: float

    max displacement from center to crop

    - keep_aspect: boolean

    if true, then original aspect ratio is kept

  • Returns:

    - The "image" field of input "dict" is substituted by cropped ROI image.

NPResizeImage

Resize the 2D numpy array (channel x rows x height) as an image.

  • init_args:

    - applied_keys: string

    one key_value to apply, e.g. "image".

    - output_shape: list of int with length 2

    e.g., [256,256].

    - data_format: string

    ''channels_first', 'channels_last', or 'grayscale'.

NP2DRotate

Rotate 2D numpy array, or channelled 2D array. If random is set to true, then rotate within the range of [ -angle, angle]

  • init_args:

    - applied_keys: string

    one key_value to apply, e.g. "image".

    - angle: float

    e.g. 7.

    - random: boolean

    default is false.

NPExpandDims

Add a singleton dimension to the selected axis of the numpy array.

  • init_args:

    - applied_keys: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - expand_axis: int

    axis to expand, default is 0

NPRepChannels

Repeat a numpy array along specified axis, e.g., turn a grayscale image into a 3-channel image.

  • init_args:

    - applied_keys: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - channel_axis: int

    the axis along which to repeat values.

    - repeat: int

    the number of repetitions for each element.

CenterData

Center numpy array's value by subtracting a subtrahend and dividing by a divisor.

  • init_args:

    - applied_keys: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - subtrahend: float

    subtrahend. If None, it is computed as the mean of dict[key_value].

    - divisor: float

    divisor. If None, it is computed as the std of dict[key_value]

NPRandomFlip3D

Flip the 3D numpy array along random axes with the provided probability.

  • init_args:

    - applied_keys: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - probability: float

    probability to apply the flip, value between 0 and 1.0.

NPRandomZoom3D

Apply a random zooming to the 3D numpy array.

  • init_args:

    - applied_keys: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - lower_limits: list of float

    lower limit of the zoom along each dimension.

    - upper_limits: list of float

    upper limit of the zoom along each dimension.

    - data_format: string

    'channels_first' or "channels_last".

    - use_gpu: boolean

    whether to use cupy for GPU acceleration. Default is false.

    - keep_size: boolean

    default is false which means this function will change the size of the data array after zooming. Setting keep_size to True will result in an output of the same size as the input.

CropForegroundObject

Crop the 4D numpy array and resize. The numpy array must have foreground voxels.

  • init_args:

    - size: list of int

    resized size.

    - image_field: string

    "image".

    - label_field: string

    "label".

  • pad: int number of voxels for adding a margin around the object
  • foreground_only: boolean whether to treat all foreground labels as one binary label (default) or whether to select foreground label at random.
  • keep_classes : boolean; if true, keep original label indices in label image (no thresholding), useful for multi-class tasks.
  • pert: int; random perturbation in each dimension added to padding (in voxels).

NPRandomRot90_XY

Rotate the 4D numpy array along random axes on XY plane (axis = (1, 2)).

  • init_args:

    - applied_keys: string

    one key_value to apply, e.g. "image".

    - probability: float

    probability to utilize the transform, between 0 and 1.0.

AddExtremePointsChannel

Add and additional channel to 4D numpy array where the extreme points of the foreground labels are modeled as Gaussians.

  • init_args:

    - image_field: string

    "image".

    - label_field: string

    "label".

    - sigma: float

    size of Gaussian.

    - pert: boolean

    random perturbation added to the extreme points.

NormalizeNonzeroIntensities

Normalize 4D numpy array to zero mean and unit std, based on non-zero elements only for each input channel individually.

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

SplitAcrossChannels

Splits the 4D numpy array across channels to create new dict entries. New key_values shall be applied_key+channel number, e.g. "image1".

  • init_args:

    - applied_key: string

    one key_value to apply, e.g. "image".

LoadResolutionFromNifty

Get the image resolution from an NifTI image

  • init_args:

- applied_key: string

one key_value to apply, e.g. "image".

  • Returns:

    - "dict" has a new key-value pair: dict[applied_key+"_resolution"]: resolution of the NIfTI image

Load3DShapeFromNumpy

Get the image shape from an NifTI image

  • init_args:

- applied_key: string

one key_value to apply, e.g. "image".

  • Returns:

    - "dict" has a new key-value pair: dict[applied_key+"_shape"]: shape of the NIfTI image

ResampleVolume

Resample the 4D numpy array from current resolution to a specific resolution

  • init_args:

- applied_key: string

one key_value to apply, e.g. "image".

- resolution: list of float

input image resolution.

- target_resolution: list of float

target resolution.

BratsConvertLabels

Brats data specific. Convert input labels format (indices 1,2,4) into proper format.

  • init_args:

- fields: string or list of strings

key_values to apply, e.g. ["image", "label"].

CropSubVolumeRandomWithinBounds

Crops a random subvolume from within the bounds of 4D numpy array.

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - size: list of int

    the size of the crop region e.g. [224,224,128].

FlipAxisRandom

Flip the numpy array along its dimensions randomly.

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - axis : list of ints

    which axes to attempt to flip (e.g. [0,1,2] - for all 3 dimensions) - the axis indices must be provided only for spatial dimensions.

CropSubVolumeCenter

Crops a center subvolume from within the bounds of 4D numpy array.

  • init_args:

    - fields: string or list of strings

    key_values to apply, e.g. ["image", "label"].

    - size: list of int

    the size of the crop region e.g. [224,224,128] (similar to CropSubVolume

    RandomWithinBounds,

    but crops the center)

Client C++ API

AIAA services invoked by the AIAAClient

The AIAA object is constructed with the IP address, port, and version of the AIAA server

client = client_api.AIAAClient(ip, port, api_version)

AIAA Client provides these methods

  1. models(): supported model list;
  2. dextr3d(): 3D image segmentation using DEXTR3D method;
  3. segmentation(): 3D image segmentation using classical segmentation model;
  4. mask2PolygonConversion(): 3D binary mask to polygon representation conversion
  5. fixPolygon(): 2D polygon update with single point edit.

Here's an example using the AIAA Client APIs

#include <nvidia/aiaa/client.h>

// Create AIAA Client Object
nvidia::aiaa::Client client("http://10.110.45.66:5000/v1");

try {

 // models():: Get all the models supported by AIAA Server
 nvidia::aiaa::ModelList modelList = client.models();
 std::string label = "spleen";

 // Get Matching Model for a given label
 nvidia::aiaa::Model model = modelList.getMatchingModel(label);
 std::cout << model << std::endl;
 std::string inputImageFile = "image.nii.gz";
 std::string outputDextra3dImageFile = "result.nii.gz";
 nvidia::aiaa::Point3DSet pointSet = nvidia::aiaa::Point3DSet::fromJson("[[1,2,3]"]);

 // dextra3d():: Call dextra3D API for a given model, pointSet
 int ret1 = client.dextr3d(model, pointSet, inputImageFile, outputDextra3dImageFile);

 // dextra3d():: Call dextra3D API for a given label, pointSet, PAD and ROI
 double PAD = 20;
 std::string ROI_SIZE = "128x128x128";
 double SIGMA = 3;
 int ret2 = client.dextr3d(label, pointSet, inputImageFile, outputDextra3dImageFile, PAD, ROI_SIZE, SIGMA);


 // mask2Polygon():: Call mask2Polygon API to get slice-wise polygon set
 int pointRatio = 10;
 nvidia::aiaa::PolygonsList result = client.mask2Polygon(pointRatio, outputDextra3dImageFile);
 std::cout << result << std::endl;
 // [[], [[[169,66],[163,74],[[[169,66],[163,74],[175,66]]]]


 int neighborhoodSize = 10;

 nvidia::aiaa::Polygons p1 = nvidia::aiaa::Polygons::fromJson("[[10,20],[20,30]"
 nvidia::aiaa::Polygons p2 = nvidia::aiaa::Polygons::fromJson("[[10,20],[21,31]]");
 int polygonIndex = 0;
 int vertexIndex = 1;
 std::string input2DImageFile = "image_slice_2D.png";
 std::string output2DImageFile = "updated_slice_2D.png";

 // fixPolygon():: Call Fix Polygon with new Polygon Points
 nvidia::aiaa::Polygons result = client.fixPolygon(p1, p2, neighborhoodSize, polygonIndex, vertexIndex, input2DImageFile, output2DImageFile);
 std::cout << result << std::endl;
 // [[11,21],[21,31]]

}
catch (aiaa::nvidia::exception &e) {
 std::cerr << "nvidia::aiaa::exception => nvidia.aiaa.error." << e.id << "; description: " << e.name() << std::endl;
}

nvidiaAIAAListModels:

nvidiaAIAAListModels: provides implementation for nvidia::aiaa::Client::model()

These options are available:

Option Description Default Example
-h Prints the help information    
-server Server URI for the annotation server   -server

http://10.110.45.66:

5000/v1
-label Label Name for matching   -label liver
-model Model Name   -

modelannotation_

liver
-type Filter model by annotation/segmentation   -type annotation
-output Save output result into a file   -output models.json

Here's an example:

bin/nvidiaAIAAListModels \
                     -server http://10.110.45.66:5000/v1 \
                     -label spleen

nvidiaAIAADEXTR3D:

nvidiaAIAADEXTR3D: provides implementation for nvidia::aiaa::Client::dextra3d()

These options are available:

Option Description Default Example
-h Prints the help information    
-server Server URI for AIAA Server  

-server

http://10.110.45.66:

5000/v
-label Label Name for matching; Either -model or -label is required   -label liver
-model Model name   -model Dextr3DLiver
-points JSON Array of 3D Points (Image Indices) in [[x,y,z]+] format   -points [[70,172,86],...,[105,161,180]]
-image Input image filename where image is stored in 3D format   -image image.nii.gz
-output File name to store 3D binary mask image result from AIAA server   -output result.nii.gz
-pad Padding size for input Image 20 -pad 20
-roi ROI Image size in XxYxZ format which is used while training the AIAA Model 128x128x128 -roi 96x96x96
-sigma Sigma value for the annotation server 3 -sigma 3

Here's an example using the label:

bin/nvidiaAIAADEXTR3D \
                     -server http://10.110.45.66:5000/v1 \
                     -label spleen \
                     -points `cat ../test/data/pointset.json` \
                     -image _image.nii.gz \
                     -output tmp_out.nii.gz \
                     -pad 20 \
                     -roi 128x128x128 \
                     -sigma 3

Here's an example using the model:

bin/nvidiaAIAADEXTR3D \
                     -server http://10.110.45.66:5000/v1 \
                     -model annotation_spleen \
                     -points `cat ../test/data/pointset.json` \
                     -image _image.nii.gz \
                     -output tmp_out.nii.gz \
                     -pad 20 \
                     -roi 128x128x128 \
                     -sigma 3

nvidiaAIAASegmentation:

nvidiaAIAASegmentation: provides implementation for nvidia::aiaa::Client::segmentation()

These options are available:

Option Description Default Example
-h Prints the help information    
-server Server URI for the annotation server  

-server

http://10.110.45.66:

5000/v
-label Label Name for matching; Either -model or -label is required   -label liver
-model Model Name   -model SegmentationLiver
-image Input image filename where image is stored in 3D format   -image image.nii.gz
-output File name to store 3D binary mask image result from the annotation server   -output result.nii.gz

Here's are some examples:

bin/nvidiaAIAASegmentation \
                    -server http://10.110.45.66:5000/v1 \
                    -label spleen \
                    -image _image.nii.gz \
                    -output tmp_out.nii.gz

bin/nvidiaAIAASegmentation \
                    -server http://10.110.45.66:5000/v1 \
                    -model segmentation_spleen \
                    -image _image.nii.gz \
                    -output tmp_out.nii.gz

nvidiaAIAAMaskPolygon:

nvidiaAIAAMaskPolygon: provides implementation for nvidia::aiaa::Client::model().

These options are available:

Option Description Default Example
-h Prints the help information    
-server Server URI for the annotation server  

-server

http://10.110.45.66:

5000/v
-ratio Point Ratio 10 -ratio 10
-input Input 3D binary mask image file name (which is an output of dextra3d)   -input tmp_out.nii.gz
-output Save output result (JSON Array) representing the list of polygons for each slice into a file   -output polygonlist.json

Here's an example:

bin/nvidiaAIAAMaskPolygon \
                     -server http://10.110.45.66:5000/v1 \
                     -image tmp_out.nii.gz \
                     -output polygonlist.json

nvidiaAIAAFixPolygon:

nvidiaAIAAFixPolygon: provides implementation for nvidia::aiaa::Client::mask2Polygon()

These options are available

Option Description Default Example
-h Prints the help information    
-server Server URI for AIAA Server  

-server

http://10.110.45.66:

5000/v
-neighbor Neighborhood size for propagation 10 -neighbor 10
-poly New 2D Polygon Array in [[[x,y]+]] format   -poly [[[54,162],…,[62,140]]]
-ppoly Current or Old 2D Polygon Array in [[[x,y]+]] format   -poly [[[53,162],…,[62,140]]]
-pindex Polygon Index within new Polygon Array which needs to be updated   -pindex 0
-vindex Vertical Index within new Polygon Array which needs to be updated   -vindex 17
-image Input 2D image slice   -image image_slice_2D.png
-output Output file name to the updated image   -output updated_image_

2D.png

Here's an example:

bin/nvidiaAIAAFixPolygon \
                     -server http://10.110.45.66:5000/v1 \
                     -neighbor 10 \
                     -poly `cat ../test/data/polygons.json` \
                     -ppoly `cat ../test/data/polygons.json` \
                     -pindex 0 \
                     -vindex 17 \
                     -image ../test/data/image_slice_2D.png \
                     -output updated_image_2D.png

Client Python API

AIAA services invoked by the AIAAClient object

The AIAA object is constructed with the IP address, port, and version of the AIAA server

client = client_api.AIAAClient(ip, port, api_version)

AIAA Client provides these methods:

  1. model_list: supported model list;
  2. dextr3d: 3D image annotation using DEXTR3D method;
  3. segmentation: 3D image segmentation using traditional segmentation models;
  4. mask2polygon: 3D binary mask to polygon representation conversion;
  5. fixpolygon: 2D polygon update with single point edit.

Details of the four methods:

  1. model_list
    Input:
    1. label: label to filter the model selection; If not provided will list all the models
    2. type: type of model; It can be either segmentation or annotation; If not given then both types are selected
    Output:
    1. A json string containing current supported object names along with their corresponding object model names for DEXTR3D or Segmentation
  2. dextr3d
    Input:
    1. object model name, according to the output of GetModelList()
    2. temporary folder path, needed for http request/response
    3. point set: a json string containing the extreme points' indices
    4. input 3D image file name
    5. output 3D binary mask image file name
    6. optional padding size in mm (default is 20)
    7. optional ROI Image size (default is '128x128x128')
    8. optional sigma param for inference
    Output
    1. output 3D binary mask will be saved to the specified file
  3. segmentation
    Input:
    1. object model name, according to the output of GetModelList()
    2. temporary folder path, needed for http request/response
    3. point set: a json string containing the extreme points' indices
    4. input 3D image file name
    5. output 3D binary mask image file name
    6. optional padding size in mm (default is 20)
    7. optional ROI Image size (default is '128x128x128')
    8. optional sigma param for inference
    Output
    1. set of extreme polygon points for the segmentation mask
    2. output 3D binary mask will be saved to the specified file
  4. mask2polygon
    Input:
    1. point ratio controlling how many polygon vertices will be generated
    2. input 3D binary mask image file name
    Output:
    1. A json string containing the indices of all polygon vertices slice by slice.
  5. fixpolygon
    Input:
    1. A json string containing parameters of polygon editing:
      1. Neighborhood size
      2. Index of the changed polygon
      3. Index of the changed vertex
      4. Polygon before editing
      5. Polygon after editing
    2. input 2D image file name
    3. output 2D mask image file name
    Output:
    1. A json string the indices of updated polygon vertices
    2. output binary mask will be saved to the specified name

Examples:

test_aiaa_server.py gives method to test the API under configurations specified by aas_tests.json:
  1. server information: IP, port, version
  2. test-specific information:
    1. test name
    2. disable flag for running / skipping a particular test
    3. api name for selecting different methods
    4. test-dependent parameters: input/output file path, other parameters

Notices

Notice

THE INFORMATION IN THIS GUIDE AND ALL OTHER INFORMATION CONTAINED IN NVIDIA DOCUMENTATION REFERENCED IN THIS GUIDE IS PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE INFORMATION FOR THE PRODUCT, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the product described in this guide shall be limited in accordance with the NVIDIA terms and conditions of sale for the product.

THE NVIDIA PRODUCT DESCRIBED IN THIS GUIDE IS NOT FAULT TOLERANT AND IS NOT DESIGNED, MANUFACTURED OR INTENDED FOR USE IN CONNECTION WITH THE DESIGN, CONSTRUCTION, MAINTENANCE, AND/OR OPERATION OF ANY SYSTEM WHERE THE USE OR A FAILURE OF SUCH SYSTEM COULD RESULT IN A SITUATION THAT THREATENS THE SAFETY OF HUMAN LIFE OR SEVERE PHYSICAL HARM OR PROPERTY DAMAGE (INCLUDING, FOR EXAMPLE, USE IN CONNECTION WITH ANY NUCLEAR, AVIONICS, LIFE SUPPORT OR OTHER LIFE CRITICAL APPLICATION). NVIDIA EXPRESSLY DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY OF FITNESS FOR SUCH HIGH RISK USES. NVIDIA SHALL NOT BE LIABLE TO CUSTOMER OR ANY THIRD PARTY, IN WHOLE OR IN PART, FOR ANY CLAIMS OR DAMAGES ARISING FROM SUCH HIGH RISK USES.

NVIDIA makes no representation or warranty that the product described in this guide will be suitable for any specified use without further testing or modification. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to ensure the product is suitable and fit for the application planned by customer and to do the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this guide. NVIDIA does not accept any liability related to any default, damage, costs or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this guide, or (ii) customer product designs.

Other than the right for customer to use the information in this guide with the product, no other license, either expressed or implied, is hereby granted by NVIDIA under this guide. Reproduction of information in this guide is permissible only if reproduction is approved by NVIDIA in writing, is reproduced without alteration, and is accompanied by all associated conditions, limitations, and notices.

Trademarks

NVIDIA, the NVIDIA logo, and cuBLAS, CUDA, cuDNN, cuFFT, cuSPARSE, DIGITS, DGX, DGX-1, DGX Station, GRID, Jetson, Kepler, NVIDIA GPU Cloud, Maxwell, NCCL, NVLink, Pascal, Tegra, TensorRT, Tesla and Volta are trademarks and/or registered trademarks of NVIDIA Corporation in the Unites States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.