Model Details

The following information of all models is available at NVIDIA NGC Accelerated Software Models - Medical Imaging.

Segmentation models

Here is a list of the segmentation models. All the models are trained using 1x1x1mm resolution data.

Brain tumor segmentation

segmentation_mri_brain_tumors_br16_full

A pre-trained model for volumetric (3D) segmentation of brain tumors from multi-modal MRIs based on BraTS 2018 data.

https://www.med.upenn.edu/sbia/brats2018/data.html

The model is trained to segment 3 nested subregions of primary (gliomas) brain tumors: the “enhancing tumor” (ET), the “tumor core” (TC), and the “whole tumor” (WT), based on 4 input MRI scans ( T1c, T1, T2, FLAIR). The ET is described by areas that show hyper-intensity in T1c when compared to T1, but also when compared to “healthy” white matter in T1c. The TC describes the bulk of the tumor, which is what is typically resected. The TC encompasses the ET, as well as the necrotic (fluid-filled) and the non-enhancing (solid) parts of the tumor. The WT describes the complete extent of the disease, as it entails the TC and the peritumoral edema (ED), which is typically depicted by hyper-intense signal in FLAIR.

The dataset is available at “Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018.” The provided labelled data was partitioned, based our own split, into training (243 studies) and validation (42 studies) datasets.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at: https://www.med.upenn.edu/sbia/brats2018/data.html.

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1].

The provided training configuration required 16GB GPU memory.

Model Input Shape: 224 x 224 x 128

Training Script: train.sh

Model input and output:

  • Input: 4 channel 3D MRIs (T1c, T1, T2, FLAIR)

  • Output: 3 channels of tumor subregion 3D masks

The model was trained with 285 cases with our own split, as shown in the datalist json file in the config folder.

  • Tumor core (TC): 0.8624

  • Whole tumor (WT): 0.9020

  • Enhancing tumor (ET): 0.7770

segmentation_mri_brain_tumors_br16_t1c2tc

A pre-trained model for volumetric (3D) brain tumor segmentation (only TC from T1c images). The model is trained to segment “tumor core” (TC) based on 1 input MRI scan (T1c).

The dataset is available at “Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018.” The provided labelled data was partitioned, based our own split, into training (243 studies) and validation (42 studies) datasets, as shown in config/seg_brats18_datalist_t1c.json.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at:

https://www.med.upenn.edu/sbia/brats2018/data.html

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1].

The provided training configuration required 16GB GPU memory.

Model Input Shape: 224 x 224 x 128

Training Script: train.sh

Model input and output:

  • Input: 1 channel 3D MRI (T1c)

  • Output: 1 channel of tumor core 3D masks

The achieved mean Dice score on the validation data is: Tumor core (TC): 0.839

Liver and Tumor segmentation

segmentation_ct_liver_and_tumor

A pre-trained model for volumetric (3D) segmentation of the liver and lesion in portal venous phase CT image.

This model is trained using the runnerup [2] awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” using the AHnet architecture [3].

This model was trained with Liver dataset, as part of “Medical Segmentation Decathlon Challenge 2018”. It consists of 131 labelled data and 70 unlabelled data. The labelled data was partitioned, based on our own split, into 104 training images and 27 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of “Medical Segmentation Decathlon Challenge 2018,” at:

http://medicaldecathlon.com/.

The training dataset is Task03_Liver.tar from the link above.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}

Note: To match the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR’s config folder.

The provided training configuration required 12GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Model input shape: dynamic

Training Script: train.sh

Model input and output:

Input: 1 channel CT image

Output: 3 channels:

  • Label 1: liver

  • Label 2: tumor

  • Label 0: everything else

This Dice scores on the validation data achieved by this model are:

  • Liver: 0.932

  • Tumor: 0.495

Hippocampus segmentation

segmentation_mri_hippocampus

A pre-trained model for volumetric (3D) segmentation of the hippocampus head and body from mono-modal MRI image.

This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 208 training images and 52 validation images.

Training Data Source: Task04_Hippocampus.tar from http://medicaldecathlon.com/

The data was converted to resolution 1mm x 1mm x 1mm for training, using the following command:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

The training was performed with command train.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

Model input and output:

Input: 1 channel MRI image Output: 2 channels:

  • Label 1: hippocampus

  • Label 0: everything else

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Hippocampus: 0.872
    • mean_dice1: 0.882

    • mean_dice_dice2: 0.862

Lung Tumor segmentation

segmentation_ct_lung_tumor

A pre-trained model for volumetric (3D) segmentation of the lung tumor from CT image. This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 50 training images and 13 validation images.

Training Data Source:

Task06_Lung.tar from http://medicaldecathlon.com/

The data was converted to resolution 1mm x 1mm x 1mm for training, using the following command:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

The training was performed with command train_2gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

Model input and output:

Input: 1 channel CT image Output: 2 channels:

  • Label 1: lung tumor

  • Label 0: everything else

This Dice scores on the validation data (our own split) achieved by this model are:

  • lung: 0.417

Prostrate segmentation

segmentation_mri_prostate_cg_and_pz

A pre-trained model for volumetric (3D) segmentation of the prostate central gland and peripheral zone from the multimodal MR (T2, ADC). This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 25 training image pairs and 7 validation images.

Training Data Source: Task05_Prostate.tar from http://medicaldecathlon.com/. The data was converted to resolution 1mm x 1mm x 1mm for training, using the following command:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

The training was performed with command train_4gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 32

Model input and output:

Input: 2 channel MRI image Output: 2 channels:

  • Label 1: prostate peripheral zone

  • Label 0: everything else

This model achieve the following Dice score on the validation data (our own split from the training dataset):

Prostate: 0.724 (mean_dice1: 0.485 mean_dice2: 0.871)

Left atrium segmentation

segmentation_mri_left_atrium

A pre-trained model for volumetric (3D) segmentation of the left atrium from MRI image.

This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 16 training images and 4 validation images.

Training Data Source: Task02_Heart.tar from http://medicaldecathlon.com/ The data was converted to resolution 1mm x 1mm x 1mm for training, using the following command:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

The training was performed with command train_2gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

Model input and output:

Input: 1 channel MRI image Output: 2 channels:

  • Label 1: heart

  • Label 0: everything else

This Dice scores on the validation data (our own split) achieved by this model are:

heart: 0.9158

Pancreas and tumor segmentation

segmentation_ct_pancreas_and_tumor

A pre-trained model for volumetric (3D) segmentation of the pancreas and tumor from portal venous phase CT.

This model is trained using the runnerup [2] awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” using the AHnet architecture [3].

This model is trained with Pancreas dataset, as part of “Medical Segmentation Decathlon Challenge 2018”. It consists of 281 labelled data and 139 non-labelled data. The labelled data was partitioned, based on our own split, into 224 training images and 57 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of “Medical Segmentation Decathlon Challenge 2018,” see http://medicaldecathlon.com/.

The training dataset is Task07_Pancreas.tar from the link above.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}

Note

To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR’s config folder.

The provided training configuration required 12GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Model Input Shape: dynamic

Training Script: train.sh

The training dataset is Task07_Pancreas.tar from the link above. The data must be converted to 1mm resolution before training:

Data Conversion: convert to resolution 1mm x 1mm x 1mm

Model input shape: dynamic

Model input and output:

  • Input: 1 channel CT image

Output: 3 channels:
  • Label 1: pancreas

  • Label 2: tumor

  • Label 0: everything else

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Pancreas: 0.739

  • Tumor: 0.348

Colon tumor segmentation

segmentation_ct_colon_tumor

A pre-trained model for volumetric (3D) segmentation of the Colon from CT image.

This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 100 training images and 26 validation images.

Training Data Source: Task10_Colon.tar from http://medicaldecathlon.com/ The data was converted to resolution 1mm x 1mm x 1mm for training, using the following command.

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

The training was performed with command train_2gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

Training Graph Input Shape: dynamic

Model input and output:

Input: 1 channel CT image Output: 2 channels:

  • Label 1: colon tumor

  • Label 0: everything else

This Dice scores on the validation data (our own split) achieved by this model are:
  • colon cancer: 0.367

Hepatic vessel and tumor segmentation

segmentation_ct_hepatic_vessel_and_tumor

A pre-trained model for volumetric (3D) segmentation of the hepatic vessel and tumor from CT image.

This model is trained using the runnerup [2] awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” using the AHnet architecture [3].

This model was trained with Hepatic Vessel dataset, as part of “Medical Segmentation Decathlon Challenge 2018”. It consists of 303 labelled data and 140 non-labelled data. The labelled data was partitioned, based on our own split, into 242 training images and 61 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of “Medical Segmentation Decathlon Challenge 2018,” please see http://medicaldecathlon.com/.

The training dataset is Task08_HepaticVessel.tar from the link above.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}

Note

to match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR’s config folder.

The provided training configuration required 12GB GPU memory.

Data Conversion: convert to resolution 1mm x 1mm x 1mm.

Model Input Shape: dynamic

Training Script: train.sh

Model input and output:

Input: 1 channel CT image Output: 3 channels:

  • Label 1: hepatic vessel

  • Label 2: liver tumor

  • Label 0: everything else

This Dice scores on the validation data achieved by this model are:

  • Hepatic vessel: 0.523

  • Liver tumor: 0.422

Spleen segmentation

segmentation_ct_spleen

A pre-trained model for volumetric (3D) segmentation of the spleen from CT image.

This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 32 training images and 9 validation images.

The training dataset is Task09_Spleen.tar from http://medicaldecathlon.com/.

The data must be converted to 1mm resolution before training:

tlt-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

Note

To match up with the default setting, we suggest that ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR’s config folder.

The training was performed with command train_2gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

Model input and output:

Input: 1 channel CT image Output: 2 channels:

  • Label 1: spleen

  • Label 0: everything else

This model achieves the following Dice score on the validation data (our own split from the training dataset):
  • Spleen: 0.951

For details of model architecture, see [1]” (Liu et al.)

[1] Myronenko, Andriy. “3D MRI brain tumor segmentation using autoencoder regularization.” International MICCAI Brainlesion Workshop. Springer, Cham, 2018. https://arxiv.org/abs/1810.11654.[2] Xia, Yingda, et al. “3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training.” arXiv preprint arXiv:1811.12506 (2018). https://arxiv.org/abs/1811.12506.[3] Liu, Siqi, et al. “3d anisotropic hybrid network: Transferring convolutional features from 2d images to 3d anisotropic volumes.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. https://arxiv.org/abs/1711.08580.

Classification models

Chest X-ray Classification

classification_chestxray

A pre-trained densenet121 model for disease pattern detection in chest x-rays.

This model is trained using PLCO training data and evaluated on the PLCO validation data.

You can apply for access to the dataset at: https://biometry.nci.nih.gov/cdas/learn/plco/images/

The provided training configuration required 12GB-memory GPUs. The training was performed with command train.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: 256 x 256

Input: 16-bit CXR png

Output: 15 binary labels, each bit is corresponding to the prediction of ‘Nodule’, ‘Mass’, ‘Distortion of Pulmonary Architecture’, ‘Pleural Based Mass’, ‘Granuloma’, ‘Fluid in Pleural Space’, ‘Right Hilar Abnormality’, ‘Left Hilar Abnormality’, ‘Major Atelectasis’, ‘Infiltrate’, ‘Scarring’, ‘Pleural Fibrosis’, ‘Bone/Soft Tissue Lesion’, ‘Cardiac Abnormality’, ‘COPD’

Please refer to “medical/segmentation/examples/brats/tutorial_brats.ipynb” inside the docker and the files in the same folder for details.

This model achieves the following AUC score on the validation data:

Averaged AUC over all disease categories: 0.8680

Annotation Models

For all of our annotation models, please check NVIDIA NGC Model Catalog.