Model details

The following information of all models is available at NVIDIA NGC Accelerated Software Models - Medical Imaging.

Note the above link contains the most updated information and models. The following information is accurate at time of current release, and may change afterward.

Note

The models released have been trained both with and without automatic mixed precision (AMP). Please find the version you are interested in below.

Segmentation models

Here is a list of the segmentation models. All the models are trained using 1x1x1mm resolution data.

Brain tumor segmentation

clara_mri_seg_brain_tumors_br16_full_amp

A pre-trained model for volumetric (3D) segmentation of brain tumors from multi-modal MRIs based on BraTS 2018 data.

https://www.med.upenn.edu/sbia/brats2018/data.html

The model is trained to segment 3 nested subregions of primary (gliomas) brain tumors: the “enhancing tumor” (ET), the “tumor core” (TC), and the “whole tumor” (WT), based on 4 input MRI scans ( T1c, T1, T2, FLAIR). The ET is described by areas that show hyper-intensity in T1c when compared to T1, but also when compared to “healthy” white matter in T1c. The TC describes the bulk of the tumor, which is what is typically resected. The TC encompasses the ET, as well as the necrotic (fluid-filled) and the non-enhancing (solid) parts of the tumor. The WT describes the complete extent of the disease, as it entails the TC and the peritumoral edema (ED), which is typically depicted by hyper-intense signal in FLAIR.

The dataset is available at “Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018.” The provided labelled data was partitioned, based our own split, into training (243 studies) and validation (42 studies) datasets.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at: https://www.med.upenn.edu/sbia/brats2018/data.html.

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1].

The provided training configuration required 16GB GPU memory.

Model Input Shape: 224 x 224 x 128

Training Script: train.sh

The training task uses Automatic Mixed Precision (AMP) for speed improvements.

Model input and output:

  • Input: 4 channel 3D MRIs (T1c, T1, T2, FLAIR)

  • Output: 3 channels of tumor subregion 3D masks

The model was trained with 285 cases with our own split, as shown in the datalist json file in the config folder.

  • Tumor core (TC): 0.859

  • Whole tumor (WT): 0.904

  • Enhancing tumor (ET): 0.786

clara_mri_seg_brain_tumors_br16_full_no_amp

A pre-trained model for volumetric (3D) segmentation of brain tumors from multi-modal MRIs based on BraTS 2018 data.

https://www.med.upenn.edu/sbia/brats2018/data.html

The model is trained to segment 3 nested subregions of primary (gliomas) brain tumors: the “enhancing tumor” (ET), the “tumor core” (TC), and the “whole tumor” (WT), based on 4 input MRI scans ( T1c, T1, T2, FLAIR). The ET is described by areas that show hyper-intensity in T1c when compared to T1, but also when compared to “healthy” white matter in T1c. The TC describes the bulk of the tumor, which is what is typically resected. The TC encompasses the ET, as well as the necrotic (fluid-filled) and the non-enhancing (solid) parts of the tumor. The WT describes the complete extent of the disease, as it entails the TC and the peritumoral edema (ED), which is typically depicted by hyper-intense signal in FLAIR.

The dataset is available at “Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018.” The provided labelled data was partitioned, based our own split, into training (243 studies) and validation (42 studies) datasets.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at: https://www.med.upenn.edu/sbia/brats2018/data.html.

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1].

The provided training configuration required 16GB GPU memory.

Model Input Shape: 224 x 224 x 128

Training Script: train.sh

Model input and output:

  • Input: 4 channel 3D MRIs (T1c, T1, T2, FLAIR)

  • Output: 3 channels of tumor subregion 3D masks

The model was trained with 285 cases with our own split, as shown in the datalist json file in the config folder.

  • Tumor core (TC): 0.851

  • Whole tumor (WT): 0.903

  • Enhancing tumor (ET): 0.773

clara_mri_seg_brain_tumors_br16_t1c2tc_amp

A pre-trained model for volumetric (3D) brain tumor segmentation (only TC from T1c images). The model is trained to segment “tumor core” (TC) based on 1 input MRI scan (T1c).

The dataset is available at “Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018.” The provided labelled data was partitioned, based our own split, into training (243 studies) and validation (42 studies) datasets, as shown in config/seg_brats18_datalist_t1c.json.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at:

https://www.med.upenn.edu/sbia/brats2018/data.html

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1].

The provided training configuration required 16GB GPU memory.

Model Input Shape: 224 x 224 x 128

Training Script: train.sh

The training task uses Automatic Mixed Precision (AMP) for speed improvements.

Model input and output:

  • Input: 1 channel 3D MRI (T1c)

  • Output: 1 channel of tumor core 3D masks

The achieved mean Dice score on the validation data is:

  • Tumor core (TC): 0.818

clara_mri_seg_brain_tumors_br16_t1c2tc_no_amp

A pre-trained model for volumetric (3D) brain tumor segmentation (only TC from T1c images). The model is trained to segment “tumor core” (TC) based on 1 input MRI scan (T1c).

The dataset is available at “Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018.” The provided labelled data was partitioned, based our own split, into training (243 studies) and validation (42 studies) datasets, as shown in config/seg_brats18_datalist_t1c.json.

For more detailed description of tumor regions, please see the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 data page at:

https://www.med.upenn.edu/sbia/brats2018/data.html

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1].

The provided training configuration required 16GB GPU memory.

Model Input Shape: 224 x 224 x 128

Training Script: train.sh

Model input and output:

  • Input: 1 channel 3D MRI (T1c)

  • Output: 1 channel of tumor core 3D masks

The achieved mean Dice score on the validation data is:

  • Tumor core (TC): 0.850

Liver and tumor segmentation

clara_ct_seg_liver_and_tumor_amp

A pre-trained model for volumetric (3D) segmentation of the liver and lesion in portal venous phase CT image.

This model is trained using the runnerup [2] awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” using the AHnet architecture [3].

This model was trained with Liver dataset, as part of “Medical Segmentation Decathlon Challenge 2018”. It consists of 131 labelled data and 70 unlabelled data. The labelled data was partitioned, based on our own split, into 104 training images and 27 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of “Medical Segmentation Decathlon Challenge 2018,” at:

http://medicaldecathlon.com/.

The training dataset is Task03_Liver.tar from the link above.

The data must be converted to 1mm resolution before training:

nvmidl-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}

Note

To match the default setting, set ${DESTINATION_IMAGE_ROOT} to match DATA_ROOT as defined in environment.json in this MMAR’s config folder. Also, the -l flag needs to be used for converting label data in order to use the proper interpolation algorithm.

The provided training configuration required 12GB GPU memory.

Model input shape: dynamic

Training Script: train.sh

The training task uses Automatic Mixed Precision (AMP) for speed improvements.

Model input and output:

Input: 1 channel CT image

Output: 3 channels:

  • Label 1: liver

  • Label 2: tumor

  • Label 0: everything else

This Dice scores on the validation data achieved by this model are:

  • Liver: 0.924

  • Tumor: 0.465

clara_ct_seg_liver_and_tumor_no_amp

A pre-trained model for volumetric (3D) segmentation of the liver and lesion in portal venous phase CT image.

This model is trained using the runnerup [2] awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” using the AHnet architecture [3].

This model was trained with Liver dataset, as part of “Medical Segmentation Decathlon Challenge 2018”. It consists of 131 labelled data and 70 unlabelled data. The labelled data was partitioned, based on our own split, into 104 training images and 27 validation images for this training task, as shown in config/dataset_0.json.

For more detailed description of “Medical Segmentation Decathlon Challenge 2018,” at:

http://medicaldecathlon.com/.

The training dataset is Task03_Liver.tar from the link above.

The data must be converted to 1mm resolution before training:

nvmidl-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii -o ${DESTINATION_IMAGE_ROOT}

Note

To match the default setting, set ${DESTINATION_IMAGE_ROOT} to match DATA_ROOT as defined in environment.json in this MMAR’s config folder. Also, the -l flag needs to be used for converting label data in order to use the proper interpolation algorithm.

The provided training configuration required 12GB GPU memory.

Model input shape: dynamic

Training Script: train.sh

Model input and output:

Input: 1 channel CT image

Output: 3 channels:

  • Label 1: liver

  • Label 2: tumor

  • Label 0: everything else

This Dice scores on the validation data achieved by this model are:

  • Liver: 0.922

  • Tumor: 0.505

Spleen segmentation

clara_ct_seg_spleen_amp

A pre-trained model for volumetric (3D) segmentation of the spleen from CT image.

This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 32 training images and 9 validation images.

The training dataset is Task09_Spleen.tar from http://medicaldecathlon.com/.

The data must be converted to 1mm resolution before training:

nvmidl-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

Note

To match up with the default setting, set ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR’s config folder. Also, the -l flag needs to be used for converting label data in order to use the proper interpolation algorithm.

The training was performed with command train_2gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

The training task uses Automatic Mixed Precision (AMP) for speed improvements.

Model input and output:

Input: 1 channel CT image

Output: 2 channels:

  • Label 1: spleen

  • Label 0: everything else

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Spleen: 0.949

For details of model architecture, see [1]” (Liu et al.)

[1] Myronenko, Andriy. “3D MRI brain tumor segmentation using autoencoder regularization.” International MICCAI Brainlesion Workshop. Springer, Cham, 2018. https://arxiv.org/abs/1810.11654.[2] Xia, Yingda, et al. “3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training.” arXiv preprint arXiv:1811.12506 (2018). https://arxiv.org/abs/1811.12506.[3] Liu, Siqi, et al. “3d anisotropic hybrid network: Transferring convolutional features from 2d images to 3d anisotropic volumes.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. https://arxiv.org/abs/1711.08580.

clara_ct_seg_spleen_no_amp

A pre-trained model for volumetric (3D) segmentation of the spleen from CT image.

This model is trained using the runner-up awarded pipeline of the “Medical Segmentation Decathlon Challenge 2018” with 32 training images and 9 validation images.

The training dataset is Task09_Spleen.tar from http://medicaldecathlon.com/.

The data must be converted to 1mm resolution before training:

nvmidl-dataconvert -d ${SOURCE_IMAGE_ROOT} -r 1 -s .nii.gz -e .nii.gz -o ${DESTINATION_IMAGE_ROOT}

Note

To match up with the default setting, set ${DESTINATION_IMAGE_ROOT} match DATA_ROOT as defined in environment.json in this MMAR’s config folder. Also, the -l flag needs to be used for converting label data in order to use the proper interpolation algorithm.

The training was performed with command train_2gpu.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: dynamic

Actual Model Input: 96 x 96 x 96

Model input and output:

Input: 1 channel CT image

Output: 2 channels:

  • Label 1: spleen

  • Label 0: everything else

This model achieves the following Dice score on the validation data (our own split from the training dataset):

  • Spleen: 0.95

For details of model architecture, see [1]” (Liu et al.)

[1] Myronenko, Andriy. “3D MRI brain tumor segmentation using autoencoder regularization.” International MICCAI Brainlesion Workshop. Springer, Cham, 2018. https://arxiv.org/abs/1810.11654.[2] Xia, Yingda, et al. “3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training.” arXiv preprint arXiv:1811.12506 (2018). https://arxiv.org/abs/1811.12506.[3] Liu, Siqi, et al. “3d anisotropic hybrid network: Transferring convolutional features from 2d images to 3d anisotropic volumes.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. https://arxiv.org/abs/1711.08580.

Classification models

Chest x-ray classification

clara_xray_classification_chest_amp

A pre-trained densenet121 model for disease pattern detection in chest x-rays.

This model is trained using PLCO training data and evaluated on the PLCO validation data.

You can apply for access to the dataset at: https://biometry.nci.nih.gov/cdas/learn/plco/images/

The provided training configuration required 12GB-memory GPUs. The training was performed with command train.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: 256 x 256

Input: 16-bit CXR png

Output: 15 binary labels, each bit is corresponding to the prediction of ‘Nodule’, ‘Mass’, ‘Distortion of Pulmonary Architecture’, ‘Pleural Based Mass’, ‘Granuloma’, ‘Fluid in Pleural Space’, ‘Right Hilar Abnormality’, ‘Left Hilar Abnormality’, ‘Major Atelectasis’, ‘Infiltrate’, ‘Scarring’, ‘Pleural Fibrosis’, ‘Bone/Soft Tissue Lesion’, ‘Cardiac Abnormality’, ‘COPD’

Please refer to “medical/segmentation/examples/brats/tutorial_brats.ipynb” inside the docker and the files in the same folder for details.

This model achieves the following AUC score on the validation data:

Averaged AUC over all disease categories: 0.886

clara_xray_classification_chest_no_amp

A pre-trained densenet121 model for disease pattern detection in chest x-rays.

This model is trained using PLCO training data and evaluated on the PLCO validation data.

You can apply for access to the dataset at: https://biometry.nci.nih.gov/cdas/learn/plco/images/

The provided training configuration required 12GB-memory GPUs. The training was performed with command train.sh, which required 12GB-memory GPUs.

Training Graph Input Shape: 256 x 256

Input: 16-bit CXR png

Output: 15 binary labels, each bit is corresponding to the prediction of ‘Nodule’, ‘Mass’, ‘Distortion of Pulmonary Architecture’, ‘Pleural Based Mass’, ‘Granuloma’, ‘Fluid in Pleural Space’, ‘Right Hilar Abnormality’, ‘Left Hilar Abnormality’, ‘Major Atelectasis’, ‘Infiltrate’, ‘Scarring’, ‘Pleural Fibrosis’, ‘Bone/Soft Tissue Lesion’, ‘Cardiac Abnormality’, ‘COPD’

Please refer to “medical/segmentation/examples/brats/tutorial_brats.ipynb” inside the docker and the files in the same folder for details.

This model achieves the following AUC score on the validation data:

Averaged AUC over all disease categories: 0.797

Annotation models

For all of our annotation models, please check NVIDIA NGC Model Catalog.