Overview
========

.. _open_model_architectures:

Transfer Learning Toolkit supports image classification, six object detection architectures--including YOLOv3, YOLOv4, FasterRCNN, SSD, DSSD, RetinaNet, and DetectNet_v2--and a semantic and instance segmentation architecture, namely UNet and MaskRCNN.
In addition, there are 16 classification backbones supported by
TLT. For a complete list of all the permutations that are supported by TLT, see the matrix
below:

+---------------------------+-------------------------+--------------------------------------------------------------------------------------------------+---------------------------+---------------------------+
|                           | **ImageClassification** | **Object Detection**                                                                             | **Instance Segmentation** | **Semantic Segmentation** |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
| **Backbone**              |                         | **DetectNet_V2** | **FasterRCNN** | **SSD** | **YOLOv3** | **RetinaNet** | **DSSD** | **YOLOv4** | **MaskRCNN**              | **UNet**                  |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|ResNet10/18/34/50/101      | Yes                     | Yes              | Yes            | Yes     | Yes        | Yes           | Yes      | Yes        |  Yes                      | Yes                       |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|VGG 16/19                  | Yes                     | Yes              | Yes            | Yes     | Yes        | Yes           | Yes      | Yes        |                           | Yes                       |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|GoogLeNet                  | Yes                     | Yes              | Yes            | Yes     | Yes        | Yes           | Yes      | Yes        |                           |                           |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|MobileNet V1/V2            | Yes                     | Yes              | Yes            | Yes     | Yes        | Yes           | Yes      | Yes        |                           |                           |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|SqueezeNet                 | Yes                     | Yes              |                | Yes     | Yes        | Yes           | Yes      | Yes        |                           |                           |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|DarkNet 19/53              | Yes                     | Yes              | Yes            | Yes     | Yes        | Yes           | Yes      | Yes        |                           |                           |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|CSPDarkNet 19/53           | Yes                     |                  |                |         |            |               |          | Yes        |                           |                           |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|Efficientnet B0            | Yes                     | Yes              | Yes            | Yes     |            | Yes           | Yes      |            |                           | Yes                       |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+
|Efficientnet B1            | Yes                     |                  | Yes            |         |            |               |          |            |                           |                           |
+---------------------------+-------------------------+------------------+----------------+---------+------------+---------------+----------+------------+---------------------------+---------------------------+

Training
^^^^^^^^

The TLT container contains Jupyter notebooks and the necessary spec files to train any network
combination. The pre-trained weight for each backbone is provided on NGC. The pre-trained model is
trained on Open image dataset. The pre-trained weights provide a great starting point for applying
transfer learning on your own dataset.

To get started, first choose the type of model that you want to train, then go to the appropriate
model card on NGC and choose one of the supported backbones.

+----------------------+-------------------------------+
| **Model to train**   | **NGC model card**            |
+----------------------+-------------------------------+
| YOLOv3               | `TLT object detection`_       |
+----------------------+                               |
| YOLOv4               |                               |
+----------------------+                               |
| SSD                  |                               |
+----------------------+                               |
| FasterRCNN           |                               |
+----------------------+                               |
| RetinaNet            |                               |
+----------------------+                               |
| DSSD                 |                               |
+----------------------+-------------------------------+
| DetectNet_v2         | `TLT DetectNet_v2 detection`_ |
|                      |                               |
+----------------------+-------------------------------+
| MaskRCNN             | `TLT instance segmentation`_  |
+----------------------+-------------------------------+
| Image Classification | `TLT image classification`_   |
|                      |                               |
+----------------------+-------------------------------+
| UNet                 | `TLT semantic segmentation`_  |
+----------------------+-------------------------------+

.. _TLT object detection: https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection
.. _TLT DetectNet_v2 detection: https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_detectnet_v2
.. _TLT instance segmentation: https://ngc.nvidia.com/catalog/models/nvidia:tlt_instance_segmentation
.. _TLT image classification: https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_classification
.. _TLT semantic segmentation: https://ngc.nvidia.com/catalog/models/nvidia:tlt_semantic_segmentation

Once you pick the appropriate pre-trained model, follow the TLT workflow to use your dataset and
pre-trained model to export a tuned model that is adapted to your use case. The **TLT Workflow**
sections walk you through all the steps in training.

.. image:: ../../content/tlt_workflow.png

Deployment
^^^^^^^^^^

You can deploy most trained models on any edge device using DeepStream and TensorRT. See the
:ref:`Integrating TLT models into DeepStream <deploying_tlt_models_to_deepstream>` chapter
for deployment instructions.

Some of the models are currently not supported in DeepStream, so we have provided a separate
inference application. See the :ref:`Integrating TLT CV models with the inference pipeline
<integrating_tlt_cv_models_with_the_inference_pipeline>` chapter for deployment instructions.

We also have a reference application for deployment with Triton. See the :ref:`Integrating TLT CV
models with Triton Inference server <integrating_tlt_cv_models_with_triton_inference_server>`
chapter for Triton instructions.