TLT Quick Start Guide
=====================

.. _tlt_quick_start_guide:

This page provides a quick start guide for installing and running TLT.

Requirements
-------------

Hardware
^^^^^^^^

The following system configuration is recommended to achieve reasonable training performance with the TLT and supported models provided:

* 32 GB system RAM
* 32 GB of GPU RAM
* 8 core CPU
* 1 NVIDIA GPU
* 100 GB of SSD space

TLT is supported on A100, V100 and RTX 30x0 GPUs.

Software Requirements
^^^^^^^^^^^^^^^^^^^^^

.. _software_requirements:

+-----------------------------------+-------------+
| **Software**                      | **Version** |
+-----------------------------------+-------------+
| Ubuntu 18.04 LTS                  |   18.04     |
+-----------------------------------+-------------+
| python                            |   >=3.6.9   |
+-----------------------------------+-------------+
| docker-ce                         |   >19.03.5  |
+-----------------------------------+-------------+
| docker-API                        |    1.40     |
+-----------------------------------+-------------+
| :code:`nvidia-container-toolkit`  |   >1.3.0-1  |
+-----------------------------------+-------------+
| nvidia-container-runtime          |    3.4.0-1  |
+-----------------------------------+-------------+
| nvidia-docker2                    |   2.5.0-1   |
+-----------------------------------+-------------+
| nvidia-driver                     |    >455     |
+-----------------------------------+-------------+  
| python-pip                        |    >21.06   |
+-----------------------------------+-------------+  
| nvidia-pyindex                    |             |
+-----------------------------------+-------------+  

.. _nvidia-container-toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/overview.html


Installing the Pre-requisites
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. _install_prereq:

The tlt-launcher is strictly a python3 only package, capable of running on python 3.6.9 or 3.7.

1. Install :code:`docker-ce` by following the `official instructions`_.

   .. _official instructions: https://docs.docker.com/engine/install

   Once you have installed docker-ce, follow the `post-installation steps`_ to ensure that the
   docker can be run without :code:`sudo`.

   .. _post-installation steps: https://docs.docker.com/engine/install/linux-postinstall/

2. Install :code:`nvidia-container-toolkit` by following the `install-guide`_.

   .. _install-guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

3. Get an `NGC`_ account and API key:

   a. Go to NGC and click the **Transfer Learning Toolkit** container in the **Catalog** tab. This
      message is displayed: "Sign in to access the PULL feature of this repository".
   b. Enter your Email address and click **Next**, or click **Create an Account**.
   c. Choose your organization when prompted for **Organization/Team**.
   d. Click **Sign In**.

   .. _NGC: https://ngc.nvidia.com/

4. Log in to the NGC docker registry (:code:`nvcr.io`) using the command
   :code:`docker login nvcr.io` and enter the following credentials:

    .. code:: shell

      a. Username: "$oauthtoken"
      b. Password: "YOUR_NGC_API_KEY"

   where :code:`YOUR_NGC_API_KEY` corresponds to the key you generated from step 3.

.. Note:: DeepStream 5.0 - `NVIDIA SDK for IVA inference`_ is recommended.

.. _NVIDIA SDK for IVA inference: https://developer.nvidia.com/deepstream-sdk

Installing TLT
^^^^^^^^^^^^^^

The Transfer Learning Toolkit (TLT) is a Python pip package that is hosted on the NVIDIA PyIndex.
The package uses the docker restAPI under the hood to interact with the NGC Docker registry to
pull and instantiate the underlying docker containers. You must have an NGC account and an API
key associated with your account. See the  :ref:`Installation Prerequisites<install_prereq>` section
for details on creating an NGC account and obtaining an API key.

1. Create a new :code:`virtualenv` using :code:`virtualenvwrapper`.

   You may follow the instructions in this `link`_ to set up a Python virtualenv using a
   virtualenvwrapper.

   .. _link: https://python-guide-cn.readthedocs.io/en/latest/dev/virtualenvs.html

   Once you have followed the instructions to install :code:`virtualenv` and :code:`virtualenvwrapper`,
   set the Python version in the :code:`virtualenv`. This can be done in either of the following ways:
   
      * Defining the environment variable called VIRTUALENVWRAPPER_PYTHON.
        This variable should point to the path where the python3 binary is installed in your local
        machine. You can also add it to your :code:`.bashrc` or :code:`.bash_profile` for setting
        your Python :code:`virtualenv` by default.

         .. code:: shell

            export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3

      * Setting the path to the python3 binary when creating your :code:`virtualenv` using the
        :code:`virtualenvwrapper` wrapper
         
         .. code:: shell

            mkvirtualenv launcher -p /path/to/your/python3

   Once you have logged into the :code:`virtualenv`, the command prompt should show the name of
   your virtual environment

   .. code:: shell

      (launcher) py-3.6.9 desktop:

   When you are done with you session, you may deactivate your :code:`virtualenv` using the
   :code:`deactivate` command:

         .. code:: shell

            deactivate

   You may re-instantiate this created :code:`virtualenv` env using the :code:`workon` command.

   .. code:: shell

      workon launcher

2. Install the tlt launcher Python package called :code:`nvidia-tlt`.

   .. code:: shell

      pip3 install nvidia-pyindex
      pip3 install nvidia-tlt

   .. Note::
   
      The :code:`nvidia-tlt` package is hosted in the :code:`nvidia-pyindex`,
      which has to be installed as a pre-requisite to install :code:`nvidia-tlt`.

      If you had installed an older version of :code:`nvidia-tlt` launcher, you may upgrade
      to the latest version by running the following command.

      .. code-block:: shell

         pip3 install --upgrade nvidia-tlt

3. Invoke the entrypoints using the :code:`tlt` command.

    .. code:: shell

      tlt --help


    The sample output of the above command is:

    .. code:: text

        usage: tlt [-h]
                 {list,stop,info,augment,bpnet,classification,detectnet_v2,dssd,emotionnet,faster_rcnn,fpenet,gazenet,gesturenet,
                 heartratenet,intent_slot_classification,lprnet,mask_rcnn,punctuation_and_capitalization,question_answering,
                 retinanet,speech_to_text,ssd,text_classification,tlt-converter,token_classification,unet,yolo_v3,yolo_v4}
                 ...

        Launcher for TLT

        optional arguments:
        -h, --help            show this help message and exit

        tasks:
              {list,stop,info,augment,bpnet,classification,detectnet_v2,dssd,emotionnet,faster_rcnn,fpenet,gazenet,gesturenet,heartratenet
              ,intent_slot_classification,lprnet,mask_rcnn,punctuation_and_capitalization,question_answering,retinanet,speech_to_text,
              ssd,text_classification,tlt-converter,token_classification,unet,yolo_v3,yolo_v4}  


    Note that under tasks you can see all the launcher-invokable tasks. The following are the
    specific tasks that help with handling the launched commands using the TLT launcher:

    - list
    - stop
    - info


Running the Transfer Learning Toolkit
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Information about the TLT launcher CLI and details on using it to run TLT supported tasks are
captured in the :ref:`section <tlt_launcher>`.

Use the examples
****************

Example Jupyter notebooks for all the tasks that are supported in TLT are available in NGC `resources`_. TLT provides
sample workflows for :ref:`Computer Vision<cv_samples>` and :ref:`Conversational AI<conv_ai_samples>`. 

.. _resources: https://ngc.nvidia.com/catalog/resources

**Computer Vision**

.. _cv_samples:

All the samples for the supported computer vision tasks are hosted on ngc under the `TLT Computer Vision Samples`_.
To run the available examples, download this sample resource by using the following commands.

.. _TLT Computer Vision Samples: https://ngc.nvidia.com/catalog/resources/nvidia:tlt_cv_samples

.. code-block:: shell

     wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tlt_cv_samples/versions/v1.1.0/zip -O tlt_cv_samples_v1.1.0.zip
     unzip -u tlt_cv_samples_v1.1.0.zip  -d ./tlt_cv_samples_v1.1.0 && rm -rf tlt_cv_samples_v1.1.0.zip && cd ./tlt_cv_samples_v1.1.0

**Conversational AI**

.. _conv_ai_samples:

The TLT Conversational AI package, provides several end to end sample workflows to train conversational AI models 
using TLT and subsequently deploying them to jarvis. You can find these samples at:

+--------------------------------+--------------------------------------------------------------------------------------------------------+
|  **Conversational AI Task**    |                                      **Jupyter Notebooks**                                             |
+================================+========================================================================================================+
| Speech to Text                 | `Speech to Text Notebook`_                                                                             |
+--------------------------------+--------------------------------------------------------------------------------------------------------+
| Speech to Text Citrinet        | `Speech to Text Citrinet Notebook`_                                                                    |
+--------------------------------+--------------------------------------------------------------------------------------------------------+
| Question Answering             | `Question Answering Notebook`_                                                                         |
+--------------------------------+--------------------------------------------------------------------------------------------------------+
| Text Classification            | `Text Classification Notebook`_                                                                        |
+--------------------------------+--------------------------------------------------------------------------------------------------------+
| Token Classification           | `Token Classification Notebook`_                                                                       |
+--------------------------------+--------------------------------------------------------------------------------------------------------+
| Punctuation and Capitalization | `Punctuation Capitalization Notebook`_                                                                 |
+--------------------------------+--------------------------------------------------------------------------------------------------------+
| Intent and Slot Classification | `Intent Slot Classification Notebook`_                                                                 |
+--------------------------------+--------------------------------------------------------------------------------------------------------+

.. _Speech to Text Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:speechtotext_notebook
.. _Speech to Text Citrinet Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:speechtotext_citrinet_notebook
.. _Question Answering Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:questionanswering_notebook
.. _Text Classification Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:textclassification_notebook
.. _Token Classification Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:tokenclassification_notebook
.. _Punctuation Capitalization Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:punctuationcapitalization_notebook
.. _Intent Slot Classification Notebook: https://ngc.nvidia.com/resources/nvidia:tlt-jarvis:intentslotclassification_notebook

You can download these resources, by using the NGC CLI command available at the NGC resource page.
Once you download the respective tutorial resource, you may instantiate the jupyter notebook server.

.. code-block:: bash

   pip3 install jupyter
   jupyter notebook --ip 0.0.0.0 --allow-root --port 8888

Copy and paste the link produced from this command into your browser to access the notebook. The /workspace/examples folder will contain a demo notebook.
Feel free to use any free port available to host the notebook if port 8888 is unavailable.


Downloading the Models
^^^^^^^^^^^^^^^^^^^^^^

.. _downloading_the_models:

The Transfer Learning Toolkit Docker gives you access to a repository of pretrained models that can
serve as a starting point when training deep neural networks. These models are hosted on the NGC.
To download the models, please download the NGC CLI and install it. More information about the NGC
Catalog CLI is available `here`_. Once you have installed the CLI, you may follow the instructions below
to configure the NGC CLI and download the models.

.. _here: https://docs.nvidia.com/ngc/ngc-catalog-cli-user-guide/index.html"

Configure the NGC API key
*************************

Using the NGC API Key obtained in :ref:`Installation Prerequisites<install_prereq>`,
configure the enclosed ngc cli by executing this command and following the prompts:

.. code-block:: bash

   ngc config set

Get a list of models
********************

Use this command to get a list of models that are hosted in the NGC model registry:

.. code-block:: bash

   ngc registry model list <model_glob_string>

For the computer vision models, here is an example of using this command:

.. code-block:: bash

   ngc registry model list nvidia/tlt_pretrained_*

.. Note::

   All our classification models have names based on this template:
   ``nvidia/tlt_pretrained_classification:<template>``.

To view all the conversational AI models, you may using the following command:

.. code-block:: bash

   ngc registry model list nvidia/tlt-jarvis/*

Download a model
****************

Use this command to download the model you have chosen from the NGC model registry:

.. code-block:: bash

   ngc registry model download-version <ORG/model_name:version> -dest <path_to_download_dir>

For example, use this command to download the resnet 18 classification model to the
:code:`$USER_EXPERIMENT_DIR` directory:

.. code-block:: bash

   ngc registry model download-version 
   nvidia/tlt_pretrained_classification:resnet18 --dest 
   $USER_EXPERIMENT_DIR/pretrained_resnet18

.. code-block:: bash

   Downloaded 82.41 MB in 9s, Download speed: 9.14 MB/s                
   ----------------------------------------------------
   Transfer id: tlt_iva_classification_resnet18_v1 Download status: Completed.
   Downloaded local path: /workspace/tlt-experiments/pretrained_resnet18/
   tlt_resnet18_classification_v1
   Total files downloaded: 2 
   Total downloaded size: 82.41 MB
   Started at: 2019-07-16 01:29:53.028400
   Completed at: 2019-07-16 01:30:02.053016
   Duration taken: 9s seconds

Training with Jupyter Notebook
------------------------------

Install the Python Virtual Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Follow these `instructions`_ to install the virtualenv with Python 3.6.9.
Once virtualenvwrapper is set up, set the version of python to be
used in the virtual env by using the :code:`VIRTUALENVWRAPPER_PYTHON`
variable. You may do so by running the following:

.. code:: shell

     export VIRTUALENVWRAPPER_PYTHON=/path/to/bin/python3.x

where x >= 6 and <= 8.

.. _instructions: https://virtualenvwrapper.readthedocs.io/en/latest/install.html

Use the following command to instantiate a virtual environment:

.. code:: shell

     mkvirtualenv launcher -p /path/to/bin/python3.x

where x >= 6 and <= 8 

Download Jupyter Notebook
^^^^^^^^^^^^^^^^^^^^^^^^^

TLT provides samples notebooks to walk through an prescrible TLT workflow. These samples are hosted
on NGC as a resource and can be downloaded from NGC by executing the command mentioned below.

.. code-block:: shell

     wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tlt_cv_samples/versions/v1.1.0/zip -O tlt_cv_samples_v1.1.0.zip
     unzip -u tlt_cv_samples_v1.1.0.zip  -d ./tlt_cv_samples_v1.1.0 && rm -rf tlt_cv_samples_v1.1.0.zip && cd ./tlt_cv_samples_v1.1.0

The list with their corresponding samples mapped are mentioned below.

+---------------------------+-------------------------------------+
|      **Model Name**       |        **Jupyter Notebook**         |
+===========================+=====================================+
| VehicleTypeNet            | classification/classification.ipynb |
+---------------------------+-------------------------------------+
| VehicleMakeNet            | classification/classification.ipynb |
+---------------------------+-------------------------------------+
| TrafficCamNet             | detectnet_v2/detectnet_v2.ipynb     |
+---------------------------+-------------------------------------+
| PeopleSegNet              | mask_rcnn/mask_rcnn.ipynb           |
+---------------------------+-------------------------------------+
| PeopleNet                 | detectnet_v2/detectnet_v2.ipynb     |
+---------------------------+-------------------------------------+
| License Plate Recognition | lprnet/lprnet.ipynb                 |
+---------------------------+-------------------------------------+
| License Plate Detection   | detectnet_v2/detectnet_v2.ipynb     |
+---------------------------+-------------------------------------+
| Heart Rate Estimation     | heartratenet/heartratenet.ipynb     |
+---------------------------+-------------------------------------+
| Gesture Recognition       | gesturenet/gesturenet.ipynb         |
+---------------------------+-------------------------------------+
| Gaze Estimation           | gazenet/gazenet.ipynb               |
+---------------------------+-------------------------------------+
| Facial Landmark           | fpenet/fpenet.ipynb                 |
+---------------------------+-------------------------------------+
| FaceDetectIR              | detectnet_v2/detectnet_v2.ipynb     |
+---------------------------+-------------------------------------+
| FaceDetect                | facenet/facenet.ipynb               |
+---------------------------+-------------------------------------+
| Emotion Recognition       | emotionnet/emotionnet.ipynb         |
+---------------------------+-------------------------------------+
| DashCamNet                | detectnet_v2/detectnet_v2.ipynb     |
+---------------------------+-------------------------------------+
| BodyPoseNet               | bpnet/bpnet.ipynb                   |
+---------------------------+-------------------------------------+

Open model architecture:
^^^^^^^^^^^^^^^^^^^^^^^^

+-------------------------+--------------------------------------+
|Open model architecture  |      Jupyter notebook                |
+=========================+======================================+
| DetectNet_v2            | detectnet_v2/detectnet_v2.ipynb      |
+-------------------------+--------------------------------------+
| FasterRCNN              | faster_rcnn/faster_rcnn.ipynb        |
+-------------------------+--------------------------------------+
| YOLOV3                  | yolo_v3/yolo_v3.ipynb                |
+-------------------------+--------------------------------------+
| YOLOV4                  | yolo_v4/yolo_v4.ipynb                |
+-------------------------+--------------------------------------+
| SSD                     | ssd/ssd.ipynb                        |
+-------------------------+--------------------------------------+
| DSSD                    | dssd/dssd.ipynb                      |
+-------------------------+--------------------------------------+
| RetinaNet               | retinanet/retinanet.ipynb            |
+-------------------------+--------------------------------------+
| MaskRCNN                | mask_rcnn/mask_rcnn.ipynb            |
+-------------------------+--------------------------------------+
| UNET                    | unet/unet_isbi.ipynb                 |
+-------------------------+--------------------------------------+
| Classification          | classification/classification.ipynb  |
+-------------------------+--------------------------------------+

Start Jupyter Notebook
^^^^^^^^^^^^^^^^^^^^^^

Once the notebook samples are downloaded, you may start the notebook using the below commands:

.. code-block:: shell
 
     jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root

Open an internet browser on localhost and navigate to the following URL:

.. code-block:: shell

     http://0.0.0.0:8888

.. note:: If you want to run the notebook from a remote server then please follow these `steps`_.

.. _steps: https://docs.anaconda.com/anaconda/user-guide/tasks/remote-jupyter-notebook/

1. Train the Model
^^^^^^^^^^^^^^^^^^

Follow the Notebook instructions to train the model.

You may now deploy these models to DeepStream by following the instructions :ref:`here <deploying_tlt_models_to_deepstream>`.