Integrating TLT Models into DeepStream ====================================== .. _deploying_tlt_models_to_deepstream: .. include:: excerpts/deploying_to_deepstream.rst The following TLT models can be integrated into DeepStream 5.1: * Image Classification * Object Detection * Yolo V3 * Yolo V4 * DSSD * SSD * RetinaNet * DetectNet_v2 * FasterRCNN * Instance Segmentation * Mask-RCNN * Semantic Segmentation * UNet * Character Recognition * MultiTask Classification Of these models, the following *do not* support direct integration of the :code:`.etlt` files into DeepStream: * YOLOv3 * YOLOv4 * UNet * Character Recognition You can deploy most trained models with DeepStream SDK. Follow the instructions below to integrate TLT models with Deepstream. We recommend running TLT models with DeepStream 5.1. .. image:: ../content/tlt_overview.png The following models have been integrated into DeepStream 5.1 with other models on the roadmap for future releases. .. tabularcolumns:: |>{\centering\arraybackslash}\X{1}{5}|>{\raggedleft\arraybackslash}\X{2}{5}| .. csv-table:: Open Model Deployment :file: ../content/open_arch_tables.csv :class: longtable :header-rows: 1 :stub-columns: 1 .. tabularcolumns:: |>{\centering\arraybackslash}\X{1}{5}|>{\raggedleft\arraybackslash}\X{2}{5}| .. csv-table:: Pre-trained Model Deployment :file: ../content/cv_ptm_ds_deployment.csv :class: longtable :header-rows: 1 :stub-columns: 1 Installation Prerequisites -------------------------- 1. Install `Jetpack 4.5.1`_ for Jetson devices. **Note**: For Jetson devices, use the following commands to manually increase the `Jetson Power mode`_ and maximize performance further by using the `Jetson Clocks mode`_: .. code-block:: shell sudo nvpmodel -m 0 sudo /usr/bin/jetson_clocks .. _Jetson Power mode: https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fpower_management_jetson_xavier.html%23wwpID0E0NO0HA .. _Jetson Clocks mode: https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fpower_management_jetson_xavier.html%23wwpID0E0SB0HA .. _Jetpack 4.5.1: https://developer.nvidia.com/embedded/jetpack 2. Install `Deepstream`_. .. _Deepstream: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Quickstart.html Deployment Files ---------------- The following files are required to run each TLT model with Deepstream: * :code:`ds_tlt.c`: The application main file * :code:`nvdsinfer_custombboxparser_tlt`: A custom parser function on infernece end nodes * Models: TLT models from NGC * Model configuration files: The Deepstream Inference configuration file Sample Application ------------------- We have provided several reference applications on GitHub. Reference app for YOLOv3/YOLOv4, FasterRCNN, SSD/DSSD, RetinaNet, MaskRCNN, UNet - `DeepStream TLT reference app`_ Reference app for License plate detection and Recognition - `DeepStream LPR app`_ .. _DeepStream TLT reference app: https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps .. _DeepStream LPR app: https://github.com/NVIDIA-AI-IOT/deepstream_lpr_app Pre-trained models - License Plate Detection (LPDNet) and Recognition (LPRNet) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The following steps outline how to run the License Plate Detection and Recognition application: `DeepStream LPR app`_ Download the Repository *********************** .. code:: shell git clone https://github.com/NVIDIA-AI-IOT/deepstream_lpr_app.git Download the Models ******************* .. code-block:: shell cd deepstream_lpr_app mkdir -p ./models/tlt_pretrained_models/trafficcamnet cd ./models/tlt_pretrained_models/trafficcamnet wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_trafficcamnet/versions/pruned_v1.0/files/trafficnet_int8.txt wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_trafficcamnet/versions/pruned_v1.0/files/resnet18_trafficcamnet_pruned.etlt cd - mkdir -p ./models/LP/LPD cd ./models/LP/LPD wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_lpdnet/versions/pruned_v1.0/files/usa_pruned.etlt wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_lpdnet/versions/pruned_v1.0/files/usa_lpd_cal.bin wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_lpdnet/versions/pruned_v1.0/files/usa_lpd_label.txt cd - mkdir -p ./models/LP/LPR cd ./models/LP/LPR wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_lprnet/versions/deployable_v1.0/files/us_lprnet_baseline18_deployable.etlt touch labels_us.txt cd - Convert the Models to TRT Engine ******************************** See the :ref:`TLT Converter ` section. .. code-block:: shell ./tlt-converter -k nvidia_tlt -p image_input,1x3x48x96,4x3x48x96,16x3x48x96 models/LP/LPR/us_lprnet_baseline18_deployable.etlt -t fp16 -e models/LP/LPR/lpr_us_onnx_b16.engine Build and Run ************* .. code-block:: shell make cd deepstream-lpr-app For US car plate recognition: .. code-block:: shell cp dict_us.txt dict.txt Start to run the application: .. code-block:: shell ./deepstream-lpr-app <1:US car plate model|2: Chinese car plate model> <1: output as h264 file| 2:fakesink 3:display output> [0:ROI disable|1:ROI enable] [input mp4 file path and name] [input mp4 file path and name] ... [input mp4 file path and name] [output 264 file path and name] For detailed instructions about running this application, refer to this GitHub `repository`_. .. _repository: https://github.com/NVIDIA-AI-IOT/deepstream_lpr_app Pre-trained models - PeopleNet, TrafficCamNet, DashCamNet, FaceDetectIR, Vehiclemakenet, Vehicletypenet, PeopleSegNet, PeopleSemSegNet ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * :ref:`Deepstream-TLT PeopleNet model Deployment` * :ref:`Deepstream-TLT TrafficCamNet model Deployment` * :ref:`Deepstream-TLT DashCamNet model deployment` * :ref:`Deepstream-TLT FaceDetectIR model deployment` * :ref:`Deepstream-TLT PeopleSegNet model deployment` * :ref:`Deepstream-TLT PeopleSemSegNet model deployment` * `Deepstream-TLT apps`_ .. _Deepstream-TLT apps: https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps .. _deepstream_peoplenet: PeopleNet ********* Follow these instructions to run the PeopleNet model in DeepStream: 1. Download the model: .. code-block:: shell mkdir -p $HOME/peoplenet && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_peoplenet/versions/pruned_v2.1/files/resnet34_peoplenet_pruned.etlt \ -O $HOME/peoplenet/resnet34_peoplenet_pruned.etlt 2. Run the application: .. code-block:: shell xhost + docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME:/opt/nvidia/deepstream/deepstream-5.1/samples/models/tlt_pretrained_models \ -w /opt/nvidia/deepstream/deepstream-5.1/samples/configs/tlt_pretrained_models nvcr.io/nvidia/deepstream:5.1-21.02-samples \ deepstream-app -c deepstream_app_source1_peoplenet.txt .. _deepstream_trafficcamnet: TrafficCamNet ************* Follow these instructions to run the TrafficCamNet model in DeepStream: 1. Download the model: .. code-block:: shell mkdir -p $HOME/trafficcamnet && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_trafficcamnet/versions/pruned_v1.0/files/resnet18_trafficcamnet_pruned.etlt \ -O $HOME/trafficcamnet/resnet18_trafficcamnet_pruned.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_trafficcamnet/versions/pruned_v1.0/files/trafficnet_int8.txt \ -O $HOME/trafficcamnet/trafficnet_int8.txt 2. Run the application: .. code-block:: shell xhost + docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME:/opt/nvidia/deepstream/deepstream-5.1/samples/models/tlt_pretrained_models \ -w /opt/nvidia/deepstream/deepstream-5.1/samples/configs/tlt_pretrained_models nvcr.io/nvidia/deepstream:5.1-21.02-samples \ deepstream-app -c deepstream_app_source1_trafficcamnet.txt .. _deepstream_dashcamnet: DashCamNet + Vehiclemakenet + Vehicletypenet ******************************************** Follow these instructions to run the DashCamNet model as primary detector and Vehiclemakenet and Vehicletypenet as secondary classifier in DeepStream: 1. Download the model: .. code-block:: shell mkdir -p $HOME/dashcamnet && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_dashcamnet/versions/pruned_v1.0/files/resnet18_dashcamnet_pruned.etlt \ -O $HOME/dashcamnet/resnet18_dashcamnet_pruned.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_dashcamnet/versions/pruned_v1.0/files/dashcamnet_int8.txt \ -O $HOME/dashcamnet/dashcamnet_int8.txt mkdir -p $HOME/vehiclemakenet && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_vehiclemakenet/versions/pruned_v1.0/files/resnet18_vehiclemakenet_pruned.etlt \ -O $HOME/vehiclemakenet/resnet18_vehiclemakenet_pruned.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_vehiclemakenet/versions/pruned_v1.0/files/vehiclemakenet_int8.txt \ -O $HOME/vehiclemakenet/vehiclemakenet_int8.txt mkdir -p $HOME/vehicletypenet && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_vehicletypenet/versions/pruned_v1.0/files/resnet18_vehicletypenet_pruned.etlt \ -O $HOME/vehicletypenet/resnet18_vehicletypenet_pruned.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_vehicletypenet/versions/pruned_v1.0/files/vehicletypenet_int8.txt \ -O $HOME/vehicletypenet/vehicletypenet_int8.txt 2. Run the application: .. code-block:: shell xhost + docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME:/opt/nvidia/deepstream/deepstream-5.1/samples/models/tlt_pretrained_models \ -w /opt/nvidia/deepstream/deepstream-5.1/samples/configs/tlt_pretrained_models nvcr.io/nvidia/deepstream:5.1-21.02-samples \ deepstream-app -c deepstream_app_source1_dashcamnet_vehiclemakenet_vehicletypenet.txt .. _deepstream_facedetection: FaceDetectIR ************ Follow these instructions to run the FaceDetectIR model in DeepStream: 1. Download the model: .. code-block:: shell mkdir -p $HOME/facedetectir && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_facedetectir/versions/pruned_v1.0/files/resnet18_facedetectir_pruned.etlt \ -O $HOME/facedetectir/resnet18_facedetectir_pruned.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_facedetectir/versions/pruned_v1.0/files/facedetectir_int8.txt \ -O $HOME/facedetectir/facedetectir_int8.txt 2. Run the application: .. code-block:: shell xhost + docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME:/opt/nvidia/deepstream/deepstream-5.1/samples/models/tlt_pretrained_models \ -w /opt/nvidia/deepstream/deepstream-5.1/samples/configs/tlt_pretrained_models nvcr.io/nvidia/deepstream:5.1-21.02-samples \ deepstream-app -c deepstream_app_source1_facedetectir.txt .. _deepstream_peoplesegnet: PeopleSegNet ************ Follow these instructions to run the PeopleSegNet model in DeepStream: 1. Download the Repository: .. code-block:: shell git clone https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps.git 2. Download the model: .. code-block:: shell ngc registry model download-version "nvidia/tlt_peoplesegnet:deployable_v2.0" or .. code-block:: shell wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/tlt_peoplesegnet/versions/deployable_v2.0/zip \ -O tlt_peoplesegnet_deployable_v2.0.zip 3. Build TRT OSS Plugin: TRT-OSS instructions are provided in https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/tree/master#1-build-trt-oss-plugin 4. Build the application: .. code-block:: shell export CUDA_VER=xy.z // xy.z is CUDA version, e.g. 10.2 make 5. Run the application: .. code-block:: shell SHOW_MASK=1 ./apps/ds-tlt -c configs/peopleSegNet_tlt/pgie_peopleSegNetv2_tlt_config.txt -i \ /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.h264 -d .. _deepstream_peoplesemsegnet: PeopleSemSegNet *************** Follow these instructions to run the PeopleSemSegNet model in DeepStream: 1. Download tlt-converter and the model: .. code-block:: shell mkdir $HOME/deepstream cd $HOME/deepstream wget https://developer.nvidia.com/cuda111-cudnn80-trt72 unzip cuda111-cudnn80-trt72 cp cuda11.1_cudnn8.0_trt7.2/tlt-converter ./ chmod 0777 tlt-converter ngc registry model download-version "nvidia/tlt_peoplesemsegnet:deployable_v1.0" 2. Run the application: .. code-block:: shell xhost + docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME:$HOME -w $HOME/deepstream \ nvcr.io/nvidia/deepstream:5.1-21.02-devel ./tlt-converter -k tlt_encode -p input_1,1x3x544x960,1x3x544x960,1x3x544x960 -t fp16 -e \ tlt_peoplesemsegnet_vdeployable_v1.0/unet_resnet18.etlt_b1_gpu0_fp16.engine tlt_peoplesemsegnet_vdeployable_v1.0/peoplesemsegnet.etlt ; \ git clone https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps.git ; cd deepstream_tlt_apps ; export CUDA_VER=11.1 ; export SHOW_MASK=1 ; make ; \ sed -i "s/..\/..\/models\/unet\/unet_resnet18.etlt_b1_gpu0_fp16.engine/..\/..\/..\/tlt_peoplesemsegnet_vdeployable_v1.0\/unet_resnet18.etlt_b1_gpu0_fp16.engine/g" \ configs/unet_tlt/pgie_unet_tlt_config.txt ; sed -i "s/infer-dims=3;608;960/infer-dims=3;544;960/g" configs/unet_tlt/pgie_unet_tlt_config.txt ; \ sed -i "s/unet_labels.txt/..\/..\/..\/tlt_peoplesemsegnet_vdeployable_v1.0\/labels.txt/g" configs/unet_tlt/pgie_unet_tlt_config.txt ; \ sed -i "s/num-detected-classes=3/num-detected-classes=2/g" configs/unet_tlt/pgie_unet_tlt_config.txt ; ./apps/ds-tlt -c configs/unet_tlt/pgie_unet_tlt_config.txt \ -i /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.h264 -d .. _deepstream_general_purpose: General purpose CV model architecture - Classification, Object detection and Segmentation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A sample DeepStream app to run a classification, object detection, and semantic and instance segmentation network is provided in `here `_. TRT-OSS instructions are provided in the repo above. For more information about each individual model architecture, see the **Deploying to DeepStream** section under each model: * :ref:`Image Classification ` * Object Detection * :ref:`YOLOv3 ` * :ref:`YOLOv4 ` * :ref:`DSSD ` * :ref:`SSD ` * :ref:`RetinaNet ` * :ref:`DetectNet_v2 ` * :ref:`FasterRCNN ` * Instance Segmentation * :ref:`MaskRCNN ` * Semantic Segmentation * :ref:`UNet ` * :ref:`Character Recognition ` * :ref:`Multi-Task Classification `