Gst-nvdspostprocess in DeepStream

The Gst-nvdspostprocess plugin is released in DeepStream 6.1. The plugin supports parsing of various inferencing models in DeepStream SDK. The plugin can perform parsing on the tensors of the output layers provided by the Gst-nvinfer and Gst-nvinferserver. The aim of this document is to provide guidance on how to use the Gst-nvdspostprocess plugin for various inference models. The document is divided into four parts. First section will cover Detector models such as Yolo V3 and Faster RCNN, second section will demonstrate using classification model as primary classifier with Gst-nvinferserver, while third section will cover Mask RCNN. Finally the document is concluded by providing table for using various custom functions that can be used for parsing of output layers.

Detector models

For using Yolo V3 detector follow the prequisite steps mentioned in /opt/nvidia/deepstream/deepstream/sources/objectDetector_Yolo/README. Check if the setup configured correctly by running below test pipelines in following folder /opt/nvidia/deepstream/deepstream/sources/objectDetector_Yolo/.

#For dGPU
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin !  \
m.sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! nvinfer config-file-path=config_infer_primary_yoloV3.txt ! \
nvvideoconvert ! nvdsosd ! nveglglessink sync=0

#For Jetson
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin !  \
m.sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! nvinfer config-file-path=config_infer_primary_yoloV3.txt ! \
nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0

Inorder to update the above pipeline to use the post processing plugin for parsing the /opt/nvidia/deepstream/deepstream/sources/objectDetector_Yolo/config_infer_primary_yoloV3.txt needs to be modified by changing the network-type=0 to network-type=100 by doing this output post processing is disabled in nvinfer plugin. Another parameter that needs to be set is output-tensor-meta=1, nvinfer plugin then attaches the tensor meta to the input buffer. Store the modified file as config_infer_primary_yoloV3_modified.txt. The post processing plugin config file in YAML format has to be created as below.

property:
  gpu-id: 0 #Set the GPU id
  process-mode: 1 # Set the mode as primary inference
  num-detected-classes: 80 # Change according the models output
  gie-unique-id: 1  # This should match the one set in inference config
  ## 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
  cluster-mode: 2  # Set  appropriate clustering algorithm
  network-type: 0  # Set the network type as detector
  labelfile-path: labels.txt # Set the path of labels wrt to this config file
  parse-bbox-func-name: NvDsPostProcessParseCustomYoloV3 # Set custom parsing function

class-attrs-all: # Set as done in the original infer configuration
  nms-iou-threshold: 0.5
  pre-cluster-threshold: 0.7

Save the above config as config_detector.yml. The following pipeline can be executed as given below.

#For dGPU
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin ! \
sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! nvinfer config-file-path=config_infer_primary_yoloV3_modified.txt ! \
nvdspostprocess postprocesslib-config-file=config_detector.yml \
postprocesslib-name=/opt/nvidia/deepstream/deepstream/lib/libpostprocess_impl.so ! nvvideoconvert ! nvdsosd ! nveglglessink sync=0

#For Jetson
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin ! \
sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! nvinfer config-file-path=config_infer_primary_yoloV3_modified.txt ! \
nvdspostprocess postprocesslib-config-file=config_detector.yml \
postprocesslib-name=/opt/nvidia/deepstream/deepstream/lib/libpostprocess_impl.so ! nvvideoconvert ! nvdsosd ! \
nvegltransform ! nveglglessink sync=0

Note

The NvDsPostProcessParseCustomYoloV3 function is defined in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspostprocess/postprocesslib_impl/post_processor_custom_impl.cpp

Similar process as given above can be followed to demonstrate, usage of Faster RCNN network (/opt/nvidia/deepstream/deepstream/sources/objectDetector_FasterRCNN/README), with nvdspostprocess plugin with below config_detector.yml

property:
  gpu-id: 0 #Set the GPU id
  process-mode: 1 # Set the mode as primary inference
  num-detected-classes: 21 # Change according the models output
  gie-unique-id: 1  # This should match the one set in inference config
  ## 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
  cluster-mode: 2  # Set  appropriate clustering algorithm
  network-type: 0  # Set the network type as detector
  labelfile-path: labels.txt # Set the path of labels wrt to this config file
  parse-bbox-func-name: NvDsPostProcessParseCustomFasterRCNN # Set custom parsing function FRCNN

class-attrs-all: # Set as done in the original infer configuration
  topk: 20
  nms-iou-threshold: 0.4
  pre-cluster-threshold: 0.5

class-attrs-0:
  pre-cluster-threshold: 1.1

The pipeline for running the Faster RCNN network with modified nvinfer config and post process plugin is given below.

#For dGPU
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin !  \
m.sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! nvinfer config-file-path=config_infer_primary_fasterRCNN_modified.txt ! \
nvdspostprocess postprocesslib-config-file=config_detector.yml postprocesslib-name=/opt/nvidia/deepstream/deepstream/lib/libpostprocess_impl.so ! \
nvvideoconvert ! nvdsosd ! nveglglessink sync=0

#For Jetson
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin !  \
m.sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! nvinfer config-file-path=config_infer_primary_fasterRCNN_modified.txt ! \
nvdspostprocess postprocesslib-config-file=config_detector.yml postprocesslib-name=/opt/nvidia/deepstream/deepstream/lib/libpostprocess_impl.so ! \
nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0

Primary Classification model

The primary classification model would be demonstrated using the DeepStream Triton Docker Containers on dGPU. Once the docker is running the model repo and classification video should be created. Execute following commands to download the model repo and create a sample classification video.

cd /opt/nvidia/deepstream/deepstream/samples
./prepare_ds_triton_model_repo.sh
apt install ffmpeg
./prepare_classification_test_video.sh
cd /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app-triton

Check by running following sample classification pipeline

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/classification_test_video.mp4  ! decodebin ! \
m.sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 ! \
nvinferserver config-file-path=config_infer_primary_classifier_densenet_onnx.txt  \
! nvvideoconvert ! nvdsosd ! nveglglessink sync=1

Note

To use nveglglessink inside docker ensure xhost + done from the host, and set appropriate DISPLAY enviornment variable inside the docker.

Now, update the config_infer_primary_classifier_densenet_onnx.txt to disable post processing and attaching tensor output meta in nvinferserver. This can be done by updating configuration file with following parameters infer_config { postprocess { other {} } } and output_control { output_tensor_meta : true }

infer_config {
  unique_id: 5
  gpu_ids: [0]
  max_batch_size: 1
  backend {
    triton {
      model_name: "densenet_onnx"
      version: -1
      model_repo {
        root: "../../triton_model_repo"
        strict_model_config: true
        tf_gpu_memory_fraction: 0.0
        tf_disable_soft_placement: 0
      }
    }
  }
  preprocess {
    network_format: IMAGE_FORMAT_RGB
    tensor_order: TENSOR_ORDER_LINEAR
    maintain_aspect_ratio: 0
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    normalize {
    scale_factor: 0.0078125
    channel_offsets: [128, 128, 128]
    }
  }
  #Disable post processing in nvinferserver
  postprocess {
    other {
    }
  }
  extra {
    copy_input_to_host_buffers: false
    output_buffer_pool_size: 2
  }
}
input_control {
  process_mode: PROCESS_MODE_FULL_FRAME
  interval: 0
}
#Enable attaching output tensor meta in nvinferserver
output_control {
  output_tensor_meta: true
}

Save the above config as config_infer_primary_classifier_densenet_onnx_modified.txt. Create a config_classifier.yml as given below.

property:
  gpu-id: 0
  network-type: 1 # Type of network i.e. classifier
  process-mode: 1 # Operate in primary mode i.e. operate on full frame
  classifier-threshold: 0.2 #Set classifier threshold
  gie-unique-id: 5 # Set the unique_id matching one in the inference
  classifier-type: ObjectClassifier # type of classfier
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/triton_model_repo/densenet_onnx/densenet_labels.txt #Path of the labels fine

The following pipeline with nvdspostprocess plugin can now be executed to view the classification results

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/classification_test_video.mp4  ! decodebin ! \
 m.sink_0 nvstreammux name=m batch-size=1 width=1920  height=1080 !   nvinferserver \
 config-file-path=config_infer_primary_classifier_densenet_onnx_modified.txt ! \
 nvdspostprocess postprocesslib-config-file= config_classifier.yml postprocesslib-name= \
 /opt/nvidia/deepstream/deepstream/lib/libpostprocess_impl.so ! nvvideoconvert ! nvdsosd ! nveglglessink sync=1

Mask RCNN Model

To use the instance segmentation model follow the README in package /opt/nvidia/deepstream/deepstream/samples/configs/tao_pretrained_models/README.md to obtain TAO toolkit config files and PeopleSegNet model. Once setup is done, execute following pipeline to validate the model.

cd /opt/nvidia/deepstream/deepstream/samples/configs/tao_pretrained_models
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin ! \
m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= config_infer_primary_peopleSegNet.txt  ! \
nvvideoconvert ! nvdsosd display-mask=1 process-mode=0 ! nveglglessink sync=0

Note

For correct operation ensure the Tensor-RT OSS plugin is compiled and replaced as mentioned in the TAO README.

As mentioned in earlier sections update the nvinfer configuration file to disable post processing and enable attaching tensor output meta. This is done by changing the network-type=100 and output-tensor-meta=1. Store the file by the name config_infer_primary_peopleSegNet_modified.txt. The config_mrcnn.yml can be created as given below.

property:
  gpu-id: 0
  process-mode: 1 # Process on full frame
  num-detected-classes: 2 #Total Detected classes
  gie-unique-id: 1  #Match with gie-unique-id of inference config
  ## 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
  cluster-mode: 4 # Disable clustering
  network-type: 3 # Network is instance segmentation
  labelfile-path: peopleSegNet_labels.txt
  parse-bbox-instance-mask-func-name: NvDsPostProcessParseCustomMrcnnTLTV2

class-attrs-all:
  pre-cluster-threshold: 0.8

Following pipeline can be used for testing the nvdspostprocess plugin with MRCNN network, using the above configuration files.

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! decodebin !   \
m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! \
nvinfer config-file-path= config_infer_primary_peopleSegNet.txt ! \
nvdspostprocess postprocesslib-name= /opt/nvidia/deepstream/deepstream/lib/libpostprocess_impl.so \
postprocesslib-config-file= config_mrcnn.yml  !   nvvideoconvert ! nvdsosd display-mask=1 process-mode=0 ! nveglglessink sync=0

Custom Parsing functions

This section mentions the parsing functions present in postprocess library for available network architectures.

Custom Parsing functions supported

Custom Parsing Function	Description
NvDsPostProcessParseCustomResnet	Parsing Resnet 10 model packaged in DeepStream
NvDsPostProcessParseCustomTfSSD	Tensorflow/Onnx SSD detector
NvDsPostProcessParseCustomNMSTLT	Parsing TAO Toolkit Open Architecture Models SSD, FRCNN, DSSD, RetinaNet
NvDsPostProcessParseCustomBatchedNMSTLT	Parsing of TAO Toolkit Open Architecture Models Yolo V3, Yolo V4
NvDsPostProcessParseCustomMrcnnTLTV2	Parsing of TAO Toolkit Open Architecture Model MaskRCNN
NvDsPostProcessParseCustomFasterRCNN	Parsing of Faster R-CNN Network
NvDsPostProcessClassiferParseCustomSoftmax	Parsing Resnet 18 vehicle type classifier model packaged in DeepStream
NvDsPostProcessParseCustomSSD	Parsing of SSD Network
NvDsPostProcessParseCustomYoloV3	Parsing of Yolo V3 Network
NvDsPostProcessParseCustomYoloV3Tiny	Parsing of Yolo V3 Tiny Network
NvDsPostProcessParseCustomYoloV2	Parsing of Yolo V2 Network
NvDsPostProcessParseCustomYoloV2Tiny	Parsing of Yolo V2 Tiny Network