Skip to navigation
DeepStream Plugin GuidePlugins

Gst-nvdspostprocess

View as Markdown

The Gst-nvdspostprocess plugin is a customizable plugin which provides a custom library interface for post processing on Tensor output of inference plugins (nvinfer/nvinferserver). Currently the plugin supports detection and classification models for parsing. The plugin connects to post processing lib provided as plugin parameter. The parsing parameters can be specified via a config file. Refer to section Gst-nvdspostprocess File Configuration Specifications below for more details.

For the plugin to perform parsing of inference tensor output, it is necessary to attach tensor output by setting output-tensor-meta as 1 in inference configuration file and disabling parsing in the inference plugin by setting network-type as 100 i.e. network type other. Similarly, in Gst-nvinferserver we can enable the following configs in the configuration file. output_control { output_tensor_meta : true } and to disable native post processing, update: infer_config { postprocess { other {} } }.

Default plugin implementation provides following functionalities.

  1. It parses the tensor meta attached to frame meta or object meta.
  2. Attaches the output of tensor parsing either to frame meta or object meta.

The default custom library (postprocesslib_impl) provided with the plugin implements these functionalities. It can be configured to parse detection and classification networks. For multiple inference plugins in pipeline require multiple post process plugin instances corresponding to them.

Gst-nvdspostprocess

Inputs and Outputs

  • Inputs

    • Input Video Gst Buffers
    • Metadata (NvDsBatchMeta)
    • Tensor Meta (NvDsInferTensorMeta)

  • Control parameters

    • postprocesslib-config-file
    • postprocesslib-name
    • gpu-id

  • Output

    • Output Video Gst Buffers
    • Original Metadata (NvDsBatchMeta) (with addition of Object Meta/Classifier Meta)

Features

The following table summarizes the features of the plugin.

Gst-nvdspostprocess features

FeatureDescriptionRelease
Detection output parsingParses detector tensor data and attaches the results as object metadataDS 6.1
Classification output parsingParses Classification tensor data and attaches the result as classifier attributes in metadataDS 6.1
Yolo detector (YoloV3/V3-tiny/V2/V2-tiny) parsingSupport for parsing of Yolo detector outputDS 6.1
FasterRCNN output parsing---DS 6.1
Single Shot Detector (SSD) output parsing---DS 6.1

Gst-nvdspostprocess File Configuration Specifications

The Gst-nvdspreprocess configuration file uses a “YAML” format Refer to examples like config_detector.yml, config_classifier_car_color.yml located at /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspostprocess/. The property group configures the general behavior of the plugin. The class-attrs-all group configures parameters for all classes for detector post processor. The class-attrs-<id> group configures parameters <id> specific class for detector post processor.

The following two tables describes the keys supported for property groups and class-attrs-<id> groups respectively.

Gst-nvdspostprocess property Group Supported Keys

PropertyMeaningType and RangeExample
gpu-idGPU to be used for processingIntegergpu-id: 0
process-modeMode of operation Full Frame (Primary 1) or on Object (Secondary 2)Integer 1=Primary(On Full Frame) ,2=Secondary (On Object)process-mode: 1
gie-unique-idPerform tensor meta parsing on output of gie-unique-idInteger >0gie-unique-id: 1
num-detected-classesNumber of classes detected by Detector networkInteger >0num-detected-classes: 4
cluster-modeClustering mode to be used on detector outputInteger 1=DBSCAN 2=NMS 3=DBSCAN+NMS Hybrid 4=None(No Clustering)cluster-mode: 2
output-blob-namesArray of output layer names which are to be parsedString delimited by semicolonoutput-blob-names: conv2d_bbox;conv2d_cov/Sigmoid
network-typeType of network to be parsedInteger 0=Detector 1=Classifiernetwork-type: 1
labelfile-pathPathname of a text file containing the labels for the modelStringlabelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
classifier-thresholdMinimum threshold label probability. The GIE outputs the label having the highest probability if it is greater than this thresholdFloat, >0.0classifier-threshold: 0.4
operate-on-gie-idUnique ID of the GIE on whose metadata (bounding boxes) this GIE is to operate on, applicable for process-mode=2Integer >0operate-on-gie-id: 1
parse-classifier-func-nameName of the custom classifier output parsing function. If not specified, post process lib uses the parsing function for softmax layers.Stringparse-classifier-func-name: NvDsPostProcessClassiferParseCustomSoftmax
parse-bbox-func-nameName of the custom bounding box parsing function. If not specified, post process lib uses the function for the resnet model provided by the SDKStringparse-bbox-func-name: NvDsPostProcessParseCustomResnet

Gst-nvdspostprocess class-attr-<id> Group Supported Keys

PropertyMeaningType and RangeExample
topkKeep only top K objects with highest detection scores.Integer, >0. -1 to disabletopk: 10
nms-iou-thresholdMaximum IOU score between two proposals after which the proposal with the lower confidence will be rejected.Float, >0.0nms-iou-threshold: 0.2
pre-cluster-thresholdDetection threshold to be applied prior to clustering operationFloat, >0.0pre-cluster-threshold: 0.5
post-cluster-thresholdDetection threshold to be applied post clustering operationFloat, >0.0post-cluster-threshold: 0.5
epsEpsilon values for DBSCAN algorithmFloat, >0.0eps: 0.2
dbscan-min-scoreMinimum sum of confidence of all the neighbors in a cluster for it to be considered a valid cluster.Float, >0.0dbscan-min-score: 0.7

Gst Properties

The following table describes the Gst properties of the Gst-nvdspostprocess plugin.

Gst-nvspostprocess gst properties

PropertyMeaningType and RangeExample notes
gpu-idDevice ID of GPU to use for post-processing (dGPU only)Integer,0-4,294,967,295gpu-id=1
postprocesslib-nameLow level Post process library to be used for output parsingstringpostprocesslib-name=./postprocesslib_impl/libpostprocess_impl.so
postprocesslib-config-fileSet postprocess yaml config file to be usedstringpostprocesslib-config-file= config_detector.yml
preprocessor-supportEnables postprocessor parsing support for custom preprocessor tensorsBooleanpreprocessor-support=0

Sample pipelines

Given below are some sample pipelines, please set appropriate configuration file and library paths.

For multi-stream detector and classifier (dGPU):

gst-launch-1.0 uridecodebin \
 uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 \
! m.sink_0 nvstreammux name=m width=1920 height=1080 batch-size=2   ! \
 nvinfer config-file-path=config_infer_primary_post_process.txt  \
! nvdspostprocess \
postprocesslib-config-file=config_detector.yml \
postprocesslib-name= ./postprocesslib_impl/libpostprocess_impl.so \
! queue ! nvinfer \
config-file-path= config_infer_secondary_vehiclemake_postprocess.txt \
! nvdspostprocess \
postprocesslib-config-file= config_classifier_vehicle_make.yml \
postprocesslib-name= ./postprocesslib_impl/libpostprocess_impl.so ! queue ! nvinfer \
config-file-path= config_infer_secondary_vehicletypes_postprocess.txt \
! nvdspostprocess  postprocesslib-config-file= \
config_classifier_vehicle_type.yml  postprocesslib-name= \
./postprocesslib_impl/libpostprocess_impl.so    ! queue ! nvmultistreamtiler !  nvvideoconvert \
gpu-id=0 ! nvdsosd ! nveglglessink sync=1 -v  uridecodebin \
uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! m.sink_1

For multi-stream detector and classifier (Jetson):

gst-launch-1.0 uridecodebin \
 uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 \
! m.sink_0 nvstreammux name=m width=1920 height=1080 batch-size=2   ! \
nvinfer config-file-path=config_infer_primary_post_process.txt  \
! nvdspostprocess \
postprocesslib-config-file=config_detector.yml \
postprocesslib-name= ./postprocesslib_impl/libpostprocess_impl.so \
! queue ! nvinfer \
config-file-path= config_infer_secondary_vehiclemake_postprocess.txt \
! nvdspostprocess \
postprocesslib-config-file=config_classifier_vehicle_make.yml \
postprocesslib-name= ./postprocesslib_impl/libpostprocess_impl.so ! queue ! nvinfer \
config-file-path= config_infer_secondary_vehicletypes_postprocess.txt \
! nvdspostprocess  postprocesslib-config-file= \
config_classifier_vehicle_type.yml  postprocesslib-name= \
./postprocesslib_impl/libpostprocess_impl.so ! queue ! nvmultistreamtiler !  nvvideoconvert \
gpu-id=0 ! nvdsosd ! nv3dsink sync=1 -v  uridecodebin \
uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! m.sink_1