Skip to navigation
DeepStream Plugin GuidePlugins

Gst-nvdspreprocess

View as Markdown

The Gst-nvdspreprocess plugin is a customizable plugin which provides a custom library interface for preprocessing on input streams. Each stream can have its own preprocessing requirements. (e.g. per stream ROIs - Region of Interests processing) Streams with same preprocessing requirements are grouped and processed together. These requirements can be specified via a config file. Refer to section Gst-nvdspreprocess File Configuration Specifications below for more details.

Default plugin implementation provides two functionalities.

  1. Streams with predefined ROIs (Region of Interests) are scaled and format converted as per the network requirements for inference. Per stream ROIs are specified in the config file.

  2. It prepares a raw tensor from the scaled & converted ROIs. It is passed to the downstream plugins via user metadata. Downstream plugins can access this tensor for inference.

The default custom library (nvdspreprocess_lib) provided with the plugin implements these functionalities. It contains the group transformation (scaling and conversion functionality per group) and tensor preparation functions for 2d convolution NCHW/NHWC models. User can implement its own library with group transformation functions and tensor preparation function as per the requirements.

This plugin can operate in two modes PGIE and SGIE. Preprocess with PGIE mode is used to process the given ROI/Frame on which we want to perform primary inferencing. Preprocess with SGIE mode is used to process the detected objects within the given ROI/Frame on which we want to perform secondary inferencing.

Gst-nvdspreprocess

Inputs and Outputs

  • Inputs

    • Input Video Gst Buffers
    • Metadata (NvDsBatchMeta)

  • Control parameters

    • config-file

  • Output

    • Output Video Gst Buffers
    • Metadata (NvDsBatchMeta) + User Metadata at batch level (NvDsPreProcessBatchMeta)

Features

The following table summarizes the features of the plugin.

Gst-nvdspreprocess features

FeatureDescriptionRelease
Per stream/group of streams ROI processingProcess on the pre-defined ROIs having same processing algorithm inside a groupDS 6.0
Processing on full frames/ROIsEnable process-on-roi inside [group-*] to process on ROIsDS 6.0
Custom library interfaceCustom functionality using group transformation and tensor preparation functionsDS 6.0
Custom group transformation functionFor each group separate custom transformation functions can be provided from custom lib (e.g. scale & conversion)DS 6.0
Custom tensor preparation functionCustom tensor preparation function from custom library to prepare raw tensor from already transformed ROIs/full-framesDS 6.0
Attach user meta at batch levelUser meta (NvDsPreProcessBatchMeta) having raw tensor and Scaled & converted ROIsDS 6.0
Support for SGIE ModeSupports processing on objects detected by PGIE modelDS 6.2

Custom library Interfaces

  • custom_transform => An Interface to implement custom transformation for different group of streams. Default implementation provides group transformation functionalities of ROIs specified in the config fie.

  • custom_tensor_function => An interface to provide custom tensor preparation function. Default implementation prepares raw tensor from transformed ROIs

Gst-nvdspreprocess File Configuration Specifications

The Gst-nvdspreprocess configuration file uses a “Key File” format described in https://specifications.freedesktop.org/desktop-entry-spec/latest. Refer to config_preprocess.txt at /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt. The [property] group configures the general behavior of the plugin. The [group-<id>] group configures ROIs/full-frames for a group of streams with src-ids and custom-input-transformation-function from custom lib. The [user-configs] group configures parameters required by the custom library, which is passed on to custom lib through a map of <string,string> as key-value pair. custom lib needs to parse the values accordingly.

The following two tables describes the keys supported for [property] groups and [group-<id>] groups respectively.

Gst-nvdspreprocess property Group Supported Keys

PropertyMeaningType and RangeExample
enableIf set then to enable the plugin else in passthrough modeBooleanenable=1
unique-idUniquely identify the metadata generated by this elementInteger, | 0 to 4,294,967,295unique-id=1
gpu-idDevice ID of GPU to use for pre-processing (dGPU only)Integer,0-4,294,967,295gpu-id=1
process-on-framePreprocessing Modes 1=PGIE Mode 0=SGIE ModeBooleanprocess-on-frame=1
target-unique-idsList of component gie-id for which tensor is preparedSemicolon delimited integer(gie-ids) array of values >=0target-unique-ids=3;4;5
operate-on-gie-idUnique gie-id whose metadata is to be preprocessed (ignored if process-on-frame enabled)Integer, 0 to 4,294,967,295operate-on-gie-id=1
network-input-orderOrder of the network input layerInteger 0=NCHW 1=NHWC 2=CUSTOMnetwork-input-order=0
network-input-shapeTensor shape of network input layer as per network-input-orderSemicolon delimited integer array of values > 0network-input-shape=60;3;368;640
maintain-aspect-ratioIf set then maintain the aspect ratio while scalingBooleanmaintain-aspect-ratio
symmetric-paddingIf set then pad symmetrically (ignored if maintain-aspect-ratio disabled)Booleansymmetric-padding
processing-widthWidth at which ROIs scaledInteger>0processing-width=640
processing-heightHeight at which ROIs scaledInteger>0processing-height=368
scaling-buf-pool-sizeSize of scaling buffer poolInteger>0scaling-buf-pool-size=6
tensor-buf-pool-sizeSize of tensor buffer poolInteger>0tensor-buf-pool-size=6
network-color-formatColor format of modelInteger 0=RGB 1=BGR 2=GRAYnetwork-color-format=0
tensor-data-typeData format for inferenceInteger 0=FP32 1=UINT8 2=INT8 3=UINT32 4=INT32 5=INT64 6=UINT64 7=FP16tensor-data-type=0
tensor-nameNetwork input layer nameStringtensor-name=input_1:0
scaling-pool-memory-typeMemory type for scaling buffer poolInteger 0=NVBUF_MEM_DEFAULT 1=NVBUF_MEM_CUDA_PINNED 2=NVBUF_MEM_CUDA_DEVICE 3=NVBUF_MEM_CUDA_UNIFIED 4=NVBUF_MEM_SURFACE_ARRAYscaling-pool-memory-type=2
scaling-pool-compute-hwType of computing hardware for scalingInteger 0=NvBufSurfTransformCompute_Default 1=NvBufSurfTransformCompute_GPU 2=NvBufSurfTransformCompute_VICscaling-pool-compute-hw=0
scaling-filterScaling interpolation methodInteger 0=NvBufSurfTransformInter_Nearest 1=NvBufSurfTransformInter_Bilinear 2=NvBufSurfTransformInter_Algo 3=NvBufSurfTransformInter_Algo2 4=NvBufSurfTransformInter_Algo3 5=NvBufSurfTransformInter_Algo4 6=NvBufSurfTransformInter_Defaultscaling-filter=0
custom-lib-pathPath of custom library .so fileStringcustom-lib-path=/opt/nvidia/deepstream/deepstream/lib/gst-plugins/libcustom2d_preprocess.so
custom-tensor-preparation-functionName of tensor preparation function from custom libStringcustom-tensor-preparation-function=CustomTensorPreparation

Gst-nvdspreprocess group-<id> Group Supported Keys

PropertyMeaningType and RangeExample
src-idsSource IDs on which this group appliesSemicolon delimited integer array of values >=0src-ids=0;1;2;3
custom-input-transformation-functionName of Custom input transformation function from custom libraryStringcustom-input-transformation-function=CustomTransformation
process-on-roiIf enabled process on ROIs else on full-frames (ignored if process-on-frame disabled)Booleanprocess-on-roi=1
roi-params-src-<id>Roi coordinates for source <id>. for each ROI specify left;top;width;height defining the ROI if process-on-roi enabled.Semicolon delimited integer array of values >=0roi-params-0=0;540;900;500;960;0;900;500
operate-on-class-idsPreprocessing is done only on the objects belongs to these class-ids (ignored if process-on-frame enabled)Semicolon delimited integer(class-ids) array of values >=0operate-on-class-ids=0;1
process-on-all-objectsIf enabled process all the objects else process the objects within the roi (ignored if process-on-frame enabled)Booleanprocess-on-all-objects=1
draw-roiIf enabled specified roi’s are drawn on display else roi’s are not drawnBooleandraw-roi=1
roi-colorColor of roi to be displayedSemicolon delimited color components values R;G;B;Aroi_color=0;1;1;1
input-object-min-widthMinimum width of objects which are to be processedInteger, ≥0input-object-min-width=100
input-object-min-heightMinimum height of objects which are to be processedInteger, ≥0input-object-min-height=100
input-object-max-widthMaximum width of objects which are to be processedInteger, ≥0input-object-max-width=500
input-object-max-heightMaximum height of objects which are to be processedInteger, ≥0input-object-max-height=500

The following table describes the keys supported for [user-configs] group for custom library nvdspreprocess_lib.

Gst-nvdspreprocess user-configs Group Supported Keys

PropertyMeaningType and RangeExample
pixel-normalization-factorfactor at which pixels are scaledFloatpixel-normalization-factor=0.031
offsetsArray of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by pixel-normalization-factorSemicolon delimited float array of values ≥0offsets=77.5;21.2;11.8
mean-filepathname of mean data file (PPM format)Stringmean-file=/home/ubuntu/model_meanfile.ppm

The following table describes the keys supported for [user-configs] group for custom library libnvds_custom_sequence_preprocess.so. which gets used in deepstream-3d-action-recognition sample app.

user-configs properties for custom sequence preprocess

PropertyMeaningType and RangeExample
channel-scale-factorsscale factor list for each channelSemicolon delimited float arraychannel-scale-factors=
0.007843137;0.007843137;0.007843137
channel-mean-offsetsdata mean offsets for each channelSemicolon delimited float arraychannel-mean-offsets=127.5;127.5;127.5
stridesequence sliding stride for each batched sequenceUnsigned Integer, value >= 1stride=1
subsampleSubsample rates for inference images in each sequenceUnsigned Integer, value >= 0subsample=0

Gst Properties

The following table describes the Gst properties of the Gst-nvdspreprocess plugin.

Gst-nvspreprocess gst properties

PropertyMeaningType and RangeExample notes
unique-idUniquely identify the metadata generated by this elementInteger, | 0 to 4,294,967,295unique-id=1
gpu-idDevice ID of GPU to use for pre-processing (dGPU only)Integer,0-4,294,967,295gpu-id=1
config-filePath of configuration file for the Gst-nvdspreprocess elementStringconfig-file=config_preprocess.txt
enableEnable gst-nvdspreprocess plugin or set in passthrough modeBooleanenable=1
process-on-framePreprocessing Modes 1=PGIE Mode 0=SGIE ModeBooleanprocess-on-frame=1
target-unique-idsList of component gie-id for which tensor is preparedSemicolon delimited integer(gie-ids) array of values >=0target-unique-ids=3;4;5
operate-on-gie-idUnique gie-id whose metadata is to be preprocessed (ignored if process-on-frame enabled)Integer, 0 to 4,294,967,295operate-on-gie-id=1

Sample pipelines

Given below are some sample pipelines to demonstrate preprocess plugin’s usage, please set the appropriate configuration file and library paths.

Preprocess in PGIE mode for single stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7  ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink

Preprocess in PGIE mode for single stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=2 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink

Preprocess in PGIE mode for Multi-stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink \
filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3

Preprocess in PGIE mode for Multi-stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink \
filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3

Preprocess in PGIE and SGIE Mode for single stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink

Preprocess in PGIE and SGIE Mode for single stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink

Preprocess in PGIE and SGIE Mode for Multi-stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3

Preprocess in PGIE and SGIE Mode for Multi-stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3