Gst-nvdspreprocess (Alpha)

The Gst-nvdspreprocess plugin is a customizable plugin which provides a custom library interface for preprocessing on input streams. Each stream can have its own preprocessing requirements. (e.g. per stream ROIs - Region of Interests processing) Streams with same preprocessing requirements are grouped and processed together. These requirements can be specified via a config file. Refer to section Gst-nvdspreprocess File Configuration Specifications below for more details.

Default plugin implementation provides two functionalities.

  1. Streams with predefined ROIs (Region of Interests) are scaled and format converted as per the network requirements for inference. Per stream ROIs are specified in the config file.

  2. It prepares a raw tensor from the scaled & converted ROIs. It is passed to the downstream plugins via user metadata. Downstream plugins can access this tensor for inference.

The default custom library (nvdspreprocess_lib) provided with the plugin implements these functionalities. It contains the group transformation (scaling and conversion functionality per group) and tensor preparation functions for 2d convolution NCHW/NHWC models. User can implement its own library with group transformation functions and tensor preparation function as per the requirements.

This plugin can operate in two modes PGIE and SGIE. Preprocess with PGIE mode is used to process the given ROI/Frame on which we want to perform primary inferencing. Preprocess with SGIE mode is used to process the detected objects within the given ROI/Frame on which we want to perform secondary inferencing.

Gst-nvdspreprocess

Inputs and Outputs

  • Inputs

    • Input Video Gst Buffers

    • Metadata (NvDsBatchMeta)

  • Control parameters

    • config-file

  • Output

    • Output Video Gst Buffers

    • Metadata (NvDsBatchMeta) + User Metadata at batch level (NvDsPreProcessBatchMeta)

Features

The following table summarizes the features of the plugin.

Gst-nvdspreprocess features

Feature

Description

Release

Per stream/group of streams ROI processing

Process on the pre-defined ROIs having same processing algorithm inside a group

DS 6.0

Processing on full frames/ROIs

Enable process-on-roi inside [group-*] to process on ROIs

DS 6.0

Custom library interface

Custom functionality using group transformation and tensor preparation functions

DS 6.0

Custom group transformation function

For each group separate custom transformation functions can be provided from custom lib (e.g. scale & conversion)

DS 6.0

Custom tensor preparation function

Custom tensor preparation function from custom library to prepare raw tensor from already transformed ROIs/full-frames

DS 6.0

Attach user meta at batch level

User meta (NvDsPreProcessBatchMeta) having raw tensor and Scaled & converted ROIs

DS 6.0

Support for SGIE Mode

Supports processing on objects detected by PGIE model

DS 6.2

Custom library Interfaces

  • custom_transform => An Interface to implement custom transformation for different group of streams. Default implementation provides group transformation functionalities of ROIs specified in the config fie.

  • custom_tensor_function => An interface to provide custom tensor preparation function. Default implementation prepares raw tensor from transformed ROIs

Gst-nvdspreprocess File Configuration Specifications

The Gst-nvdspreprocess configuration file uses a “Key File” format described in https://specifications.freedesktop.org/desktop-entry-spec/latest. Refer to config_preprocess.txt at /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt. The [property] group configures the general behavior of the plugin. The [group-<id>] group configures ROIs/full-frames for a group of streams with src-ids and custom-input-transformation-function from custom lib. The [user-configs] group configures parameters required by the custom library, which is passed on to custom lib through a map of <string,string> as key-value pair. custom lib needs to parse the values accordingly.

The following two tables describes the keys supported for [property] groups and [group-<id>] groups respectively.

Gst-nvdspreprocess property Group Supported Keys

Property

Meaning

Type and Range

Example

enable

If set then to enable the plugin else in passthrough mode

Boolean

enable=1

unique-id

Uniquely identify the metadata generated by this element

Integer, | 0 to 4,294,967,295

unique-id=1

gpu-id

Device ID of GPU to use for pre-processing (dGPU only)

Integer,0-4,294,967,295

gpu-id=1

process-on-frame

Preprocessing Modes 1=PGIE Mode 0=SGIE Mode

Boolean

process-on-frame=1

target-unique-ids

List of component gie-id for which tensor is prepared

Semicolon delimited integer(gie-ids) array of values >=0

target-unique-ids=3;4;5

operate-on-gie-id

Unique gie-id whoes metadata is to be preprocessed (ignored if process-on-frame enabled)

Integer, 0 to 4,294,967,295

operate-on-gie-id=1

network-input-order

Order of the network input layer

Integer 0=NCHW 1=NHWC 2=CUSTOM

network-input-order=0

network-input-shape

Tensor shape of network input layer as per network-input-order

Semicolon delimited integer array of values > 0

network-input-shape=60;3;368;640

maintain-aspect-ratio

If set then maintain the aspect ratio while scaling

Boolean

maintain-aspect-ratio

symmetric-padding

If set then pad symmetrically (ignored if maintain-aspect-ratio disabled)

Boolean

symmetric-padding

processing-width

Width at which ROIs scaled

Integer>0

processing-width=640

processing-height

Height at which ROIs scaled

Integer>0

processing-height=368

scaling-buf-pool-size

Size of scaling buffer pool

Integer>0

scaling-buf-pool-size=6

tensor-buf-pool-size

Size of tensor buffer pool

Integer>0

tensor-buf-pool-size=6

network-color-format

Color format of model

Integer 0=RGB 1=BGR 2=GRAY

network-color-format=0

tensor-data-type

Data format for inference

Integer 0=FP32 1=UINT8 2=INT8 3=UINT32 4=INT32 5=FP16

tensor-data-type=0

tensor-name

Network input layer name

String

tensor-name=Input_1

scaling-pool-memory-type

Memory type for scaling buffer pool

Integer 0=NVBUF_MEM_DEFAULT 1=NVBUF_MEM_CUDA_PINNED 2=NVBUF_MEM_CUDA_DEVICE 3=NVBUF_MEM_CUDA_UNIFIED 4=NVBUF_MEM_SURFACE_ARRAY

scaling-pool-memory-type=2

scaling-pool-compute-hw

Type of computing hardware for scaling

Integer 0=NvBufSurfTransformCompute_Default 1=NvBufSurfTransformCompute_GPU 2=NvBufSurfTransformCompute_VIC

scaling-pool-compute-hw=0

scaling-filter

Scaling interpolation method

Integer 0=NvBufSurfTransformInter_Nearest 1=NvBufSurfTransformInter_Bilinear 2=NvBufSurfTransformInter_Algo 3=NvBufSurfTransformInter_Algo2 4=NvBufSurfTransformInter_Algo3 5=NvBufSurfTransformInter_Algo4 6=NvBufSurfTransformInter_Default

scaling-filter=0

custom-lib-path

Path of custom library .so file

String

custom-lib-path=/opt/nvidia/deepstream/deepstream/lib/gst-plugins/libcustom2d_preprocess.so

custom-tensor-preparation-function

Name of tensor preparation function from custom lib

String

custom-tensor-preparation-function=CustomTensorPreparation
Gst-nvdspreprocess group-<id> Group Supported Keys

Property

Meaning

Type and Range

Example

src-ids

Source IDs on which this group applies

Semicolon delimited integer array of values >=0

src-ids=0;1;2;3

custom-input-transformation-function

Name of Custom input transformation function from custom library

String

custom-input-transformation-function=CustomTransformation

process-on-roi

If enabled process on ROIs else on full-frames (ignored if process-on-frame disabled)

Boolean

process-on-roi=1

roi-params-src-<id>

Roi coordinates for source <id>. for each ROI specify left;top;width;height defining the ROI if process-on-roi enabled.

Semicolon delimited integer array of values >=0

roi-params-0=0;540;900;500;960;0;900;500

operate-on-class-ids

Preprocessing is done only on the objects belongs to these class-ids (ignored if process-on-frame enabled)

Semicolon delimited integer(class-ids) array of values >=0

operate-on-class-ids=0;1

process-on-all-objects

If enabled process all the objects else process the objects within the roi (ignored if process-on-frame enabled)

Boolean

process-on-all-objects=1

draw-roi

If enabled specified roi’s are drawn on display else roi’s are not drawn

Boolean

draw-roi=1

roi-color

Color of roi to be displayed

Semicolon delimited color components values R;G;B;A

roi_color=0;1;1;1

input-object-min-width

Minimum width of objects which are to be processed

Integer, ≥0

input-object-min-width=100

input-object-min-height

Minimum height of objects which are to be processed

Integer, ≥0

input-object-min-height=100

input-object-max-width

Maximum width of objects which are to be processed

Integer, ≥0

input-object-max-width=500

input-object-max-height

Maximum height of objects which are to be processed

Integer, ≥0

input-object-max-height=500

The following table describes the keys supported for [user-configs] group for custom library nvdspreprocess_lib.

Gst-nvdspreprocess user-configs Group Supported Keys

Property

Meaning

Type and Range

Example

pixel-normalization-factor

factor at which pixels are scaled

Float

pixel-normalization-factor=­0.031

offsets

Array of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by pixel-normalization-factor

Semicolon delimited float array of values ≥0

offsets=77.5;21.2;11.8

mean-file

pathname of mean data file (PPM format)

String

mean-file=/home/ubuntu/model_meanfile.ppm

The following table describes the keys supported for [user-configs] group for custom library libnvds_custom_sequence_preprocess.so. which gets used in deepstream-3d-action-recognition sample app.

user-configs properties for custom sequence preprocess

Property

Meaning

Type and Range

Example

channel-scale-factors

scale factor list for each channel

Semicolon delimited float array

channel-scale-factors= 0.007843137;0.007843137;0.007843137

channel-mean-offsets

data mean offsets for each channel

Semicolon delimited float array

channel-mean-offsets=127.5;127.5;127.5

stride

sequence sliding stride for each batched sequnece

Unsigned Integer, value >= 1

stride=1

subsample

Subsample rates for inference images in each sequence

Unsigned Integer, value >= 0

subsample=0

Gst Properties

The following table describes the Gst properties of the Gst-nvdspreprocess plugin.

Gst-nvspreprocess gst properties

Property

Meaning

Type and Range

Example notes

unique-id

Uniquely identify the metadata generated by this element

Integer, | 0 to 4,294,967,295

unique-id=1

gpu-id

Device ID of GPU to use for pre-processing (dGPU only)

Integer,0-4,294,967,295

gpu-id=1

config-file

Path of configuration file for the Gst-nvdspreprocess element

String

config-file=config_preprocess.txt

enable

Enable gst-nvdspreprocess plugin or set in passthrough mode

Boolean

enable=1

process-on-frame

Preprocessing Modes 1=PGIE Mode 0=SGIE Mode

Boolean

process-on-frame=1

target-unique-ids

List of component gie-id for which tensor is prepared

Semicolon delimited integer(gie-ids) array of values >=0

target-unique-ids=3;4;5

operate-on-gie-id

Unique gie-id whoes metadata is to be preprocessed (ignored if process-on-frame enabled)

Integer, 0 to 4,294,967,295

operate-on-gie-id=1

Sample pipelines

Given below are some sample pipelines to demonstrate preprocess plugin’s usage, please set the appropriate configuration file and library paths.

Preprocess in PGIE mode for single stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7  ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink

Preprocess in PGIE mode for single stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=2 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink

Preprocess in PGIE mode for Multi-stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink \
filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3

Preprocess in PGIE mode for Multi-stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink \
filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3

Preprocess in PGIE and SGIE Mode for single stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink

Preprocess in PGIE and SGIE Mode for single stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink

Preprocess in PGIE and SGIE Mode for Multi-stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3

Preprocess in PGIE and SGIE Mode for Multi-stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3