Gst-nvdspreprocess (Alpha)

The Gst-nvdspreprocess plugin is a customizable plugin which provides a custom library interface for preprocessing on input streams. Each stream can have its own preprocessing requirements. (e.g. per stream ROIs - Region of Interests processing) Streams with same preprocessing requirements are grouped and processed together. These requirements can be specified via a config file. Refer to section Gst-nvdspreprocess File Configuration Specifications below for more details.

Default plugin implementation provides two functionalities.

  1. Streams with predefined ROIs (Region of Interests) are scaled and format converted as per the network requirements for inference. Per stream ROIs are specified in the config file.

  2. It prepares a raw tensor from the scaled & converted ROIs. It is passed to the downstream plugins via user metadata. Downstream plugins can access this tensor for inference.

The default custom library (nvdspreprocess_lib) provided with the plugin implements these functionalities. It contains the group transformation (scaling and conversion functionality per group) and tensor preparation functions for 2d convolution NCHW/NHWC models. User can implement its own library with group transformation functions and tensor preparation function as per the requirements.

Gst-nvdspreprocess

Inputs and Outputs

  • Inputs

    • Input Video Gst Buffers

    • Metadata (NvDsBatchMeta)

  • Control parameters

    • config-file

    • gpu-id

  • Output

    • Output Video Gst Buffers

    • Metadata (NvDsBatchMeta) + User Metadata at batch level (NvDsPreProcessBatchMeta)

Features

The following table summarizes the features of the plugin.

Gst-nvdspreprocess features

Feature

Description

Release

Per stream/group of streams ROI processing

process on the pre-defined ROIs having same processing algorithm inside a group

DS 6.0

processing on full frames/ROIs

enable process-on-rois inside [group-*] to process on ROIs

DS 6.0

custom library interface

custom functionality using group transformation and tensor preparation functions

DS 6.0

custom group transformation function

for each group separate custom transformation functions can be provided from custom lib (e.g. scale & conversion)

DS 6.0

custom tensor preparation function

custom tensor preparation function from custom library to prepare raw tensor from already transformed ROIs/full-frames

DS 6.0

attach user meta at batch level

user meta (NvDsPreProcessBatchMeta) having raw tensor and Scaled & converted ROIs

DS 6.0

Custom library Interfaces

  • custom_transform => An Interface to implement custom transformation for different group of streams. Default implementation provides group transformation functionalities of ROIs specified in the config fie.

  • custom_tensor_function => An interface to provide custom tensor preparation function. Default implementation prepares raw tensor from transformed ROIs

Gst-nvdspreprocess File Configuration Specifications

The Gst-nvdspreprocess configuration file uses a “Key File” format described in https://specifications.freedesktop.org/desktop-entry-spec/latest. Refer to config_preprocess.txt at /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt. The [property] group configures the general behavior of the plugin. The [group-<id>] group configures ROIs/full-frames for a group of streams with src-ids and custom-input-transformation-function from custom lib. The [user-configs] group configures parameters required by the custom library, which is passed on to custom lib through a map of <string,string> as key-value pair. custom lib needs to parse the values accordingly.

The following two tables describes the keys supported for [property] groups and [group-<id>] groups respectively.

Gst-nvdspreprocess property Group Supported Keys

Property

Meaning

Type and Range

Example

enable

if set then to enable the plugin else in passthrough mode

Boolean

enable=1

target-unique-ids

set of unique ids of target components

Semicolon delimited integer array of values >=0

target-unique-ids=1;2

network-input-order

order of the network input layer

Integer 0=NCHW 1=NHWC 2=CUSTOM

network-input-order=0

network-input-shape

tensor shape of network input layer as per network-input-order

Semicolon delimited integer array of values > 0

network-input-shape=60;3;368;640

processing-width

width at which ROIs scaled

Integer>0

processing-width=640

processing-height

height at which ROIs scaled

Integer>0

processing-height=368

scaling-buf-pool-size

size of scaling buffer pool

Integer>0

scaling-buf-pool-size=6

tensor-buf-pool-size

size of tensor buffer pool

Integer>0

tensor-buf-pool-size=6

network-color-format

color format of model

Integer 0=RGB 1=BGR 2=GRAY

network-color-format=0

tensor-data-type

data format for inference

Integer 0=FP32 1=UINT8 2=INT8 3=UINT32 4=INT32 5=FP16

tensor-data-type=0

tensor-name

network input layer name

String

tensor-name=Input_1

scaling-pool-memory-type

memory type for scaling buffer pool

Integer 0=NVBUF_MEM_DEFAULT 1=NVBUF_MEM_CUDA_PINNED 2=NVBUF_MEM_CUDA_DEVICE 3=NVBUF_MEM_CUDA_UNIFIED 4=NVBUF_MEM_SURFACE_ARRAY

scaling-pool-memory-type=2

scaling-pool-compute-hw

type of computing hardware for scaling

Integer 0=NvBufSurfTransformCompute_Default 1=NvBufSurfTransformCompute_GPU 2=NvBufSurfTransformCompute_VIC

scaling-pool-compute-hw=0

scaling-filter

scaling interpolation method

Integer 0=NvBufSurfTransformInter_Nearest 1=NvBufSurfTransformInter_Bilinear 2=NvBufSurfTransformInter_Algo 3=NvBufSurfTransformInter_Algo2 4=NvBufSurfTransformInter_Algo3 5=NvBufSurfTransformInter_Algo4 6=NvBufSurfTransformInter_Default

scaling-filter=0

custom-lib-path

path of custom library .so file

String

custom-lib-path=/opt/nvidia/deepstream/deepstream-6.0/lib/gst-plugins/libcustom2d_preprocess.so

custom-tensor-preparation-function

name of tensor preparation function from custom lib

String

custom-tensor-preparation-function=CustomTensorPreparation
Gst-nvdspreprocess group-<id> Group Supported Keys

Property

Meaning

Type and Range

Example

src-ids

Source IDs on which this group applies

Semicolon delimited integer array of values >=0

src-ids=0;1;2;3

custom-input-transformation-function

Name of Custom input transformation function from custom library

String

custom-input-transformation-function=CustomTransformation

process-on-rois

if enabled process on ROIs else on full-frames

Boolean

process-on-rois=1

roi-params-src-<id>

roi coordinates for source <id>. for each ROI specify left;top;width;height defining the ROI if process-on-rois enabled.

Semicolon delimited integer array of values >=0

roi-params-0=0;540;900;500;960;0;900;500

The following table describes the keys supported for [user-configs] group for custom library nvdspreprocess_lib.

Gst-nvdspreprocess user-configs Group Supported Keys

Property

Meaning

Type and Range

Example

pixel-normalization-factor

factor at which pixels are scaled

Float

pixel-normalization-factor=­0.031

offsets

Array of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by pixel-normalization-factor

Semicolon delimited float array of values ≥0

offsets=77.5;21.2;11.8

mean-file

pathname of mean data file (PPM format)

String

mean-file=/home/ubuntu/model_meanfile.ppm

The following table describes the keys supported for [user-configs] group for custom library libnvds_custom_sequence_preprocess.so. which gets used in deepstream-3d-action-recognition sample app.

user-configs properties for custom sequence preprocess

Property

Meaning

Type and Range

Example

channel-scale-factors

scale factor list for each channel

Semicolon delimited float array

channel-scale-factors= 0.007843137;0.007843137;0.007843137

channel-mean-offsets

data mean offsets for each channel

Semicolon delimited float array

channel-mean-offsets=127.5;127.5;127.5

stride

sequence sliding stride for each batched sequnece

Unsigned Integer, value >= 1

stride=1

subsample

Subsample rates for inference images in each sequence

Unsigned Integer, value >= 0

subsample=0

Gst Properties

The following table describes the Gst properties of the Gst-nvdspreprocess plugin.

Gst-nvspreprocess gst properties

Property

Meaning

Type and Range

Example notes

unique-id

Unique ID identifying metadata generated by this GIE

Integer, | 0 to 4,294,967,295

unique-id=1

gpu-id

Device ID of GPU to use for pre-processing (dGPU only)

Integer,0-4,294,967,295

gpu-id=1

config-file

path of configuration file for the Gst-nvdspreprocess element

string

config-file=config_preprocess.txt

enable

enable gst-nvdspreprocess plugin or set in passthrough mode

boolean

enable=1

Sample pipelines

For single stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7  ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink

For single stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7  ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink

For Multi-stream (dGPU):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7  ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink \
filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3

For Multi-stream (Jetson):

gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt  ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7  ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink \
filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3