Gst-nvdspreprocess (Alpha)#
The Gst-nvdspreprocess plugin is a customizable plugin which provides a custom library interface for preprocessing on input streams. Each stream can have its own preprocessing requirements. (e.g. per stream ROIs - Region of Interests processing) Streams with same preprocessing requirements are grouped and processed together. These requirements can be specified via a config file. Refer to section Gst-nvdspreprocess File Configuration Specifications below for more details.
Default plugin implementation provides two functionalities.
Streams with predefined ROIs (Region of Interests) are scaled and format converted as per the network requirements for inference. Per stream ROIs are specified in the config file.
It prepares a raw tensor from the scaled & converted ROIs. It is passed to the downstream plugins via user metadata. Downstream plugins can access this tensor for inference.
The default custom library (nvdspreprocess_lib) provided with the plugin implements these functionalities. It contains the group transformation (scaling and conversion functionality per group) and tensor preparation functions for 2d convolution NCHW/NHWC models. User can implement its own library with group transformation functions and tensor preparation function as per the requirements.
This plugin can operate in two modes PGIE and SGIE. Preprocess with PGIE mode is used to process the given ROI/Frame on which we want to perform primary inferencing. Preprocess with SGIE mode is used to process the detected objects within the given ROI/Frame on which we want to perform secondary inferencing.
Inputs and Outputs#
Inputs
Input Video Gst Buffers
Metadata (NvDsBatchMeta)
Control parameters
config-file
Output
Output Video Gst Buffers
Metadata (NvDsBatchMeta) + User Metadata at batch level (NvDsPreProcessBatchMeta)
Features#
The following table summarizes the features of the plugin.
# Feature
Description
Release
Per stream/group of streams ROI processing
Process on the pre-defined ROIs having same processing algorithm inside a group
DS 6.0
Processing on full frames/ROIs
Enable process-on-roi inside [group-*] to process on ROIs
DS 6.0
Custom library interface
Custom functionality using group transformation and tensor preparation functions
DS 6.0
Custom group transformation function
For each group separate custom transformation functions can be provided from custom lib (e.g. scale & conversion)
DS 6.0
Custom tensor preparation function
Custom tensor preparation function from custom library to prepare raw tensor from already transformed ROIs/full-frames
DS 6.0
Attach user meta at batch level
User meta (NvDsPreProcessBatchMeta) having raw tensor and Scaled & converted ROIs
DS 6.0
Support for SGIE Mode
Supports processing on objects detected by PGIE model
DS 6.2
Custom library Interfaces#
custom_transform => An Interface to implement custom transformation for different group of streams. Default implementation provides group transformation functionalities of ROIs specified in the config fie.
custom_tensor_function => An interface to provide custom tensor preparation function. Default implementation prepares raw tensor from transformed ROIs
Gst-nvdspreprocess File Configuration Specifications#
The Gst-nvdspreprocess configuration file uses a “Key File” format described in https://specifications.freedesktop.org/desktop-entry-spec/latest.
Refer to config_preprocess.txt
at /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt
.
The [property]
group configures the general behavior of the plugin.
The [group-<id>]
group configures ROIs/full-frames for a group of streams with src-ids and custom-input-transformation-function from custom lib.
The [user-configs]
group configures parameters required by the custom library, which is passed on to custom lib through a map of <string,string> as key-value pair.
custom lib needs to parse the values accordingly.
The following two tables describes the keys supported for [property]
groups and [group-<id>]
groups respectively.
# Property
Meaning
Type and Range
Example
enable
If set then to enable the plugin else in passthrough mode
Boolean
enable=1
unique-id
Uniquely identify the metadata generated by this element
Integer, | 0 to 4,294,967,295
unique-id=1
gpu-id
Device ID of GPU to use for pre-processing (dGPU only)
Integer,0-4,294,967,295
gpu-id=1
process-on-frame
Preprocessing Modes 1=PGIE Mode 0=SGIE Mode
Boolean
process-on-frame=1
target-unique-ids
List of component gie-id for which tensor is prepared
Semicolon delimited integer(gie-ids) array of values >=0
target-unique-ids=3;4;5
operate-on-gie-id
Unique gie-id whoes metadata is to be preprocessed (ignored if process-on-frame enabled)
Integer, 0 to 4,294,967,295
operate-on-gie-id=1
network-input-order
Order of the network input layer
Integer 0=NCHW 1=NHWC 2=CUSTOM
network-input-order=0
network-input-shape
Tensor shape of network input layer as per network-input-order
Semicolon delimited integer array of values > 0
network-input-shape=60;3;368;640
maintain-aspect-ratio
If set then maintain the aspect ratio while scaling
Boolean
maintain-aspect-ratio
symmetric-padding
If set then pad symmetrically (ignored if maintain-aspect-ratio disabled)
Boolean
symmetric-padding
processing-width
Width at which ROIs scaled
Integer>0
processing-width=640
processing-height
Height at which ROIs scaled
Integer>0
processing-height=368
scaling-buf-pool-size
Size of scaling buffer pool
Integer>0
scaling-buf-pool-size=6
tensor-buf-pool-size
Size of tensor buffer pool
Integer>0
tensor-buf-pool-size=6
network-color-format
Color format of model
Integer 0=RGB 1=BGR 2=GRAY
network-color-format=0
tensor-data-type
Data format for inference
Integer 0=FP32 1=UINT8 2=INT8 3=UINT32 4=INT32 5=FP16
tensor-data-type=0
tensor-name
Network input layer name
String
tensor-name=Input_1
scaling-pool-memory-type
Memory type for scaling buffer pool
Integer 0=NVBUF_MEM_DEFAULT 1=NVBUF_MEM_CUDA_PINNED 2=NVBUF_MEM_CUDA_DEVICE 3=NVBUF_MEM_CUDA_UNIFIED 4=NVBUF_MEM_SURFACE_ARRAY
scaling-pool-memory-type=2
scaling-pool-compute-hw
Type of computing hardware for scaling
Integer 0=NvBufSurfTransformCompute_Default 1=NvBufSurfTransformCompute_GPU 2=NvBufSurfTransformCompute_VIC
scaling-pool-compute-hw=0
scaling-filter
Scaling interpolation method
Integer 0=NvBufSurfTransformInter_Nearest 1=NvBufSurfTransformInter_Bilinear 2=NvBufSurfTransformInter_Algo 3=NvBufSurfTransformInter_Algo2 4=NvBufSurfTransformInter_Algo3 5=NvBufSurfTransformInter_Algo4 6=NvBufSurfTransformInter_Default
scaling-filter=0
custom-lib-path
Path of custom library .so file
String
custom-lib-path=/opt/nvidia/deepstream/deepstream/lib/gst-plugins/libcustom2d_preprocess.so
custom-tensor-preparation-function
Name of tensor preparation function from custom lib
String
custom-tensor-preparation-function=CustomTensorPreparation
# Property
Meaning
Type and Range
Example
src-ids
Source IDs on which this group applies
Semicolon delimited integer array of values >=0
src-ids=0;1;2;3
custom-input-transformation-function
Name of Custom input transformation function from custom library
String
custom-input-transformation-function=CustomTransformation
process-on-roi
If enabled process on ROIs else on full-frames (ignored if process-on-frame disabled)
Boolean
process-on-roi=1
roi-params-src-<id>
Roi coordinates for source <id>. for each ROI specify left;top;width;height defining the ROI if process-on-roi enabled.
Semicolon delimited integer array of values >=0
roi-params-0=0;540;900;500;960;0;900;500
operate-on-class-ids
Preprocessing is done only on the objects belongs to these class-ids (ignored if process-on-frame enabled)
Semicolon delimited integer(class-ids) array of values >=0
operate-on-class-ids=0;1
process-on-all-objects
If enabled process all the objects else process the objects within the roi (ignored if process-on-frame enabled)
Boolean
process-on-all-objects=1
draw-roi
If enabled specified roi’s are drawn on display else roi’s are not drawn
Boolean
draw-roi=1
roi-color
Color of roi to be displayed
Semicolon delimited color components values R;G;B;A
roi_color=0;1;1;1
input-object-min-width
Minimum width of objects which are to be processed
Integer, ≥0
input-object-min-width=100
input-object-min-height
Minimum height of objects which are to be processed
Integer, ≥0
input-object-min-height=100
input-object-max-width
Maximum width of objects which are to be processed
Integer, ≥0
input-object-max-width=500
input-object-max-height
Maximum height of objects which are to be processed
Integer, ≥0
input-object-max-height=500
The following table describes the keys supported for [user-configs]
group for custom library nvdspreprocess_lib
.
# Property
Meaning
Type and Range
Example
pixel-normalization-factor
factor at which pixels are scaled
Float
pixel-normalization-factor=0.031
offsets
Array of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by pixel-normalization-factor
Semicolon delimited float array of values ≥0
offsets=77.5;21.2;11.8
mean-file
pathname of mean data file (PPM format)
String
mean-file=/home/ubuntu/model_meanfile.ppm
The following table describes the keys supported for [user-configs]
group for custom library libnvds_custom_sequence_preprocess.so
. which gets used in deepstream-3d-action-recognition
sample app.
Property |
Meaning |
Type and Range |
Example |
---|---|---|---|
channel-scale-factors |
scale factor list for each channel |
Semicolon delimited float array |
channel-scale-factors= 0.007843137;0.007843137;0.007843137 |
channel-mean-offsets |
data mean offsets for each channel |
Semicolon delimited float array |
channel-mean-offsets=127.5;127.5;127.5 |
stride |
sequence sliding stride for each batched sequnece |
Unsigned Integer, value >= 1 |
stride=1 |
subsample |
Subsample rates for inference images in each sequence |
Unsigned Integer, value >= 0 |
subsample=0 |
Gst Properties#
The following table describes the Gst properties of the Gst-nvdspreprocess plugin.
# Property
Meaning
Type and Range
Example notes
unique-id
Uniquely identify the metadata generated by this element
Integer, | 0 to 4,294,967,295
unique-id=1
gpu-id
Device ID of GPU to use for pre-processing (dGPU only)
Integer,0-4,294,967,295
gpu-id=1
config-file
Path of configuration file for the Gst-nvdspreprocess element
String
config-file=config_preprocess.txt
enable
Enable gst-nvdspreprocess plugin or set in passthrough mode
Boolean
enable=1
process-on-frame
Preprocessing Modes 1=PGIE Mode 0=SGIE Mode
Boolean
process-on-frame=1
target-unique-ids
List of component gie-id for which tensor is prepared
Semicolon delimited integer(gie-ids) array of values >=0
target-unique-ids=3;4;5
operate-on-gie-id
Unique gie-id whoes metadata is to be preprocessed (ignored if process-on-frame enabled)
Integer, 0 to 4,294,967,295
operate-on-gie-id=1
Sample pipelines#
Given below are some sample pipelines to demonstrate preprocess plugin’s usage, please set the appropriate configuration file and library paths.
Preprocess in PGIE mode for single stream (dGPU):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink
Preprocess in PGIE mode for single stream (Jetson):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=2 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink
Preprocess in PGIE mode for Multi-stream (dGPU):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink \
filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3
Preprocess in PGIE mode for Multi-stream (Jetson):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \
input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink \
filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \
nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_3
Preprocess in PGIE and SGIE Mode for single stream (dGPU):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink
Preprocess in PGIE and SGIE Mode for single stream (Jetson):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink
Preprocess in PGIE and SGIE Mode for Multi-stream (dGPU):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3
Preprocess in PGIE and SGIE Mode for Multi-stream (Jetson):
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \
h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \
nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \
batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \
nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt input-tensor-meta=1 unique-id=3 ! \
nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \
m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3