Gst-nvdspreprocess (Alpha) ========================== The Gst-nvdspreprocess plugin is a customizable plugin which provides a custom library interface for preprocessing on input streams. Each stream can have its own preprocessing requirements. (e.g. per stream ROIs - Region of Interests processing) Streams with same preprocessing requirements are grouped and processed together. These requirements can be specified via a config file. Refer to section :ref:`Gst-nvdspreprocess File Configuration Specifications` below for more details. Default plugin implementation provides two functionalities. 1. Streams with predefined ROIs (Region of Interests) are scaled and format converted as per the network requirements for inference. Per stream ROIs are specified in the config file. 2. It prepares a raw tensor from the scaled & converted ROIs. It is passed to the downstream plugins via user metadata. Downstream plugins can access this tensor for inference. The default custom library (nvdspreprocess_lib) provided with the plugin implements these functionalities. It contains the group transformation (scaling and conversion functionality per group) and tensor preparation functions for 2d convolution NCHW/NHWC models. User can implement its own library with group transformation functions and tensor preparation function as per the requirements. This plugin can operate in two modes PGIE and SGIE. Preprocess with PGIE mode is used to process the given ROI/Frame on which we want to perform primary inferencing. Preprocess with SGIE mode is used to process the detected objects within the given ROI/Frame on which we want to perform secondary inferencing. .. image:: /content/DS_plugin_gst-nvdspreprocess.png :align: center :alt: Gst-nvdspreprocess Inputs and Outputs ------------------- * Inputs * Input Video Gst Buffers * Metadata (NvDsBatchMeta) * Control parameters * config-file * Output * Output Video Gst Buffers * Metadata (NvDsBatchMeta) + User Metadata at batch level (NvDsPreProcessBatchMeta) Features --------- The following table summarizes the features of the plugin. .. csv-table:: Gst-nvdspreprocess features :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_features.csv :widths: 30, 30, 30 :header-rows: 1 Custom library Interfaces -------------------------- * custom_transform => An Interface to implement custom transformation for different group of streams. Default implementation provides group transformation functionalities of ROIs specified in the config fie. * custom_tensor_function => An interface to provide custom tensor preparation function. Default implementation prepares raw tensor from transformed ROIs .. _Gst-nvdspreprocess File Configuration Specifications: Gst-nvdspreprocess File Configuration Specifications ---------------------------------------------------- The Gst-nvdspreprocess configuration file uses a “Key File” format described in https://specifications.freedesktop.org/desktop-entry-spec/latest. Refer to ``config_preprocess.txt`` at ``/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt``. The ``[property]`` group configures the general behavior of the plugin. The ``[group-]`` group configures ROIs/full-frames for a group of streams with `src-ids` and `custom-input-transformation-function` from custom lib. The ``[user-configs]`` group configures parameters required by the custom library, which is passed on to custom lib through a map of as key-value pair. custom lib needs to parse the values accordingly. The following two tables describes the keys supported for ``[property]`` groups and ``[group-]`` groups respectively. .. csv-table:: Gst-nvdspreprocess property Group Supported Keys :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_property_grp_keys.csv :widths: 20, 20, 20, 20 :header-rows: 1 .. csv-table:: Gst-nvdspreprocess group- Group Supported Keys :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_group_grp_keys.csv :widths: 20, 20, 20, 20 :header-rows: 1 The following table describes the keys supported for ``[user-configs]`` group for custom library ``nvdspreprocess_lib``. .. csv-table:: Gst-nvdspreprocess user-configs Group Supported Keys :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_user_configs_grp_keys.csv :widths: 20, 20, 20, 20 :header-rows: 1 The following table describes the keys supported for ``[user-configs]`` group for custom library ``libnvds_custom_sequence_preprocess.so``. which gets used in ``deepstream-3d-action-recognition`` sample app. .. csv-table:: user-configs properties for custom sequence preprocess :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_custom_sequence.csv :widths: 20, 20, 20, 20 :header-rows: 1 Gst Properties -------------- The following table describes the Gst properties of the Gst-nvdspreprocess plugin. .. csv-table:: Gst-nvspreprocess gst properties :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_gst_properties.csv :widths: 20, 20, 20, 20 :header-rows: 1 Sample pipelines ---------------- Given below are some sample pipelines to demonstrate preprocess plugin’s usage, please set the appropriate configuration file and library paths. **Preprocess in PGIE mode for single stream (dGPU):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink **Preprocess in PGIE mode for single stream (Jetson):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=2 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink **Preprocess in PGIE mode for Multi-stream (dGPU):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink \ filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \ m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \ nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_3 **Preprocess in PGIE mode for Multi-stream (Jetson):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=8 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink \ filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \ m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \ nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_3 **Preprocess in PGIE and SGIE Mode for single stream (dGPU):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \ batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt input-tensor-meta=1 unique-id=3 ! \ nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink **Preprocess in PGIE and SGIE Mode for single stream (Jetson):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \ batch-size=2 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt input-tensor-meta=1 unique-id=3 ! \ nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink **Preprocess in PGIE and SGIE Mode for Multi-stream (dGPU):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \ batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt input-tensor-meta=1 unique-id=3 ! \ nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! \ qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \ m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3 **Preprocess in PGIE and SGIE Mode for Multi-stream (Jetson):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt unique-id=1 \ batch-size=8 input-tensor-meta=1 ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess_sgie.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt input-tensor-meta=1 unique-id=3 ! \ nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nv3dsink filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! \ qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \ m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream-6.2/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_3