Gst-nvdspreprocess (Alpha) ========================== The Gst-nvdspreprocess plugin is a customizable plugin which provides a custom library interface for preprocessing on input streams. Each stream can have its own preprocessing requirements. (e.g. per stream ROIs - Region of Interests processing) Streams with same preprocessing requirements are grouped and processed together. These requirements can be specified via a config file. Refer to section :ref:`Gst-nvdspreprocess File Configuration Specifications` below for more details. Default plugin implementation provides two functionalities. 1. Streams with predefined ROIs (Region of Interests) are scaled and format converted as per the network requirements for inference. Per stream ROIs are specified in the config file. 2. It prepares a raw tensor from the scaled & converted ROIs. It is passed to the downstream plugins via user metadata. Downstream plugins can access this tensor for inference. The default custom library (nvdspreprocess_lib) provided with the plugin implements these functionalities. It contains the group transformation (scaling and conversion functionality per group) and tensor preparation functions for 2d convolution NCHW/NHWC models. User can implement its own library with group transformation functions and tensor preparation function as per the requirements. .. image:: /content/DS_plugin_gst-nvdspreprocess.png :align: center :alt: Gst-nvdspreprocess Inputs and Outputs ------------------- * Inputs * Input Video Gst Buffers * Metadata (NvDsBatchMeta) * Control parameters * config-file * gpu-id * Output * Output Video Gst Buffers * Metadata (NvDsBatchMeta) + User Metadata at batch level (NvDsPreProcessBatchMeta) Features --------- The following table summarizes the features of the plugin. .. csv-table:: Gst-nvdspreprocess features :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_features.csv :widths: 30, 30, 30 :header-rows: 1 Custom library Interfaces -------------------------- • custom_transform => An Interface to implement custom transformation for different group of streams. Default implementation provides group transformation functionalities of ROIs specified in the config fie. • custom_tensor_function => An interface to provide custom tensor preparation function. Default implementation prepares raw tensor from transformed ROIs .. _Gst-nvdspreprocess File Configuration Specifications: Gst-nvdspreprocess File Configuration Specifications ---------------------------------------------------- The Gst-nvdspreprocess configuration file uses a “Key File” format described in https://specifications.freedesktop.org/desktop-entry-spec/latest. Refer to ``config_preprocess.txt`` at ``/opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt``. The ``[property]`` group configures the general behavior of the plugin. The ``[group-]`` group configures ROIs/full-frames for a group of streams with `src-ids` and `custom-input-transformation-function` from custom lib. The ``[user-configs]`` group configures parameters required by the custom library, which is passed on to custom lib through a map of as key-value pair. custom lib needs to parse the values accordingly. The following two tables describes the keys supported for ``[property]`` groups and ``[group-]`` groups respectively. .. csv-table:: Gst-nvdspreprocess property Group Supported Keys :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_property_grp_keys.csv :widths: 20, 20, 20, 20 :header-rows: 1 .. csv-table:: Gst-nvdspreprocess group- Group Supported Keys :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_group_grp_keys.csv :widths: 20, 20, 20, 20 :header-rows: 1 The following table describes the keys supported for ``[user-configs]`` group for custom library ``nvdspreprocess_lib``. .. csv-table:: Gst-nvdspreprocess user-configs Group Supported Keys :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_user_configs_grp_keys.csv :widths: 20, 20, 20, 20 :header-rows: 1 The following table describes the keys supported for ``[user-configs]`` group for custom library ``libnvds_custom_sequence_preprocess.so``. which gets used in ``deepstream-3d-action-recognition`` sample app. .. csv-table:: user-configs properties for custom sequence preprocess :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_custom_sequence.csv :widths: 20, 20, 20, 20 :header-rows: 1 Gst Properties -------------- The following table describes the Gst properties of the Gst-nvdspreprocess plugin. .. csv-table:: Gst-nvspreprocess gst properties :file: ../text/tables/Gst-nvdspreprocess tables/DS_Plugin_gst-nvdspreprocess_gst_properties.csv :widths: 20, 20, 20, 20 :header-rows: 1 Sample pipelines ---------------- **For single stream (dGPU):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink **For single stream (Jetson):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink **For Multi-stream (dGPU):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink \ filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \ m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \ nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_3 **For Multi-stream (Jetson):** :: gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvvideoconvert ! \ nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream-6.0/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! \ nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt \ input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink \ filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! \ m.sink_1 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! \ nvv4l2decoder ! m.sink_2 filesrc location = /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! \ h264parse ! nvv4l2decoder ! m.sink_3