Gst-nvdsmetautils ========================= gst-nvdsmetautils contain nvdsmetainsert and nvdsmetaextract plugins. Below is the detailed explanation of each plugin, purpose and use-cases. Gst-nvdsmetainsert ------------------- The Gst-nvdsmetainsert plugin is a NvDsMeta utils plugin which provides a custom library hooking interface for single/batched audio/video frame(s) processing. Custom lib implementation can have algorithms to serialize NvDsBatchmeta or custom gst user meta received on the input buffers depending upon the use case. The plugin accepts batched audio/video NVMM HW buffers, and outputs batched audio/video NVMM HW buffers with serialized metadata on the buffer as per the use case. .. image:: /content/DS_plugin_gst-nvdsmetainsert.png :align: center :alt: Gst-nvdsmetainsert Inputs and Outputs ----------------------- * Inputs * NVMM Gst Buffer. * Control parameters * serialize-lib * Output * NVMM Gst Buffer in place. Features ----------- The following table summarizes the features of the plugin. .. csv-table:: Gst-nvdsmetainsert features :file: ../text/tables/Gst-nvdsmetainsert tables/DS_Plugin_gst-nvdsmetainsert_features.csv :widths: 30, 30, 30 :header-rows: 1 customlib_impl Interfaces --------------------------- * serialize_data => Accepts a GstBuffer, serializes input meta and attaches it as NvDsUserMeta / gst meta on the same GstBuffer. Gst Properties ---------------- The following table describes the Gst properties of the Gst-nvdsmetainsert plugin. .. csv-table:: Gst-nvdsmetainsert gst properties :file: ../text/tables/Gst-nvdsmetainsert tables/DS_Plugin_gst-nvdsmetainsert_gst-properties.csv :widths: 20, 20, 20, 20, 20 :header-rows: 1 Use Cases & Sample Pipelines ----------------------------- Below are two reference implementations, these can be extended to suit any particular use case. **Use Case 1 - Attaching CustomMeta as SEI data in H264 :** nvdsmetainsert plugin with sei_serialization library released as part of DS SDK 6.1 onwards, serializes input metadata and attaches it on the same buffer as gst meta, nv encoder embeds this metadata in the bitstream with type as SEI unregistered data. Below pipeline demonstrates generation of dummy bounding boxes by nvdsvideotemplate plugin and this metadata is serialized by nvdsmetainsert plugin with serialization lib. Serialized metadata is used by nv encoder to insert it as SEI unregistered data in H264 bitstream. :: export USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 filesrc location=~/sample_720p.mp4 ! qtdemux ! h264parse \ ! nvv4l2decoder ! m.sink_0 nvstreammux name=m ! nvdsvideotemplate dummy-meta-insert=1 \ customlib-name=libcustom_videoimpl.so customlib-props="scale-factor:0.5" \ ! nvmultistreamtiler width=640 height=480 ! nvdsmetainsert serialize-lib = "libnvds_sei_serialization.so" \ ! nvv4l2h264enc bitrate=1000000 ! filesink location=~/ds_maxine_sei.h264 \ filesrc location=~/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 **Use Case 2 - Video NvDsBatchMeta Serialization :** nvdsmetainsert plugin with nvds_video_meta_serialization library released as part of DS SDK 6.1 onwards, serializes NvDsFrameMeta and NvDsObjectMeta and attaches as NvDsUserMeta inside NvDsFrameMeta on the same buffer. Below pipeline demonstrates serialization of NvDsFrameMeta and NvDsObjectMeta by nvdsmetainsert plugin with serialization lib. Serialized metadata is used by nvdsmetaextract plugin to de-serialize and re-construct NvDsFrameMeta and NvDsObjectMeta. Setting CLEAR_NVDS_BATCH_META=yes is needed only for the demonstration of below pipeline, in actual use-case scenario like RDMA transfer this flag is not needed. :: export CLEAR_NVDS_BATCH_META=yes export USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 filesrc location=~/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder \ ! m.sink_0 nvstreammux name=m ! nvvideoconvert ! nvinfer config-file-path= config_infer_primary.txt \ ! nvdsmetainsert serialize-lib = "libnvds_video_metadata_serialization.so" \ ! nvdsmetaextract deserialize-lib = "libnvds_video_metadata_serialization.so" \ ! nvmultistreamtiler width=1280 height=720 ! nvdsosd ! nveglglessink filesrc location=~/sample_720p.mp4 \ ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 **Use Case 3 - Audio NvDsBatchMeta Serialization :** nvdsmetainsert plugin with nvds_audio_meta_serialization library released as part of DS SDK 6.1 onwards, serializes NvDsAudioFrameMeta and attaches as NvDsUserMeta inside NvDsAudioFrameMeta on the same buffer. Below pipeline demonstrates serialization of NvDsAudioFrameMeta by nvdsmetainsert plugin with serializaiton lib. Serialized metadata is used by nvdsmetaextract plugin to de-serialize and re-construct NvDsAudioFrameMeta. Setting CLEAR_NVDS_BATCH_META=yes is needed only for the demonstration of below pipeline, in actual use-case scenario like RDMA transfer this flag is not needed. :: export CLEAR_NVDS_BATCH_META=yes export USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 \ filesrc location= ~/noise_44_1k_16bits.wav ! wavparse \ ! audioconvert ! audioresample ! "audio/x-raw, rate=(int)44100" ! queue \ ! m.sink_0 nvstreammux name=m batch-size=2 ! queue \ ! nvinferaudio audio-framesize=44100 audio-hopsize=11025 batch-size=2 \ config-file-path= config_infer_audio_sonyc.txt \ audio-transform="melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100,p2db_ref=(float)1.0,p2db_min_power=(float)0.0,p2db_top_db=(float)80.0" \ ! nvdsmetainsert serialize-lib = "libnvds_audio_metadata_serialization.so" \ ! nvdsmetaextract deserialize-lib = "libnvds_audio_metadata_serialization.so" \ ! fakesink \ filesrc location= ~/noise_44_1k_16bits.wav ! wavparse \ ! audioconvert ! audioresample ! "audio/x-raw, rate=(int)44100" ! m.sink_1 Gst-nvdsmetaextract --------------------- The Gst-nvdsmetaextract plugin is a NvDsMeta utils plugin which provides a custom library hooking interface for single/batched audio/video frame(s) processing. Custom lib implementation can have algorithms to de-serialize NvDsBatchmeta or custom gst user meta received on the input buffer depending upon the use case. The plugin accepts batched/non-batched audio/video NVMM HW buffers, and outputs batched/non-batched audio/video NVMM HW buffers with de-serialized meatdata on the buffer as per the use case. .. image:: /content/DS_plugin_gst-nvdsmetaextract.png :align: center :alt: Gst-nvdsmetaextract Inputs and Outputs ----------------------- * Inputs * NVMM Gst Buffer. * Control parameters * deserialize-lib * Output * NVMM Gst Buffer in place. Features ----------- The following table summarizes the features of the plugin. .. csv-table:: Gst-nvdsmetaextract features :file: ../text/tables/Gst-nvdsmetaextract tables/DS_Plugin_gst-nvdsmetaextract_features.csv :widths: 30, 30, 30 :header-rows: 1 customlib_impl Interfaces --------------------------- * deserialize_data => Accepts a GstBuffer, de-serializes input meta and attaches as gst meta / update existing NvDsBatchMeta on the same GstBuffer. Gst Properties ---------------- The following table describes the Gst properties of the Gst-nvdsmetaextract plugin. .. csv-table:: Gst-nvdsmetaextract gst properties :file: ../text/tables/Gst-nvdsmetaextract tables/DS_Plugin_gst-nvdsmetaextract_gst-properties.csv :widths: 20, 20, 20, 20, 20 :header-rows: 1 Use Cases & Sample Pipelines ----------------------------- Below are two reference implementations, these can be extended to suit any particular use case. **Use Case 1 : CustomMeta De-Serialization SEI in H264 :** Nv decoder sends parsed SEI data of unregistered type which is serialized metadata inside bitstream, nvdsmetaextract plugin with sei_serialization library released as part of DS SDK 6.1 onwards, de-serializes this metadata and add it on the outgoing buffer as gst metadata, this data can be consumed by downstream plugin as per the use-case. Below pipeline demonstrates parsing serialized SEI data in the bitstream, attaching it as Gst meta on the outgoing buffer and de-serializing it in nvdsmetaexract plugin:: gst-launch-1.0 filesrc location=~/ds_maxine_sei.h264 ! h264parse ! nvv4l2decoder extract-sei-type5-data=1 \ ! nvdsmetaextract deserialize-lib=libnvds_sei_serialization.so ! nveglglessink **Use Case 2 : Video NvDsBatchMeta De-Serialization :** Upstream component sends GstBuffer with serialized NvDsFrameMeta & NvDsObjectMeta attached as NvDsUserMeta inside NvDsFrameMeta, nvdsmetaextract plugin with nvds_video_meta_serialization library released as part of DS SDK 6.1 onwards, de-serializes this NvDsUserMeta and re-constructs NvDsFrameMeta and NvDsObjectMeta as part of NvDsBatchMeta. Below pipeline demonstrates serialization of NvDsFrameMeta and NvDsObjectMeta by nvdsmetainsert plugin with serialization lib. Serialized metadata is used by nvdsmetaextract plugin to de-serialize and re-construct NvDsFrameMeta and NvDsObjectMeta. Setting CLEAR_NVDS_BATCH_META=yes is needed only for the demonstration of below pipeline, in actual use-case scenario like RDMA transfer this flag is not needed. :: export CLEAR_NVDS_BATCH_META=yes export USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 filesrc location=~/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder \ ! m.sink_0 nvstreammux name=m ! nvvideoconvert ! nvinfer config-file-path= config_infer_primary.txt \ ! nvdsmetainsert serialize-lib = "libnvds_video_metadata_serialization.so" \ ! nvdsmetaextract deserialize-lib = "libnvds_video_metadata_serialization.so" \ ! nvmultistreamtiler width=1280 height=720 ! nvdsosd ! nveglglessink filesrc location=~/sample_720p.mp4 \ ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1 **Use Case 3 - Audio NvDsBatchMeta Serialization :** Upstream component sends GstBuffer with serialized NvDsAudioFrameMeta attached as NvDsUserMeta inside NvDsAudioFrameMeta, nvdsmetaextract plugin with nvds_audio_meta_serialization library released as part of DS SDK 6.1 onwards, de-serializes this NvDsUserMeta and re-constructs NvDsAudioFrameMeta as part of NvDsBatchMeta Below pipeline demonstrates serialization of NvDsAudioFrameMeta by nvdsmetainsert plugin with serializaiton lib. Serialized metadata is used by nvdsmetaextract plugin to de-serialize and re-construct NvDsAudioFrameMeta. Setting CLEAR_NVDS_BATCH_META=yes is needed only for the demonstration of below pipeline, in actual use-case scenario like RDMA transfer this flag is not needed. :: export CLEAR_NVDS_BATCH_META=yes export USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 \ filesrc location= ~/noise_44_1k_16bits.wav ! wavparse \ ! audioconvert ! audioresample ! "audio/x-raw, rate=(int)44100" ! queue \ ! m.sink_0 nvstreammux name=m batch-size=2 ! queue \ ! nvinferaudio audio-framesize=44100 audio-hopsize=11025 batch-size=2 \ config-file-path= config_infer_audio_sonyc.txt \ audio-transform="melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100,p2db_ref=(float)1.0,p2db_min_power=(float)0.0,p2db_top_db=(float)80.0" \ ! nvdsmetainsert serialize-lib = "libnvds_audio_metadata_serialization.so" \ ! nvdsmetaextract deserialize-lib = "libnvds_audio_metadata_serialization.so" \ ! fakesink \ filesrc location= ~/noise_44_1k_16bits.wav ! wavparse \ ! audioconvert ! audioresample ! "audio/x-raw, rate=(int)44100" ! m.sink_1