MetaData in the DeepStream SDK

Gst Buffer is the basic unit of data transfer in GStreamer. Each Gst Buffer has associated metadata. The DeepStream SDK attaches the DeepStream metadata object, NvDsBatchMeta, described in the following sections.
To read more about Gst Buffer, refer to GStreamer documentation.

NvDsBatchMeta: Basic Metadata Structure

DeepStream uses an extensible standard structure for metadata. The basic metadata structure NvDsBatchMeta starts with batch level metadata, created inside the Gst-nvstreammux plugin. Subsidiary metadata structures hold frame, object, classifier, and label data. DeepStream also provides a mechanism for adding user-specific metadata at the batch, frame, or object level.
DeepStream attaches metadata to a Gst Buffer by attaching an NvDsBatchMeta structure and setting GstNvDsMetaType.meta_type to NVDS_BATCH_GST_META in the Gst-nvstreammux plugin. When your application processes the Gst Buffer, it can iterate over the attached metadata to find NVDS_BATCH_GST_META.
The function gst_buffer_get_nvds_batch_meta() extracts NvDsBatchMeta from the Gst Buffer. (See the declaration in sources/include/gstnvdsmeta.h.) See the deepstream-test1 sample application for an example of this function’s usage. For more details, see NVIDIA DeepStream SDK API Reference.
A screenshot of a cell phone Description automatically generated

User/Custom Metadata Addition inside NvDsBatchMeta

To attach user-specific metadata at the batch, frame, or object level within NvDsBatchMeta, you must acquire an instance of NvDsUserMeta from the user meta pool by calling nvds_acquire_user_meta_from_pool(). (See sources/includes/nvdsmeta.h for details.) Then you must initialize NvDsUserMeta. The members you must set are user_meta_data, meta_type, copy_func, and release_func.
For more details, see the sample application source code in sources/apps/sample_apps/deepstream-user-metadata-test/deepstream_user_metadata_app.c.

Adding Custom Meta in Gst Plugins Upstream from Gst-nvstreammux

The DeepStream SDK creates batch level metadata in the Gst-nvstreammux plugin. It holds NvDsBatchMeta metadata in a hierarchy of batches, frames within batches, and objects within frames.

Adding metadata to the plugin before Gst-nvstreammux

This procedure introduces metadata to the DeepStream pipeline at a plugin before Gst-nvstreammux.
1. Set the plugin’s following members of the plugin’s NvDsMeta structure:
copy_func
free_func
meta_type
gst_to_nvds_meta_transform_func
gst_to_nvds_meta_release_func
2. Attach the metadata by calling gst_buffer_add_nvds_meta() and set the meta_type in the NvDsMeta instance returned by gst_buffer_add_nvds_meta().
3. The Gst-nvstreammux plugin transforms the input gst-meta created in step 2 from the Gst Buffer into an NvDsUserMeta object associated with the corresponding NvDsFrameMeta object. It adds this object to the frame_user_data list.
4. Search the frame_user_meta list in the NvDsFrameMeta object for the meta_type that was set in step 2, and access the attached metadata.
See the sample application source code in sources/apps/sample_apps/deepstream-gst-metadata-test/deepstream_gst_metadata.c for more details. If gst meta is not attached with gst_buffer_add_nvds_meta(), it is not transformed into DeepStream metadata. It is still available in the Gst Buffer.

Adding new metadata fields in DeepStream-5.0 GA release:

Prior to DeepStream 5.0 GA release, the bounding box (or bbox) coordinates were first generated by the detector component and overwritten by the tracker component with the updated/revised bbox coordinates if there is a tracker component in the pipeline. The value of 'confidence' field, however, kept the detector confidence value in the metadata, discarding the tracker confidence value even when there is a tracker component. It is because IOU and KLT trackers did not provide any confidence values. The NvDCF tracker introduced in DeepStream 4.0, can provide reliable tracker confidence values. This resulted in a significant change in storing bbox coordinates and confidence values in the metadata (i.e., NvDsObjectMeta) in DeepStream 5.0 GA release.
The NvDsObjectMeta structure in DeepStream 5.0 GA release has three bbox info and two confidence values:
1. detector_bbox_info - Holds bounding box parameters of the object when detected by detector.
2. tracker_bbox_info - Holds bounding box parameters of the object when processed by tracker.
3. rect_params - Holds bounding box coordinates of the object processed by the last component that updates it in the pipeline. These bounding box coordinates are clipped so that they do not fall outside frame boundary.
These bounding box parameters are used to overlay:
4. confidence - Holds a confidence value for the object, set by the inference component.
5. tracker_confidence - Holds a confidence value for the object set by NvDCF. Will be -0.1 in the case of IOU and KLT trackers.
As noted in the description above, the bbox coordinates from detector and tracker are stored separately in the newly introduced detector_bbox_info and tracker_bbox_info, respectively. The bbox info rect_params continues to hold the bbox info as in previous DeepStream versions, for backward compatibility. Note that rect_params will be deprecated in future release.
Preexisting confidence parameter holds the detector confidence value, while a newly introduced tracker_confidence parameter will hold the tracker confidence value.