MetaData in the DeepStream SDK
Gst Buffer is the basic unit of data transfer in GStreamer. Each Gst Buffer has associated metadata. The DeepStream SDK attaches the DeepStream metadata object, NvDsBatchMeta, described in the following sections. To read more about Gst Buffer, refer to GStreamer documentation.
NvDsBatchMeta: Basic Metadata Structure
DeepStream uses an extensible standard structure for metadata. The basic metadata structure
NvDsBatchMeta starts with batch level metadata, created inside the Gst-nvstreammux plugin. Subsidiary metadata structures hold frame, object, classifier, and label data. DeepStream also provides a mechanism for adding user-specific metadata at the batch, frame, or object level.
DeepStream attaches metadata to a Gst Buffer by attaching an
NvDsBatchMeta structure and setting
NVDS_BATCH_GST_META in the Gst-nvstreammux plugin. When your application processes the Gst Buffer, it can iterate over the attached metadata to find NVDS_BATCH_GST_META.
NvDsBatchMeta from the Gst Buffer. (See the declaration in
sources/include/gstnvdsmeta.h.) See the deepstream-test1 sample application for an example of this function’s usage. For more details, see the DeepStream SDK API reference documentation in DeepStream API Guides.
User/Custom Metadata Addition inside NvDsBatchMeta
To attach user-specific metadata at the batch, frame, or object level within
NvDsBatchMeta, you must acquire an instance of
NvDsUserMeta from the user meta pool by calling
sources/includes/nvdsmeta.h for details.) Then you must initialize
NvDsUserMeta. The members you must set are
For more details, see the sample application source code in
Adding Custom Meta in Gst Plugins Upstream from Gst-nvstreammux
The DeepStream SDK creates batch level metadata in the Gst-nvstreammux plugin. It holds
NvDsBatchMeta metadata in a hierarchy of batches, frames within batches, and objects within frames.
Adding metadata to the plugin before Gst-nvstreammux
This procedure introduces metadata to the DeepStream pipeline at a plugin before Gst-nvstreammux.
Set the plugin’s following members of the plugin’s NvDsMeta structure:
Attach the metadata by calling
gst_buffer_add_nvds_meta()and set the
meta_typein the NvDsMeta instance returned by
The Gst-nvstreammux plugin transforms the input gst-meta created in step 2 from the Gst Buffer into an
NvDsUserMetaobject associated with the corresponding NvDsFrameMeta object. It adds this object to the
frame_user_meta_listin the NvDsFrameMeta object for the
meta_typethat was set in step 2, and access the attached metadata.
See the sample application source code in
sources/apps/sample_apps/deepstream-gst-metadata-test/deepstream_gst_metadata.c for more details. If gst meta is not attached with
gst_buffer_add_nvds_meta(), it is not transformed into DeepStream metadata. It is still available in the Gst Buffer.
New metadata fields
NvDsObjectMeta structure from DeepStream 5.0 GA release has three bbox info and two confidence values:
detector_bbox_info- Holds bounding box parameters of the object when detected by detector.
tracker_bbox_info- Holds bounding box parameters of the object when processed by tracker.
rect_params- Holds bounding box coordinates of the object processed by the last module that updates it in the pipeline. These bounding box coordinates are clipped so that they do not fall outside frame boundary.
These bounding box parameters are used to overlay:
confidence- Holds a confidence value for the object, set by the inference module.
tracker_confidence- Holds a confidence value for the object set by NvDCF. Will be
1.0in the case of the other trackers (e.g.,
NvDeepSORT) that do not have visual tracking capability.
As noted in the description above, the bbox coordinates from detector and tracker are stored separately in the newly introduced
tracker_bbox_info, respectively. The bbox info
rect_params continues to hold the bbox info as in previous DeepStream versions, for backward compatibility. Note that
rect_params will be deprecated in future release.
confidence parameter holds the detector confidence value, while a newly introduced
tracker_confidence parameter will hold the tracker confidence value.