Metadata Migration from DeepStream Release 3.0 to 4.0.1
This chapter explains how migrate metadata from applications in DeepStream 3.0 metadata to DeepStream 4.0.1.
The DeepStream 4.0.1 metadata architecture:
• Is easy to use and intuitive
• Supports new APIs for adding custom meta
Adding and Accessing Metadata
When a DeepStream 3.0 application must inference on two or more streams, it attaches two separate metadata objects such as NvStreamMeta and NvDsFrameMeta.
The NvStreamMeta metadata contains information about the number of sources in the batch and their widths, heights, frame numbers, presentation time stamps (pts). etc. NvDsFrameMeta contains information about the number of objects in the frame and their metadata. The application iterates over all of the metadata attached to the Gst Buffer and accesses these metadata based on the type of attached metadata. The application then must associate these two metadata to find their association.
In DeepStream 4.0.1, the Gst-nvstreammux plugin is a required element in the pipeline. It creates NvDsBatchMeta objects (equivalent to DeepStream 3.0 NvStreamMeta) after receiving input from all of the sources. Only this metadata is attached to the Gst Buffer. A Gst plugin can access it by calling gst_buffer_get_nvds_batch_meta(). The function returns a list of pointers to NvDsFrameMeta objects in the batch. Each NvDsFrameMeta object holds frame information like width, height, presentation time stamp (pts), and frame number. This information is obtained by the Gst-nvstreammux plugin.
For each object detected or classified in the frame, the plugin or the application must call nvds_acquire_object_meta_from_pool(), which returns an instance of NvDsObjectMeta, fills its metadata fields with information such as bounding box coordinates, class IDs, etc., and adds the NvDsObjectMeta instance to the corresponding frame metadata by calling nvds_add_obj_meta_to_frame(). This process maintains the association between frame-specific information and the associated object metadata for that frame.
For more information about metadata access, see the sample application source file sources/apps/sample_apps/deepstream-test1/deepstrem-test1_app.c.
For more information about metadata addition, see the source file at:
sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
Adding Secondary Classifier Inference Metadata
When a DeepStream 3.0 application must add classifier metadata to objects detected by the primary inference plugin, it must fill the NvDsAttrInfo structure in NvDsObjectMeta with classifier results.
To add classifier metadata in DeepStream 4.0.1, an application must:
1. Acquire a classifier metadata by a call to nvds_add_classifier_meta_to_object()
2. Acquire label metadata by calling nvds_acquire_label_info_meta_from_pool()
3. Add the label metadata to the classifier metadata by calling nvds_add_label_info_meta_to_classifier()
4. Fill an NvDsClassifierMeta object and add it to the NvDsObjectMeta object by calling nvds_add_classifier_meta_to_object()
For more information about adding classifier metadata, see the source code for gstnvinfer_meta_utils.cpp.
Adding User Metadata
When a DeepStream 3.0 application needs to add user metadata in addition to the fields provided the metadata structure, it must call gst_buffer_add_nvds_meta(). It must set the metadata type appropriately when iterating over the Gst Buffer.
When a DeepStream 4.0.1 application needs to add metadata beyond the fields provided the metadata structure, it can add the metadata to the additional user fields provided in the metadata structures. If these fields are not sufficient, the application can allocate its own metadata and add it at the batch, frame or object level, wherever appropriate. To do this the application must acquire a user meta from the user meta pool in NvDsBatchMeta, then call nvds_acquire_user_meta_from_pool(). The application then fills in the fields and adds the metadata to the batch, frame, or object by calling nvds_add_user_meta_to_batch(), nvds_add_user_meta_to_frame(), or nvds_add_user_meta_to_obj(). The SDK provides an example of how to add custom metadata.
For more information, see the source code in sources/apps/sample_apps/deepstream-user-metadata-test.c.
Adding RoI Metadata
When a DeepStream 3.0 application needs to define a region of interest (RoI) for the frame, it must create an NvDsLineMeta object and add it to the Gst Buffer by calling gst_buffer_add_nvds_meta(). The metadata type for RoI metadata must be set to NVDS_META_LINE_INFO.
When a DeepStream 4.0.1 application needs to define region of interest, it must acquire display meta from the display meta pool in NvDsBatcMeta by a call to nvds_acquire_display_meta_from_pool().
Set RoI information by filling the NvOSD_LineParams structure embedded in the NvDsDisplayMeta structure. Then add it to the frame by calling nvds_add_display_meta_to_frame().
Adding Additional Display Metadata
An application often needs to display metadata results for a given frame, such as the number of persons displayed in the frame.
A DeepStream 3.0 application meets such a requirement by allocating an NvDsFrameMeta structure. It sets the structure’s num_rects member to 1, allocates an NvDsObjectMeta structure, and fills it by placing the text and the result summary in the child structure NvOSD_TextParams. (The result summary may be, e.g., the number of people in the frame.) This information is then attached by calling gst_buffer_add_nvds_meta().
A DeepStream 4.0.1 application meets this requirement by calling nvds_acquire_display_meta_from_pool() to acquire display meta from the display meta pool in NvDsBatchMeta. It adds the result summary by filling the NvOSD_TextParams structure in NvDsDisplayMeta and adding it to the frame by calling nvds_add_display_meta_to_frame().
For more information about display metadata usage, see the source code in sources/apps/sample_apps/deepstream-test1/deepstrem-test1_app.c.
How to Dereference Metadata
In DeepStream 3.0, all meta are attached to the Gst Buffer. To dereference a component of the metadata the application must call gst_buffer_remove_meta().
In DeepStream 4.0.1, only NvDsBatchMeta is attached to the Gst Buffer. To dereference it the application must call gst_buffer_remove_meta().
To dereference metadata in the NvDsBatchMeta hierarchy, an application must call nvds_remove_type(), where type is the type of metadata: frame, object, classifier, label, display, or user.
