Gst-nvmsgconv

The Gst-nvmsgconv plugin is responsible for converting metadata into message payloads based on schema. DeepStream 5.0 supports two variations of the schema, full and minimal. The Gst-nvmsgconv plugin can be configured to use either one of the schemas. By default, the plugin uses the full DeepStream schema to generate the payload in JSON format. The full schema supports elaborate semantics for object detection, analytics modules, events, location, sensor and 3D tracking. Each payload has information about a single object. You can use the minimal variation of the schema to communicate minimal information with the back end. This provides a small footprint for the payload to be transmitted from DeepStream to a message broker. Each payload can have information for multiple objects in the frame.

Gst-nvmsgconv plugin interface with a low level nvmsgconv library which provides APIs for payload generation using below metadata types:

1. NVDS_EVENT_MSG_META (NvDsEventMsgMeta) type metadata attached to the buffer as user metadata of frame meta. For the batched buffer, metadata of all objects of a frame must be under the corresponding frame meta. This is the default option.

2. By parsing the NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type in Gst buffer and available fields in these metadata types are used to create message payload based on schema type. To use this option, set gst property msg2p-newapi = true.

This API also supports attaching a custom user message blob(a json formatted string) for each payload if the msg blob is added as user metadata of NvDsFrameMeta. Additionally, with this API there’s also an other gst property you can use to set the frame interval at which each payload is generated. frame-interval = <interval>. If this is not set, the default frame interval at which frames are generates is every 30 frames

The generated payload (NvDsPayload) in both the above cases are attached back to the input buffer as NVDS_PAYLOAD_META type user metadata.

Additionally, Gst-nvmsgconv plugin provides optional properties:

• debug-payload-dir : Absolute path of the directory to dump payloads for debugging
• multiple-payloads : Generate multiple message payloads

Inputs and Outputs

• Inputs

  • Gst Buffer with NvDsEventMsgMeta or NvDsFrameMeta/NvDsObjectMeta

• Control parameters

  • config
  • msg2p-lib
  • payload-type
  • comp-id
  • debug-payload-dir (optional)
  • multiple-payloads (optional)
  • msg2p-newapi (optional)
  • frame-interval (optional)

• Output

  • Same Gst Buffer with additional NvDsPayload metadata. This metadata contains information about the payload generated by the plugin.

Features

The following table summarizes the features of the plugin.

Gst-nvmsgconv plugin features

Gst Properties

The following table describes the Gst-nvmsgconv plugin’s Gst properties.

Gst-nvmsgconv plugin gst properties

NvMsgConv low level library API’s

Gst-nvmsgconv plugin interface with a low level nvmsgconv library which provides APIs for payload generation:

• nvds_msg2p_generate to generate message payload using NVDS_EVENT_MSG_META (NvDsEventMsgMeta) type
• nvds_msg2p_generate_new to generate message payload using GST buffers(NVDS_FRAME_META (NvDsFrameMeta) and NVDS_OBJECT_META (NvDsObjectMeta) type)

Deepstream sample test apps have options to use either of API’s to generate message payload

deepstream-test4-app

• Use command line option —msg2p-meta to choose the metadata type to create payload.
• 0=Event Msg meta(default), Create NVDS_EVENT_MSG_META type of meta and attach to buffer
• 1=nvdsmeta, Use the fields within NvDsFrameMeta/NvDsObjectMeta to populate payload

deepstream-test5-app

• Config options present as below and default option creates payload using NvdsEventMsgMeta
• #(0): Create payload using NvdsEventMsgMeta
• #(1): New Api to create payload using NvDsFrameMeta
• msg-conv-msg2p-new-api=0

For payload generation using NVDS_EVENT_MSG_META type, users need:

• Create NVDS_EVENT_MSG_META type of meta and attach to buffer.
• Use NVDS_EVENT_MSG_META for different types of objects e.g. vehicle, person etc.
• Provide copy / free functions if meta data is extended through “extMsg” field.

Whereas using the new NvMsgConv low level library API to generate payload, users do not need to worry about creating/destroying any payload object in their sample test application. The nvmsgconv low level library directly uses GST buffers(NVDS_FRAME_META / NVDS_OBJECT_META) to create payload.

Schema Customization

This plugin can be used to implement a custom schema in two ways:

• By modifying the payload generator library: To perform a simple customization of DeepStream schema fields, modify the low-level payload generation library files inside directory sources/utils/nvmsgconv. Payload generated using NVDS_EVENT_MSG_META (NvDsEventMsgMeta) type can be customized by modifying sources/utils/nvmsgconv/deepstream_schema/eventmsg_payload.cpp. If payload is generated using metadata within DS GST buffers(NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type), you can customize payload generation by modifying the file sources/utils/nvmsgconv/deepstream_schema/dsmeta_payload.cpp.

• By implementing the nvds_msg2p interface: If a library that implements the custom schema needs to be integrated with the DeepStream SDK, wrap the library in the nvds_msg2p interface and set the plugin’s msg2p-lib property to the library’s name. Set the payload-type property to PAYLOAD_CUSTOM.

See sources/utils/nvmsgconv/nvmsgconv.cpp for an example implementation of the nvds_msg2p interface.

Payload with Custom Objects

You can add a group of custom objects to the NvDsEventMsgMeta structure in the extMsg field and specify their size in the extMsgSize field. The meta copy (copy_func) and free (release_func) functions must handle the custom fields accordingly. The payload generator library handles some standard types of objects (Vehicle, Person, Face, etc.) and generates the payload according to the schema selected. To handle custom object types, you must modify the payload generator library nvmsgconv.cpp. See deepstream-test4 for details about adding custom objects as NVDS_EVENT_MSG_META user metadata with buffers for generating a custom payload to send to back end.

If using Deepstream GST buffers(NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type) to generate payload instead of NvDsEventMsgMeta, You can attach a custom message blob(string) to payload using NVDS_CUSTOM_MSG_BLOB type by creating NvDsCustomMsgInfo object and specifying the fields message (custom message blob) and len (size of the custom message). The low-level payload generation library parses every frame metadata and checks for NVDS_CUSTOM_MSG_BLOB type and if present attaches it to the payload. This option is available when gst property msg2p-newapi = true (Using the new msgconv API).

Similar to generating custom objects to the NvDsEventMsgMeta structure, refer deepstream-test4 for details about adding custom objects, you can use the meta copy (copy_func) and free (release_func) functions to handle the custom fields and modify the application to attach a custom message using NVDS_CUSTOM_MSG_BLOB (and NvDsCustomMsgInfo object) type instead of NvDsEventMsgMeta type.