DeepStream Components#
DeepStream/Gstreamer pipeline is implemented using the Graph Specification by wrapping GStreamer element in Graph Component and adding supportive Components to enable communication with GStreamer pipeline from Graph components. This section explains components types used in DeepStream graphs.
Interfaces#
DeepStream extensions provide interfaces (abstract base types) for some commonly used types of components and functionality. These interfaces provide a way for other components to interact with a variety of components implementing or inheriting from the interface without depending on the actual type/implementation. For e.g. NvDsInfer
component has a parameter infer-model-config with handle type INvDsInferModelConfigComponent
. The INvDsInferModelConfigComponent
base type is for components representing a NN model (the model files, associated configuration, post-processing algorithm). Thus, NvDsInfer
component can load any type of model without actual depending on the concrete type of the model component by calling APIs of INvDsInferModelConfigComponent
.
The interfaces also act as base types which can be used by tools like registry and composer to filter and group together components inheriting from the interface. Thus in the above example, registry command registry comp list -b nvidia::deepstream::INvDsInferModelConfigComponent
can be used to find all the components that can be connected to the parameter “infer-model-config” of NvDsInfer
. Composer also uses this base type to provide quick access context-menu to on a handle type parameter create a component inheriting from the base type and attaching the newly created component to the parameter. However interfaces themselves are abstract and won’t be listed in the component list from the Composer for users to create the instances through drag&drop.
The rest of the section provides a high-level overview of the interfaces provided by the DeepStream extensions. These interfaces are part of the NvDsInterfaceExt
extension. The detailed API and parameter description is part of the document Graph Composer Extensions Manual.
Element - INvDsElement#
This interface is used by components representing a GStreamer element in the graph. An element is the basic building block of a GStreamer based pipeline. It represents a unit of processing. These elements can perform very specific tasks like video decoding or encoding or they might perform a set of related tasks together e.g. Object detection - pre-processing, inference and post-processing to get bounding box co-ordinates of detected objects.
This interface is primarily used by the NvDsScheduler
component to manage the lifecycle of the elements and construct a GStreamer pipeline from the elements represented by the components. It should only be used as a base class and derived from to create a component representing a GStreamer element. Ideally, no other component should ever use this interface for interacting with the elements. Other mechanisms are provided which are described later.
This interface provides following main virtual methods.
|
Concrete types must use this to create the GStreamer element and return the error/success status |
|
Concrete types must use this to add the created element to the provided GStreamer pipeline and return the error/success status |
|
Concrete types must return the raw pointer to the GStreamer element |
Most components implementing this interface are thin wrappers created over corresponding GStreamer element. In such cases, there would be a 1-to-1 mapping between the Component parameter and the GStreamer element property. Special cases like NvDsMultiSrcInput
exist which are not wrappers over a single element but elaborate implementations using GStreamer API to combine multiple elements together.
I/Os - INvDsIO/INvDsInput/INvDsOutput#
The INvDsIO
interface is used to represent to represent an input/output of an INvDsElement
based component. It is used to represent a GStreamer Pad which is the input / output port of a GStreamer element through which it receives/transmits data. The direction input or output is from the perspective of the component and not the application.
An element receives data from an input and transmits data on an output. The base type for an input component is INvDsInput
and that for an output is INvDsOutput
. Both of these inherit from INvDsIO
. Depending on the I/O (data) availability type, multiple concrete types are derived from INvDsInput
and INvDsOutput
, which are described in detail later. The complete hierarchy looks like
Any concrete pad type can interact with via INvDsIO
interface or INvDsInput/INvDsOutput
interface or the concrete pad classes themselves.
INvDsElement
based components have parameters with the handle type of the concrete pad types (depending on the direction – input/output and availability – static/on-request/dynamic/multi). In the graph, for such a component to send/receive data, an I/O component of the corresponding type must be attached to the parameter. For e.g., the NvDsStreamMux
component has a “on-request” input and a “static” output. Thus, for it to receive data a NvDsOnRequestInput
component must be attached to its video-in-%u
parameter and for it to send data a NvDsStaticOutput
component must be attached to its “video-out” parameter.
An I/O is identified by its template name in the element. GStreamer APIs can be used to find more details about the underlying pad details using this template name. These interfaces expose the following main virtual methods:
Interfaces |
Details |
---|---|
|
Called by INvDsElement based element to set the template name of the i/o. |
|
Get the owning INvDsElement based element of the i/o. |
|
Get the template name for the i/o. |
|
Get the underlying GstPad pointer as a shared_ptr. Detailed behavior of this API for each type of I/O has been documented in the “Pads” subsection of “Basic Components” section |
|
Get a list of all the underlying GstPad objects as vector of shared_ptrs. In case of multi/on-request/dynamic type of I/Os, element might have multiple underlying pads associated with the I/O. This API can be used to get the raw pointers to all such pads. |
Custom extensions will almost never need to implement a I/O based component from these interfaces. All the concrete types that would be required are already provided as a part of the NvDsBaseExt
extension. Custom components wanting to interact with pads directly can use APIs from this interface. However, if only access to the data being received/transmitted is required “Probe” based mechanism is recommended.
Connections - INvDsConnection#
The INvDsConnection interface is used by components representing a link between an output component and an input component i.e., a link between two INvDsElement
based components. Concrete implementations do the actual work of connecting the components.
The main virtual methods provided by this interface are:
Methods |
Details |
---|---|
|
Concrete implementations must link the source and target I/Os and return the status |
|
Concrete implementations must unlink the source and target I/Os |
|
Method to check if an I/O is compatible with the I/O on the other side of the connection. |
|
Notification from dynamic I/Os that a new underlying GstPad object has been added. |
|
Notification from dynamic I/Os that an existing underlying GstPad object has been removed. |
|
Notification from dynamic I/Os that no more new underlying GstPad objects would be added. |
The interface APIs are primarily used by the NvDsScheduler
component during pipeline construction to link elements. Ideally, custom components and extensions will never need to implement a component based on this interface or use the APIs provided by this interface.
The “output” component of the upstream element component must be attached to the “source” parameter of the INvDsConnection
based component and the “input” component of the downstream element component must be attached to the “target” parameter.
DeepStream Domain Component - INvDsComponent#
The INvDsComponent
interface should be used by custom components working with DeepStream components in the graph. The main feature of this interface is that it adds start()
and stop()
virtual methods on top of the Component interface.
The component gets a notification with start()
method that all other components in the graph have been initialized, underlying pipeline constructed and ready to start. In this method it is safe to call APIs of other components through their handles. This is not guaranteed to be safe in initialize()
method provided by Component because the other component may not be initialized by then.
stop()
method is called when the underlying pipeline has been stopped and before components are de-initialized.
Probe - INvDsProbe#
The INvDsProbe
interface is used by components representing a probe. A probe is used for monitoring dataflow and in-place buffer modifications by adding callbacks on the I/O components of and INvDsElement
based component.
Depending on the flags used, these callbacks are called for every buffer/query/event that flows through the I/O. INvDsProbe
based components are used to only install a probe. The components that want to implement these callbacks need to implemented the INvDsInPlaceDataHandler
interface described later.
Ideally, custom components and extensions will never need to implement this interface or use APIs of this interface, it is required by the NvDsScheduler
for setting up the probes.
Probe Callback Implementation - INvDsInPlaceDataHandler#
Custom components wanting to monitor data flow or to do in place data modifications at a particular I/O of a component must inherit from this interface. The methods of this interface are called by the INvDsProbe
based component when a buffer/event/query arrives at the I/O.
The interface provides the following virtual methods. Concrete implementations can implement any of the methods as required.
Methods |
Details |
---|---|
|
Notification to the component to handle buffer data |
|
Notification to the component to handle the passed event |
|
Notification to the component to handle passed query |
Returning true from these methods tells the probe component to let them pass through. Returning false implies that the objects should be dropped at that point. The buffer_data argument of the handle_buffer method is an Entity that contains the following data:
Key |
Value Type |
---|---|
|
GstBufferHandle |
|
NvBufSurfaceHandle |
|
NvBufAudioHandle |
|
NvDsBatchMetaHandle |
|
NvDsBatchMetaHandle |
The value types are explained in the Data components section.
Depending upon the type of data flowing through the I/O, not all values might be present in the buffer_data entity. For e.g for a I/O handling video frames, buffer_data can contain NvBufSurfaceHandle
and NvDsBatchMetaHandle
for video but not NvBufAudioHandle
and NvDsBatchMetaHandle
for audio.
The handle_buffer
implementations must check for existence of the values before directly using the values.
INvDsInPlaceDataHandler
based components must register a handle parameter of type NvDsProbeConnector
and in the initialize()
method must call set_handler()
and set_flags()
method of the NvDsProbeConnector
to receive callbacks.
For more information on GStreamer queries and events and the APIs, refer to the following links:
Action - INvDsAction#
The INvDsAction
interface is used to represent a GstElement action. An action is a trigger that signals an INvDsElement based component to perform some action corresponding to that trigger.
Every concrete INvDsElement
based component that supports actions will have a corresponding concrete type derived from INvDsAction
. This concrete type will expose a method to trigger the action.
The INvDsAction
based components act only as helper components for providing a prototype for the trigger and connecting a component that triggers an action to a component that reacts to the trigger.
The custom component that triggers the action must be implemented separately. It must register a handle parameter of the type of concrete INvDsAction
based component. The component that reacts to the trigger would also do the same. The concrete INvDsAction based component must be attached to the parameters of the other two components. When required, the custom component can trigger the action by calling the action trigger method on the handle of the INvDsAction
based component, for example, NvDsRecordAction
component. This action is used to toggle recording on/off recording of a camera source. Thus this component has methods start_record()
, stop_record()
and take_snapshot()
. This action is supported by NvDsMultiSrcInputWithRecord
component, thus it has a handle parameter “record-action” of type NvDsRecordAction
.
To trigger the action, a custom component must have a Parameter of type Handle<NvDsRecordAction>
. The component NvDsRecordAction
must be added to the graph and attached to the parameters of both, the NvDsMultiSrcInputWithRecord
component and the custom component. The custom component can then call start_record()
, stop_record()
and take_snapshot()
methods as required.
Signal - INvDsSignal#
The INvDsSignal
interface is used to represent a GstElement signal. A signal is a callback from an INvDsElement
based component on the occurrence of some event.
Every concrete component that supports signals will have a corresponding concrete type derived from INvDsSignal
. This concrete type will expose the prototype of the callback function and a method to set the callback function.
The INvDsSignal
based components act only as helper components for providing a prototype for the callback and connecting a component that emits the signal to a component that implements the callback handling the signal.
The custom component that handles the signal must be implemented separately. It must register a handle parameter of the type of concrete INvDsSignal
based component. It must implement the Handler interface exposed by the signal component and call set_handler() method on the handle of the signal component. The component that emits the signal would similarly register a handle parameter of type of the concrete signal component. The concrete INvDsSignal
based component must be attached to the parameters of the other two components. When the INvDsElement
based element emits the corresponding signal, the methods of the Handler interface will get called via the signal component for example, NvDsModelUpdateSignal
component. This signal is used to notify the status of on the fly model update. Thus the Handler interface for this signal has the method on_model_updated(int errorCode, char *configFilePath)
. This signal is emitted by NvDsInfer component, thus it has a handle parameter model-updated-signal
of type NvDsModelUpdateSignal
.
To receive the signal callback, a custom component must implement NvDsModelUpdateSignal::Handler
interface. It must register a parameter of type Handle<NvDsModelUpdateSignal>
and call set_handler()
on the handle. The component NvDsModelUpdateSignal
must be added to the graph and attached to the parameters of both NvDsInfer
and the custom component.
Element Property Controller – INvDsPropertyController#
The INvDsPropertyController
interface is used by components acting as runtime property(parameter) controllers for INvDsElement
based components.
Every concrete INvDsElement
based component that supports runtime property control will have a corresponding concrete type derived from INvDsPropertyController
. This concrete type will expose methods to get/set the properties(parameters) of the element component. A method for each property that can be controlled will be exposed.
The INvDsPropertyController
based components act only as helper components for providing prototypes for the get/set methods and connecting a component that can control the property to the element component that the property belongs to.
The custom component that controls the properties must be implemented separately. It must register a handle parameter of the type of concrete INvDsPropertyController
based component. The component the properties belong to would also do the same. The concrete INvDsPropertyController
based component must be attached to the parameters of the other two components. When required, the custom component can call the get/set methods of the properties on the handle of the INvDsAction
based component for exmaple, NvDsOSDPropertyController
component. This component can be used to get/set properties of NvDsOSD
component, thus it has a handle parameter property-controller
of type NvDsOSDPropertyController
.
To controls properties of the NvDsOSD
component, a custom component must have a Parameter of type Handle<NvDsOSDPropertyController>
. The component NvDsOSDPropertyController
must be added to the graph and attached to the paramers of both, the NvDsOSD
component and the custom component. The custom component can then call set/get methods exposed by NvDsOSDPropertyController
.
Configurations – INvDsConfigComponent template and specializations#
The INvDsConfigComponent
interface is used as a base class for components acting as configuration providers (collection of settings, associated files, binaries etc.) for other components. The aim of this interface and the components that would derive from it is to:
Simplify specification of group of parameters and their values that are fixed for a use case.
Package together all related assets (e.g. model files, custom implementation libraries)
Remove the need of knowing beforehand the paths to files during deployment and thus hardcoding these paths to related asset files in the
yaml
files. Since the files are packaged with the extension library, the component can internally determine the correct absolute path to the files.
The INvDsConfigComponent
interface is a templated type. It provides a standard virtual method fill_config(Config *config)
for any type of configuration, Config being the template type. It also provides a helper function get_absolute_path(std::string path)
which can convert a path relative to the extension’s binary to an absolute path at runtime.
Components wanting to use this template interface for reading configurations must declare a structure with supported configuration parameters and specialize the template by using the struct as a template argument. These components must have a parameter with handle type of the specialized template. The configuration can then be read by calling the fill_config()
method of the handle.
Components wanting to act as configuration providers must inherit from the specialized template and implement the fill_config()
method. The configuration provider component is responsible for populating the config structure appropriately. These components can use the get_absolute_path()
method to convert paths relative to extension binary to absolute paths.
The DeepStream extensions currently have the following configuration provider interfaces:
INvDsInferModelConfigComponent#
This interface acts as a configuration provider for the NvDsInferVideo
and NvDsInferAudio
components. Components implementing this interface must populate the following parameters of the configuration structure in the fill_config()
method.
Method |
Details |
---|---|
|
Absolute path to the nvinfer configuration file for the model |
|
Absolute path to the pre-generated TensorRT engine file for the model (OPTIONAL) |
NvDsInferVideo
and NvDsInferAudio
components expose a handle parameter infer-model-config
of type INvDsInferModelConfigComponent
to which components implementing this interface can be attached.
Components implementing this interface are model configuration providers. Thus they will package the DeepStreamSDK Gst-nvinfer/ Gst-nvinferaudio configuration file along with the model files. The specification of the Gst-nvinfer configuration file can be found at https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer-file-configuration-specifications.
INvDsVideoTemplatePluginConfigComponent / INvDsAudioTemplatePluginConfigComponent#
These interfaces act as configuration providers for the NvDsVideoTemplate
and the NvDsAudioTemplate
components respectively.
Components implementing these interfaces must populate the following parameters of the configuration structure in the fill_config()
method.
Method |
Details |
---|---|
|
Absolute path to the custom library that the audio/video template component should load |
|
name-value pairs of properties to be passed to the custom library. |
NvDsVideoTemplate
and NvDsAudioTemplate
components expose handle parameters video-template-config
and audio-template-config
respectively to which components implementing the interfaces INvDsVideoTemplatePluginConfigComponent
or INvDsAudioTemplatePluginConfigComponent
can be attached.
Components implementing these interfaces will package the custom library that must be provided to the template components.
NvDsVideoTemplate
and NvDsAudioTemplate
components are wrappers for the DeepStreamSDK Gst-nvdsvideotemplate
and Gst-nvdsaudiotemplate
plugins respectively. For information on these plugins, refer to https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsvideotemplate.html and https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsaudiotemplate.html.
Data Components#
DeepStream extensions provide some Data components which can be passed around as part of entities. This removes the need of having fixed structures / function prototypes to pass data, thus removing the problem of API breakage with changing versions. Entity provides a key-value mechanism for adding and retrieving data components. The key is of string type. Both the component producing these entities and consuming the entities must know the mapping between a key and the type of the associated value.
Components consuming such entities must also check the existence of the key value pair by having a Boolean check on the return value of the get()
method of the entity.
The rest of the section provides details on the data components provided by DeepStream extensions. These data components are part of the NvDsInterfaceExt
extension and are defined in the interfaces.hpp
header file of the extension.
GstBufferHandle#
Wraps the pointer to GstBuffer
which is the GStreamer data structure for buffers - https://gstreamer.freedesktop.org/documentation/gstreamer/gstbuffer.html . The key usually associated with this data component is defined by BUF_DATA_KEY_GST_BUFFER.
NvBufSurfaceHandle#
Wraps the pointer to NvBufSurface
which is the DeepStream data structure for batched video frames - https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/structNvBufSurface.html . The key usually associated with this data component is defined by BUF_DATA_KEY_NVBUFSURFACE
.
NvBufAudioHandle#
Wraps the pointer to NvBufAudio
which is the DeepStream data structure for batched audio data – LINK TBD. The key usually associated with this data component is defined by BUF_DATA_KEY_NVBUFAUDIO
.
NvDsBatchMetaHandle#
Wraps the pointer to NvDsBatchMeta
which is the DeepStream data structure for metadata associated with batched video frames or batched audio data – https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/struct__NvDsBatchMeta.html . The key usually associated with this data component is defined by BUF_DATA_KEY_VIDEO_BATCH_META
or BUF_DATA_KEY_AUDIO_BATCH_META
depending on whether the NvDsBatchMeta
structure contains information for video or audio.
Basic Components#
This section explains the basic concrete components that are required to create any type of DeepStream based graph. All these basic components are part of the NvDsBaseExt
extension.
I/Os#
As explained in the previous section of the INvDsIO
interface, I/Os are the input/output ports of an INvDsElement
based component.
An I/O is defined by a template which consists of:
the direction (input or output)
it’s availability
the format of the data that flows through it.
An I/O is identified by the template’s name. An element component will have one or more I/Os depending on it’s functionality and input/output requirements.
The following types of I/O availabilities are defined along with the behavior of the GstPadSPtr get_pad(char *requested_name)
method of INvDsIO
interface for each of the type.
Static – The I/O is always available on the element:
get_pad()
always returns the underlyingGstPad
object (single).requested_name
argument is ignoredOnRequest – The underlying I/O objects must be requested from the element by the application:
e.g.
NvDsStreamMux
component which aggregates frames from multiple upstream components. UnderlyingGstPad
objects for pushing data to the component must be requested from it, one for each upstream component.e.g.
NvDsTee
component which broadcasts input data to multiple outputs. UnderlyingGstPad
objects for pushing data to multiple downstream components must be requested from it, one for each downstream component.If a non-NULL
requested_name
is provided toget_pad()
, the method will try to create an underlyingGstPad
with the supplied name if it does not already exist and return it. If a pad with the provided requested_name cannot be created, NULL will be returned. Ifrequested_name
is not supplied, it creates a underlyingGstPad
with an internally generated name from the I/O’s template name and return it.Dynamic – The underlying I/O object is made available by the element at runtime when some conditions are satisfied:
e.g.
NvDsSingleSrcInput
component which reads from an input like FILE or RTSP and can decide whether it can output audio or video or both only after parsing the file headers/initial communication with the RTSP server.
get_pad()
tries to get an existing underlyingGstPad
object having name requested_name, returns NULL if no such pad is found. requested_name is mandatory.Multi – A special I/O availability created for components which output data from multiple sources. This availability exposes itself as a single I/O but internally groups together multiple outputs, one for each of the sources. Due to the group representation, it makes the graph and components that would use the I/O independent of the number of sources. This makes it easy to have a variable number of sources, and to dynamically add/remove sources at runtime without any changes to the graph.
- e.g.
NvDsMultiSrcInput
component which outputs data from multiple sources.
get_pad()
tries to get an existing underlyingGstPad
object having namerequested_name
, returns NULL if no such pad is found.requested_name
is mandatory.
Based on these concepts, the following concrete I/O types are provided:
NvDsStaticOutput
NvDsOnRequestOutput
NvDsDynamicOutput
NvDsMultiOutput
NvDsStaticInput
NvDsOnRequestInput
An element component exposes a handle parameter of a concrete I/O type for each of the I/O template it supports. For any kind of interaction with an I/O, like connecting two elements, in-place manipulation of data flowing through the I/O, a concrete type of I/P component must be attached to the parameter. Depending on the function of the element component, a variety of I/O templates and thus handle parameters will be present on the component.
Connections#
The connection components are responsible for connecting two elements. Connecting two elements means linking an output of one of the elements (called upstream element) to an input of the other element (called downstream element) which enables flow of data between the two elements through the linked I/Os. Two types of connection components are provided:
NvDsConnection#
This connection component can be used to link any type of output component to any type of input component with the exception of NvDsMultiOutput
. It provides two handle parameters “source” and “target”. To connect, an output I/O component attached to an upstream element must be attached to the “source” parameter of an NvDsConnection
component and an input I/O component attached to a downstream element must be attached to the “target” parameter of the NvDsConnection
component.
NvDsMultiSrcConnection#
As with NvDsMultiOutput
, this connection type has been specially created for handling the use case of variable number of sources and dynamic addition/removal of sources at runtime. It can only be used to link a NvDsMultiOutput
component to a NvDsOnRequestInput
component, i.e. link an element which outputs data from multiple sources to aggregation type of elements. It has two parameters, “source” to attach the NvDsMultiOutput
component to and “target” to attach the NvDsOnRequestInput
component.
Probes#
As mentioned earlier probes are used for in-place manipulation and monitoring of data flowing through the I/Os of an element component. Two components are provided to achieve this.
NvDsProbe#
This component is responsible for intercepting data (buffers/events/queries) flowing through an I/O and calling the callback functions of the target handler component whenever data arrives at the I/O. It has two handle type properties. “io” to which an I/O component must be attached which needs to be probed and “probe-handler” to which a NvDsProbeConnector
component must be attached.
NvDsProbeConnector#
This component is used to link a NvDsProbe
component to a target component derived from INvDsInPlaceDataHandler
which makes it capable of handling callbacks from the probe.
This component provides the following public methods, defined in nvds_probe_connector.hpp
header file as part of NvDsBaseExt
:
Method |
Details |
---|---|
|
Set the component that will be handling the probe callbacks. Must be called by INvDsInPlaceDataHandler based components. |
|
Get the component that will be handling the probe callbacks. |
|
Set the types of information to receive probe callbacks for. Must be called by INvDsInPlaceDataHandler based components. |
|
Get the types of information to receive probe callbacks for. |
|
Set the IO this component is connected to. This method is called by NvDsProbe. |
|
Get the IO this component is connected to. |
The flags must be a bitwise OR combination of the following:
Method |
Details |
---|---|
|
The target does not wish to handle any data |
|
The target component wants to handle buffers |
|
The target component wants to handle events |
|
The target component wants to handle queries |
The target component must register a handle parameter of type NvDsProbeConnector
. The target component in it’s initialize()
method must call set_handler()
and set_flags()
method of the NvDsProbeConnector
to receive callbacks.
NvDsScheduler#
The NvDsScheduler
component is responsible for setting up the underlying pipeline and managing its state, connecting and scheduling components in the graph and managing their lifecycle. It is also responsible for setting up the probe handlers.
There must be a NvDsScheduler
in a graph. Without the presence of a NvDsScheduler
in the graph, the DeepStream based components would never get scheduled. Additionally, there must be only one NvDsScheduler
in a graph. In case multiple NvDsSchedulers
are added to a graph, only one gets actually executed others become redundant and get skipped.