.. include:: DeepStream Components ===================== DeepStream/Gstreamer pipeline is implemented using the Graph Specification by wrapping GStreamer element in Graph Component and adding supportive Components to enable communication with GStreamer pipeline from Graph components. This section explains components types used in DeepStream graphs. Later section will explain how to create DeepStream graph using these components. Interfaces ------------ DeepStream extensions provide interfaces (abstract base types) for some commonly used types of components and functionality. These interfaces provide a way for other components to interact with a variety of components implementing or inheriting from the interface without depending on the actual type/implementation. For e.g. NvDsInfer component has a parameter `infer-model-config` with handle type ``INvDsInferModelConfigComponent``. The ``INvDsInferModelConfigComponent`` base type is for components representing a NN model (the model files, associated configuration, post-processing algorithm). Thus, NvDsInfer component can load any type of model without actual depending on the concrete type of the model component by calling APIs of ``INvDsInferModelConfigComponent``. The interfaces also act as base types which can be used by tools like registry and composer to filter and group together components inheriting from the interface. Thus in the above example, registry command ``registry comp list -b nvidia::deepstream::INvDsInferModelConfigComponent`` can be used to find all the components that can be connected to the parameter “infer-model-config” of NvDsInfer. Composer also uses this base type to provide quick access context-menu to on a handle type parameter create a component inheriting from the base type and attaching the newly created component to the parameter. .. image:: /content/DS_Zero_coding_interface.png :align: center :alt: NvdsInfer Component The rest of the section provides a high-level overview of the interfaces provided by the DeepStream extensions. These interfaces are part of the ``NvDsInterfaceExt`` extension. The detailed API and parameter description is part of the document Graph Composer Extensions Manual. Element - INvDsElement ~~~~~~~~~~~~~~~~~~~~~~~~~ This interface is used by components representing a GStreamer element in the graph. An element is the basic building block of a GStreamer based pipeline. It represents a unit of processing. These elements can perform very specific tasks like video decoding or encoding or they might perform a set of related tasks together e.g. Object detection - pre-processing, inference and post-processing to get bounding box co-ordinates of detected objects. This interface is primarily used by the ``NvDsScheduler`` component to manage the lifecycle of the elements and construct a GStreamer pipeline from the elements represented by the components. It should only be used as a base class and derived from to create a component representing a GStreamer element. Ideally, no other component should ever use this interface for interacting with the elements. Other mechanisms are provided which are described later. This interface provides following main virtual methods. +-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ | ``gxf_result_t create_element()`` | Concrete types must use this to create the GStreamer element and return the `error/success` status | +-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ | ``gxf_result_t bin_add(GstElement *pipeline)`` | Concrete types must use this to add the created element to the provided GStreamer pipeline and return the error/success status | +-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ | ``GstElement *get_element_ptr()`` | Concrete types must return the raw pointer to the GStreamer element | +-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ Most components implementing this interface are thin wrappers created over corresponding GStreamer element. In such cases, there would be a 1-to-1 mapping between the Component parameter and the GStreamer element property. Special cases like ``NvDsMultiSrcInput`` exist which are not wrappers over a single element but elaborate implementations using GStreamer API to combine multiple elements together. I/Os - INvDsIO/INvDsInput/INvDsOutput ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``INvDsIO`` interface is used to represent to represent an input/output of an ``INvDsElement`` based component. It is used to represent a GStreamer Pad which is the input / output port of a GStreamer element through which it receives/transmits data. The direction input or output is from the perspective of the component and not the application. An element receives data from an input and transmits data on an output. The base type for an input component is ``INvDsInput`` and that for an output is ``INvDsOutput``. Both of these inherit from ``INvDsIO``. Depending on the I/O (data) availability type, multiple concrete types are derived from ``INvDsInput`` and ``INvDsOutput``, which are described in detail later. The complete hierarchy looks like .. image:: /content/DS_INvDsIO_INvDsInput_INvDsOutput.png :align: center :alt: INvDsIO_INvDsInput_INvDsOutput Any concrete pad type can interact with via ``INvDsIO`` interface or ``INvDsInput/INvDsOutput`` interface or the concrete pad classes themselves. ``INvDsElement`` based components have parameters with the handle type of the concrete pad types (depending on the direction – input/output and availability – static/on-request/dynamic/multi). In the graph, for such a component to send/receive data, an I/O component of the corresponding type must be attached to the parameter. For e.g., the ``NvDsStreamMux`` component has a “on-request” input and a “static” output. Thus, for it to receive data a ``NvDsOnRequestInput`` component must be attached to its ``video-in-%u`` parameter and for it to send data a ``NvDsStaticOutput`` component must be attached to its “video-out” parameter. An I/O is identified by its template name in the element. GStreamer APIs can be used to find more details about the underlying pad details using this template name. These interfaces expose the following main virtual methods: ====================================================== ============================================================================================================================================================================================================== Interfaces Details ====================================================== ============================================================================================================================================================================================================== ``void set_template_name(const char *templ)`` Called by INvDsElement based element to set the template name of the i/o. ``INvDsElement *get_element()`` Get the owning INvDsElement based element of the i/o. ``const char *get_template_name()`` Get the template name for the i/o. ``GstPadSPtr get_pad(const char *requested_name)`` Get the underlying GstPad pointer as a shared_ptr. Detailed behavior of this API for each type of I/O has been documented in the “Pads” subsection of “Basic Components” section ``const std::vector get_all_pads()`` Get a list of all the underlying GstPad objects as vector of shared_ptrs. In case of multi/on-request/dynamic type of I/Os, element might have multiple underlying pads associated with the I/O. This API can be used to get the raw pointers to all such pads. ====================================================== ============================================================================================================================================================================================================== Custom extensions will almost never need to implement a I/O based component from these interfaces. All the concrete types that would be required are already provided as a part of the ``NvDsBaseExt`` extension. Custom components wanting to interact with pads directly can use APIs from this interface. However, if only access to the data being received/transmitted is required “Probe” based mechanism is recommended. Connections - INvDsConnection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The INvDsConnection interface is used by components representing a link between an output component and an input component i.e., a link between two ``INvDsElement`` based components. Concrete implementations do the actual work of connecting the components. The main virtual methods provided by this interface are: ============================================================== ======================================================================================================================= Methods Details ============================================================== ======================================================================================================================= ``gxf_result_t connect()`` Concrete implementations must link the source and target I/Os and return the status ``void disconnect()`` Concrete implementations must unlink the source and target I/Os ``bool is_io_compatible(INvDsIO *io, GstPadSPtr gstpad)`` Method to check if an I/O is compatible with the I/O on the other side of the connection. ``void on_pad_added(INvDsIO *io, GstPadSPtr gstpad)`` Notification from dynamic I/Os that a new underlying GstPad object has been added. ``void on_pad_removed(INvDsIO *io, GstPadSPtr gstpad)`` Notification from dynamic I/Os that an existing underlying GstPad object has been removed. ``void on_no_more_pads(INvDsIO *io)`` Notification from dynamic I/Os that no more new underlying GstPad objects would be added. ============================================================== ======================================================================================================================= The interface APIs are primarily used by the ``NvDsScheduler`` component during pipeline construction to link elements. Ideally, custom components and extensions will never need to implement a component based on this interface or use the APIs provided by this interface. The “output” component of the upstream element component must be attached to the “source” parameter of the ``INvDsConnection`` based component and the “input” component of the downstream element component must be attached to the “target” parameter. DeepStream Domain Component - INvDsComponent ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``INvDsComponent`` interface should be used by custom components working with DeepStream components in the graph. The main feature of this interface is that it adds ``start()`` and ``stop()`` virtual methods on top of the Component interface. The component gets a notification with ``start()`` method that all other components in the graph have been initialized, underlying pipeline constructed and ready to start. In this method it is safe to call APIs of other components through their handles. This is not guaranteed to be safe in ``initialize()`` method provided by Component because the other component may not be initialized by then. ``stop()`` method is called when the underlying pipeline has been stopped and before components are de-initialized. Probe - INvDsProbe ~~~~~~~~~~~~~~~~~~~~~ The ``INvDsProbe`` interface is used by components representing a probe. A probe is used for monitoring dataflow and in-place buffer modifications by adding callbacks on the I/O components of and ``INvDsElement`` based component. Depending on the flags used, these callbacks are called for every buffer/query/event that flows through the I/O. ``INvDsProbe`` based components are used to only install a probe. The components that want to implement these callbacks need to implemented the ``INvDsInPlaceDataHandler`` interface described later. Ideally, custom components and extensions will never need to implement this interface or use APIs of this interface, it is required by the ``NvDsScheduler`` for setting up the probes. Probe Callback Implementation - INvDsInPlaceDataHandler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Custom components wanting to monitor data flow or to do in place data modifications at a particular I/O of a component must inherit from this interface. The methods of this interface are called by the ``INvDsProbe`` based component when a buffer/event/query arrives at the I/O. The interface provides the following virtual methods. Concrete implementations can implement any of the methods as required. ==================================================================== =============================================================================== Methods Details ==================================================================== =============================================================================== ``bool handle_buffer(GstPad *pad, nvidia::gxf::Entity buffer_data)`` Notification to the component to handle buffer data ``bool handle_event(GstPad *pad, GstEvent *event)`` Notification to the component to handle the passed event ``bool handle_query(GstPad *pad, GstQuery *query)`` Notification to the component to handle passed query ==================================================================== =============================================================================== Returning true from these methods tells the probe component to let them pass through. Returning false implies that the objects should be dropped at that point. The buffer_data argument of the handle_buffer method is an Entity that contains the following data ======================================= ================================================ Key Value Type ======================================= ================================================ ``BUF_DATA_KEY_GST_BUFFER`` GstBufferHandle ``BUF_DATA_KEY_NVBUFSURFACE`` NvBufSurfaceHandle ``BUF_DATA_KEY_NVBUFAUDIO`` NvBufAudioHandle ``BUF_DATA_KEY_GST_VIDEO_BATCH_META`` NvDsBatchMetaHandle ``BUF_DATA_KEY_GST_AUDIO_BATCH_META`` NvDsBatchMetaHandle ======================================= ================================================ The value types are explained in the Data components section. Depending upon the type of data flowing through the I/O, not all values might be present in the buffer_data entity. For e.g for a I/O handling video frames, buffer_data can contain ``NvBufSurfaceHandle`` and ``NvDsBatchMetaHandle`` for video but not ``NvBufAudioHandle`` and ``NvDsBatchMetaHandle`` for audio. The ``handle_buffer`` implementations must check for existence of the values before directly using the values. ``INvDsInPlaceDataHandler`` based components must register a handle parameter of type ``NvDsProbeConnector`` and in the ``initialize()`` method must call ``set_handler()`` and ``set_flags()`` method of the ``NvDsProbeConnector`` to receive callbacks. For more information on GStreamer queries and events and the APIs, refer to the following links: * https://gstreamer.freedesktop.org/documentation/additional/design/query.html * https://gstreamer.freedesktop.org/documentation/gstreamer/gstquery.html * https://gstreamer.freedesktop.org/documentation/additional/design/events.html * https://gstreamer.freedesktop.org/documentation/gstreamer/gstevent.html Action - INvDsAction ~~~~~~~~~~~~~~~~~~~~~~ The ``INvDsAction`` interface is used to represent a GstElement action. An action is a trigger that signals an INvDsElement based component to perform some action corresponding to that trigger. Every concrete ``INvDsElement`` based component that supports actions will have a corresponding concrete type derived from ``INvDsAction``. This concrete type will expose a method to trigger the action. The ``INvDsAction`` based components act only as helper components for providing a prototype for the trigger and connecting a component that triggers an action to a component that reacts to the trigger. The custom component that triggers the action must be implemented separately. It must register a handle parameter of the type of concrete ``INvDsAction`` based component. The component that reacts to the trigger would also do the same. The concrete INvDsAction based component must be attached to the parameters of the other two components. When required, the custom component can trigger the action by calling the action trigger method on the handle of the ``INvDsAction`` based component e.g ``NvDsRecordAction`` component. This action is used to toggle recording on/off recording of a camera source. Thus this component has methods ``start_record()``, ``stop_record()`` and ``take_snapshot()``. This action is supported by ``NvDsMultiSrcInputWithRecord`` component, thus it has a handle parameter “record-action” of type ``NvDsRecordAction``. To trigger the action, a custom component must have a Parameter of type ``Handle``. The component ``NvDsRecordAction`` must be added to the graph and attached to the parameters of both, the ``NvDsMultiSrcInputWithRecord`` component and the custom component. The custom component can then call ``start_record()``, ``stop_record()`` and ``take_snapshot()`` methods as required. Signal - INvDsSignal ~~~~~~~~~~~~~~~~~~~~~~~ The ``INvDsSignal`` interface is used to represent a GstElement signal. A signal is a callback from an ``INvDsElement`` based component on the occurrence of some event. Every concrete component that supports signals will have a corresponding concrete type derived from ``INvDsSignal``. This concrete type will expose the prototype of the callback function and a method to set the callback function. The ``INvDsSignal`` based components act only as helper components for providing a prototype for the callback and connecting a component that emits the signal to a component that implements the callback handling the signal. The custom component that handles the signal must be implemented separately. It must register a handle parameter of the type of concrete ``INvDsSignal`` based component. It must implement the Handler interface exposed by the signal component and call set_handler() method on the handle of the signal component. The component that emits the signal would similarly register a handle parameter of type of the concrete signal component. The concrete ``INvDsSignal`` based component must be attached to the parameters of the other two components. When the ``INvDsElement`` based element emits the corresponding signal, the methods of the Handler interface will get called via the signal component. e.g ``NvDsModelUpdateSignal`` component. This signal is used to notify the status of on the fly model update. Thus the Handler interface for this signal has the method ``on_model_updated(int errorCode, char *configFilePath)``. This signal is emitted by NvDsInfer component, thus it has a handle parameter ``model-updated-signal`` of type ``NvDsModelUpdateSignal``. To receive the signal callback, a custom component must implement ``NvDsModelUpdateSignal::Handler`` interface. It must register a parameter of type ``Handle`` and call ``set_handler()`` on the handle. The component ``NvDsModelUpdateSignal`` must be added to the graph and attached to the parameters of both ``NvDsInfer`` and the custom component. Element Property Controller – INvDsPropertyController ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``INvDsPropertyController`` interface is used by components acting as runtime property(parameter) controllers for ``INvDsElement`` based components. Every concrete ``INvDsElement`` based component that supports runtime property control will have a corresponding concrete type derived from ``INvDsPropertyController``. This concrete type will expose methods to get/set the properties(parameters) of the element component. A method for each property that can be controlled will be exposed. The ``INvDsPropertyController`` based components act only as helper components for providing prototypes for the get/set methods and connecting a component that can control the property to the element component that the property belongs to. The custom component that controls the properties must be implemented separately. It must register a handle parameter of the type of concrete ``INvDsPropertyController`` based component. The component the properties belong to would also do the same. The concrete ``INvDsPropertyController`` based component must be attached to the parameters of the other two components. When required, the custom component can call the get/set methods of the properties on the handle of the ``INvDsAction`` based component. e.g ``NvDsOSDPropertyController`` component. This component can be used to get/set properties of ``NvDsOSD`` component, thus it has a handle parameter ``property-controller`` of type ``NvDsOSDPropertyController``. To controls properties of the ``NvDsOSD`` component, a custom component must have a Parameter of type ``Handle``. The component ``NvDsOSDPropertyController`` must be added to the graph and attached to the paramers of both, the ``NvDsOSD`` component and the custom component. The custom component can then call set/get methods exposed by ``NvDsOSDPropertyController``. Configurations – INvDsConfigComponent template and specializations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``INvDsConfigComponent`` interface is used as a base class for components acting as configuration providers (collection of settings, associated files, binaries etc.) for other components. The aim of this interface and the components that would derive from it is to: * Simplify specification of group of parameters and their values that are fixed for a use case. * Package together all related assets (e.g. model files, custom implementation libraries) * Remove the need of knowing beforehand the paths to files during deployment and thus hardcoding these paths to related asset files in the ``yaml`` files. Since the files are packaged with the extension library, the component can internally determine the correct absolute path to the files. The ``INvDsConfigComponent`` interface is a templated type. It provides a standard virtual method ``fill_config(Config *config)`` for any type of configuration, Config being the template type. It also provides a helper function ``get_absolute_path(std::string path)`` which can convert a path relative to the extension’s binary to an absolute path at runtime. Components wanting to use this template interface for reading configurations must declare a structure with supported configuration parameters and specialize the template by using the struct as a template argument. These components must have a parameter with handle type of the specialized template. The configuration can then be read by calling the ``fill_config()`` method of the handle. Components wanting to act as configuration providers must inherit from the specialized template and implement the ``fill_config()`` method. The configuration provider component is responsible for populating the config structure appropriately. These components can use the ``get_absolute_path()`` method to convert paths relative to extension binary to absolute paths. The DeepStream extensions currently have the following configuration provider interfaces: INvDsInferModelConfigComponent ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This interface acts as a configuration provider for the ``NvDsInferVideo`` and ``NvDsInferAudio`` components. Components implementing this interface must populate the following parameters of the configuration structure in the ``fill_config()`` method. ================================== ===================================================================================== Method Details ================================== ===================================================================================== ``std::string config_file_path`` Absolute path to the nvinfer configuration file for the model ``std::string model_engine_path`` Absolute path to the pre-generated TensorRT engine file for the model (OPTIONAL) ================================== ===================================================================================== ``NvDsInferVideo`` and ``NvDsInferAudio`` components expose a handle parameter ``infer-model-config`` of type ``INvDsInferModelConfigComponent`` to which components implementing this interface can be attached. Components implementing this interface are model configuration providers. Thus they will package the DeepStreamSDK Gst-nvinfer/ Gst-nvinferaudio configuration file along with the model files. The specification of the Gst-nvinfer configuration file can be found at https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer-file-configuration-specifications. INvDsVideoTemplatePluginConfigComponent / INvDsAudioTemplatePluginConfigComponent ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These interfaces act as configuration providers for the ``NvDsVideoTemplate`` and the ``NvDsAudioTemplate`` components respectively. Components implementing these interfaces must populate the following parameters of the configuration structure in the ``fill_config()`` method. ================================================================ ================================================================================================== Method Details ================================================================ ================================================================================================== ``std::string customlib_name`` Absolute path to the custom library that the audio/video template component should load ``std::unordered_map customlib_props`` name-value pairs of properties to be passed to the custom library. ================================================================ ================================================================================================== ``NvDsVideoTemplate`` and ``NvDsAudioTemplate`` components expose handle parameters ``video-template-config`` and ``audio-template-config`` respectively to which components implementing the interfaces ``INvDsVideoTemplatePluginConfigComponent`` or ``INvDsAudioTemplatePluginConfigComponent`` can be attached. Components implementing these interfaces will package the custom library that must be provided to the template components. ``NvDsVideoTemplate`` and ``NvDsAudioTemplate`` components are wrappers for the DeepStreamSDK ``Gst-nvdsvideotemplate`` and ``Gst-nvdsaudiotemplate`` plugins respectively. For information on these plugins, refer to https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsvideotemplate.html and https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsaudiotemplate.html.   Data Components ----------------- DeepStream extensions provide some Data components which can be passed around as part of entities. This removes the need of having fixed structures / function prototypes to pass data, thus removing the problem of API breakage with changing versions. Entity provides a key-value mechanism for adding and retrieving data components. The key is of string type. Both the component producing these entities and consuming the entities must know the mapping between a key and the type of the associated value. Components consuming such entities must also check the existence of the key value pair by having a boolean check on the return value of the ``get()`` method of the entity. The rest of the section provides details on the data components provided by DeepStream extensions. These data components are part of the ``NvDsInterfaceExt`` extension and are defined in the ``interfaces.hpp`` header file of the extension. GstBufferHandle ~~~~~~~~~~~~~~~~~ Wraps the pointer to ``GstBuffer`` which is the GStreamer data structure for buffers - https://gstreamer.freedesktop.org/documentation/gstreamer/gstbuffer.html . The key usually associated with this data component is defined by BUF_DATA_KEY_GST_BUFFER. NvBufSurfaceHandle ~~~~~~~~~~~~~~~~~~~ Wraps the pointer to ``NvBufSurface`` which is the DeepStream data structure for batched video frames - https://docs.nvidia.com/metropolis/deepstream/sdk-api/BufSurface/NvBufSurface.html . The key usually associated with this data component is defined by ``BUF_DATA_KEY_NVBUFSURFACE``. NvBufAudioHandle ~~~~~~~~~~~~~~~~~ Wraps the pointer to ``NvBufAudio`` which is the DeepStream data structure for batched audio data – LINK TBD. The key usually associated with this data component is defined by ``BUF_DATA_KEY_NVBUFAUDIO``. NvDsBatchMetaHandle ~~~~~~~~~~~~~~~~~~~~~ Wraps the pointer to ``NvDsBatchMeta`` which is the DeepStream data structure for metadata associated with batched video frames or batched audio data – https://docs.nvidia.com/metropolis/deepstream/sdk-api/Meta/_NvDsBatchMeta.html . The key usually associated with this data component is defined by ``BUF_DATA_KEY_VIDEO_BATCH_META`` or ``BUF_DATA_KEY_AUDIO_BATCH_META`` depending on whether the ``NvDsBatchMeta`` structure contains information for video or audio.   Basic Components ------------------- This section explains the basic concrete components that are required to create any type of DeepStream based graph. All these basic components are part of the ``NvDsBaseExt`` extension. I/Os ~~~~~~~ As explained in the previous section of the ``INvDsIO`` interface, I/Os are the input/output ports of an ``INvDsElement`` based component. An I/O is defined by a template which consists of * the direction (input or output) * it’s availability * the format of the data that flows through it. An I/O is identified by the template’s name. An element component will have one or more I/Os depending on it’s functionality and input/output requirements. The following types of I/O availabilities are defined along with the behavior of the ``GstPadSPtr get_pad(char *requested_name)`` method of ``INvDsIO`` interface for each of the type. * Static – The I/O is always available on the element * ``get_pad()`` always returns the underlying ``GstPad`` object (single). ``requested_name`` argument is ignored * OnRequest – The underlying I/O objects must be requested from the element by the application * e.g. ``NvDsStreamMux`` component which aggregates frames from multiple upstream components. Underlying ``GstPad`` objects for pushing data to the component must be requested from it, one for each upstream component. * e.g. ``NvDsTee`` component which broadcasts input data to multiple outputs. Underlying ``GstPad`` objects for pushing data to multiple downstream components must be requested from it, one for each downstream component. * If a non-NULL ``requested_name`` is provided to ``get_pad()``, the method will try to create an underlying ``GstPad`` with the supplied name if it does not already exist and return it. If a pad with the provided requested_name cannot be created, NULL will be returned. If ``requested_name`` is not supplied, it creates a underlying ``GstPad`` with an internally generated name from the I/O's template name and return it. * Dynamic – The underlying I/O object is made available by the element at runtime when some conditions are satisfied * e.g. ``NvDsSingleSrcInput`` component which reads from an input like FILE or RTSP and can decide whether it can output audio or video or both only after parsing the file headers/initial communication with the RTSP server. * ``get_pad()`` tries to get an existing underlying ``GstPad`` object having name requested_name, returns NULL if no such pad is found. requested_name is mandatory. * Multi – A special I/O availability created for components which output data from multiple sources. This availability exposes itself as a single I/O but internally groups together multiple outputs, one for each of the sources. Due to the group representation, it makes the graph and components that would use the I/O independent of the number of sources. This makes it easy to have a variable number of sources, and to dynamically add/remove sources at runtime without any changes to the graph. * e.g. ``NvDsMultiSrcInput`` component which outputs data from multiple sources. * ``get_pad()`` tries to get an existing underlying ``GstPad`` object having name ``requested_name``, returns NULL if no such pad is found. ``requested_name`` is mandatory. Based on these concepts, the following concrete I/O types are provided: * NvDsStaticOutput * NvDsOnRequestOutput * NvDsDynamicOutput * NvDsMultiOutput * NvDsStaticInput * NvDsOnRequestInput An element component exposes a handle parameter of a concrete I/O type for each of the I/O template it supports. For any kind of interaction with an I/O, like connecting two elements, in-place manipulation of data flowing through the I/O, a concrete type of I/P component must be attached to the parameter. Depending on the function of the element component, a variety of I/O templates and thus handle parameters will be present on the component. Connections -------------- The connection components are responsible for connecting two elements. Connecting two elements means linking an output of one of the elements (called upstream element) to an input of the other element (called downstream element) which enables flow of data between the two elements through the linked I/Os. Two types of connection components are provided: NvDsConnection ~~~~~~~~~~~~~~~~~ This connection component can be used to link any type of output component to any type of input component with the exception of ``NvDsMultiOutput``. It provides two handle parameters “source” and “target”. To connect, an output I/O component attached to an upstream element must be attached to the “source” parameter of an ``NvDsConnection`` component and an input I/O component attached to a downstream element must be attached to the “target” parameter of the ``NvDsConnection`` component. NvDsMultiSrcConnection ~~~~~~~~~~~~~~~~~~~~~~~~~ As with ``NvDsMultiOutput``, this connection type has been specially created for handling the use case of variable number of sources and dynamic addition/removal of sources at runtime. It can only be used to link a ``NvDsMultiOutput`` component to a ``NvDsOnRequestInput`` component, i.e. link an element which outputs data from multiple sources to aggregation type of elements. It has two parameters, “source” to attach the ``NvDsMultiOutput`` component to and “target” to attach the ``NvDsOnRequestInput`` component. Probes ----------- As mentioned earlier probes are used for in-place manipulation and monitoring of data flowing through the I/Os of an element component. Two components are provided to achieve this. NvDsProbe ~~~~~~~~~~~~ This component is responsible for intercepting data (buffers/events/quries) flowing through an I/O and calling the callback functions of the target handler component whenever data arrives at the I/O. It has two handle type properties. “io” to which an I/O component must be attached which needs to be probed and “probe-handler” to which a ``NvDsProbeConnector`` component must be attached. NvDsProbeConnector ~~~~~~~~~~~~~~~~~~~~ This component is used to link a ``NvDsProbe`` component to a target component derived from ``INvDsInPlaceDataHandler`` which makes it capable of handling callbacks from the probe. This component provides the following public methods, defined in ``nvds_probe_connector.hpp`` header file as part of ``NvDsBaseExt``: ============================================================== ================================================================================================================================ Method Details ============================================================== ================================================================================================================================ ``void set_handler(INvDsInPlaceDataHandler *handler)`` Set the component that will be handling the probe callbacks. Must be called by INvDsInPlaceDataHandler based components. ``INvDsInPlaceDataHandler *get_handler()`` Get the component that will be handling the probe callbacks. ``void set_flags(NvDsProbeFlags flags)`` Set the types of information to receive probe callbacks for. Must be called by INvDsInPlaceDataHandler based components. ``NvDsProbeFlags get_flags()`` Get the types of information to receive probe callbacks for. ``void set_io(INvDsIO *io)`` Set the IO this component is connected to. This method is called by NvDsProbe. ``INvDsIO *get_io()`` Get the IO this component is connected to. ============================================================== ================================================================================================================================ The flags must be a bitwise OR combination of the following: =============================== ============================================================= Method Details =============================== ============================================================= ``NvDsProbeFlags::NONE`` The target does not wish to handle any data ``NvDsProbeFlags::BUFFER`` The target component wants to handle buffers ``NvDsProbeFlags::EVENT`` The target component wants to handle events ``NvDsProbeFlags::QUERY`` The target component wants to handle queries =============================== ============================================================= The target component must register a handle parameter of type ``NvDsProbeConnector``. The target component in it’s ``initialize()`` method must call ``set_handler()`` and ``set_flags()`` method of the ``NvDsProbeConnector`` to receive callbacks. NvDsScheduler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``NvDsScheduler`` component is responsible for setting up the underlying pipeline and managing its state, connecting and scheduling components in the graph and managing their lifecycle. It is also responsible for setting up the probe handlers. There must be a ``NvDsScheduler`` in a graph. Without the presence of a ``NvDsScheduler`` in the graph, the DeepStream based components would never get scheduled. Additionally, there must be only one ``NvDsScheduler`` in a graph. In case multiple ``NvDsSchedulers`` are added to a graph, only one gets actually executed others become redundant and get skipped.