.. include::   Developing Extensions for DeepStream ---------------------------------------------------- This section provides an overview of developing extensions with a focus on developing DeepStream based components. The usage of various interfaces and components is explained through sample code snippets. For the sake of simplicity, the component definition and implementation of its methods are shown together in the sample code. However, this can be split between a header file and .cpp source file as required. .. note:: * The interfaces.hpp header provided as part of ``NvDsInterfaceExt`` contains the basic DeepStream interface definitions. * The definition of some concrete component types like the I/Os, NvDsProbeConnector are included in header files provided as part of ``NvDsBaseExt``. * The definitions of component specific Action, Signal, PropertyController, and Configuration types are included in header files provided as part of the extension the component belongs to. * All DeepStream types are defined under the namespace ``nvidia::deepstream``. Implementations should take care of this if not using the same namespace. Extension and component factory registration boilerplate ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following provides a boilerplate on registering extensions and components :: #include "gxf/std/extension_factory_helper.hpp" #include "sample_runtime_source_manipulator.hpp" ... GXF_EXT_FACTORY_BEGIN() GXF_EXT_FACTORY_SET_INFO(0x44a711e485194a68, 0x81e8be7ee4af3ff0, "NvDsSampleExt", "Sample components for demonstrating usage of various " "DeepStream interfaces and components", "NVIDIA", "0.0.1", "Proprietary"); ... GXF_EXT_FACTORY_ADD( 0x717b2c432f104fe8, 0xb96165e408ece299, nvidia::deepstream::NvDsSimpleComponent, nvidia::deepstream::INvDsComponent, "Description of a simple component"); ... GXF_EXT_FACTORY_END() * Include the header file ``gxf/std/extension_factory_helper.hpp`` containing the helper macros. Include any other header file containing the definitions of components to be registered. * ``GXF_EXT_FACTORY_BEGIN()`` and ``GXF_EXT_FACTORY_END()`` mark the beginning and end of the block of code containing registration code. * Within the factory block, call ``GXF_EXT_FACTORY_SET_INFO()`` to set the extension information including the UUID for the extension, the extension name, description, author, version and license. * For each of the component to be registered as part of the extension, call ``GXF_EXT_FACTORY_ADD()`` to add the component along with the component information including the UUID for the component, the component type, the component’s base type and a description of the component. * The UUIDs are unique 128-bit identifiers. This must be unique across all extensions and component registered. In the code, two 64-bit unsigned integers represent the upper 64 bits and lower 64bits of the 128-bit UUID. A simple DeepStream component ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: #include "extensions/nvdsinterface/interfaces.hpp" namespace nvidia { namespace deepstream { class NvDsSimpleComponent : INvDsComponent { public: // Public methods using which other components can interact with this // component via its handle. void simpleComponentMethod() { // } private: gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override { nvidia::gxf::Expected result; result &= registrar->parameter( simple_param_, // Parameter member variable "simple-param-key", // Parameter name(key). This is used to set // parameter value in a graph "Simple Parameter", // Parameter head line "Description of the simple parameter", // A description of the // parameter 100UL, // A default value for the parameter GXF_PARAMETER_FLAGS_OPTIONAL // Parameter flags marking it ); result &= registrar->parameter(handle_param_, "handle-parameter", "Handle Parameter", "Description of the handle parameter", std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL); return nvidia::gxf::ToResultCode(result); } gxf_result_t initialize() override { // This method can be used to initialize the component. ... // Check if parameter is set if (simple_param_.try_get() != std::nullopt) { uint64_t simple_param_value = simple_param_.try_get().value(); ... } return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error } gxf_result_t deinitialize() override { // This method can be used to deinitialize the component. ... return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error } gxf_result_t start() override { // Start the component. The underlying DeepStream pipeline and other // components are already initialized. It is safe to call methods of other components // via their handles. ... // Check if any component is attached to the parameter if (handle_param.try_get() != std::nullopt) { SampleOtherComponent *other_comp = handle_param.try_get().value(); other_comp->otherComponentMethod(); ... } return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error } gxf_result_t stop() override { // Pipeline has been stopped. All the components would be deinitialized // after this. Add any logic for stopping the component. return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error } nvidia::gxf::Parameter simple_param_; nvidia::gxf::Parameter> handle_param_; }; } // namespace deepstream Implementation of INvDsInPlaceDataHandler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: #include "extensions/nvdsbase/nvds_probe_connector.hpp" #include "extensions/nvdsinterface/interfaces.hpp" namespace nvidia { namespace deepstream { class NvDsSampleInPlaceDataHandler : public INvDsInPlaceDataHandler { nvidia::gxf::Parameter> probe_connector_; gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) { ... // Must register a handle parameter of type NvDsProbeConnector result &= registrar->parameter( probe_connector_, "probe-connector", "Probe Connector", "Handle to a nvidia::deepstream::NvDsProbeConnector component", std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL); ... } gxf_result_t initialize() override { ... // Need to set the handler and flags which tells NvDsProbe component // what type of data this component wants to handle. This must be done // in the initialize method. if (probe_connector_.try_get() != std::nullopt) { // The pointer to self (this) can be passed since the component // implements the INvDsInPlaceDataHandler interface. probe_connector_.try_get().value()->set_handler(this); probe_connector_.try_get().value()->set_flags(static_cast( NvDsProbeFlags::BUFFER | NvDsProbeFlags::EVENT)); } ... return GXF_SUCCESS; } bool handle_buffer(GstPad *pad, nvidia::gxf::Entity buffer_data) override { // Check for presence of NvDsBatchMeta and NvBufSurface in the buffer_data // entity since this implementation requires it. Other implementations may // want to check for presence of other data components. if (!buffer_data.get(BUF_DATA_KEY_VIDEO_BATCH_META)) { return true; } if (!buffer_data.get(BUF_DATA_KEY_NVBUFSURFACE)) { return true; } NvDsBatchMeta *batch_meta = *(buffer_data.get(BUF_DATA_KEY_VIDEO_BATCH_META) .value()); NvBufSurface *surf = *(buffer_data.get(BUF_DATA_KEY_VIDEO_BATCH_META) .value()); // Use batch_meta and surf ... return true; // allows buffer to pass through. return false to drop the buffer } bool handle_event(GstPad *pad, GstEvent *event) override { // Use the event ... return true; // allows buffer to pass through. return false to drop the event } }; } // namespace deepstream } // namespace nvidia The component ``NvDsSampleProbeMessageMetaCreation`` provided as part of the sample extension is a complete sample implementation of ``INvDsInPlaceDataHandler`` interface. Controlling Properties ~~~~~~~~~~~~~~~~~~~~~~~ This sample code demonstrates the usage of an ``INvDsPropertyController`` (``NvDsOSDPropertyController`` in this case) component by a custom component to control properties of an ``INvDsElement`` based component (``NvDsOSD`` in this case). :: // Provided as part of the NvDsVisualizationExt extension which contains the // NvDsOSD component. This header contains the definition of // NvDsOSDPropertyController component which provides methods to control // properties of NvDsOSD component. #include "nvdsvisualization_prop_controllers.hpp" namespace nvidia { namespace deepstream { class NvDsSamplePropertyControl : public INvDsComponent { nvidia::gxf::Parameter> osd_property_controller_; NvDsOSDPropertyController *osd_property_controller = nullptr; gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override { ... // Must register a handle parameter of type NvDsOSDPropertyController result &= registrar->parameter( osd_property_controller_, "nvdsosd-prop-controller", "NvDsOSD Property Controller", "Handle to a nvidia::deepstream::NvDsOSDPropertyController " "component.", std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL); ... }; void sample_thread_func() { while (1) { bool display_text; // Get the current property value osd_property_controller->get_display_text(&display_text); // Toggle the property value osd_property_controller->set_display_text(!display_text); sleep(1); // Break when signalled to stop } } gxf_result_t start() override { ... // Check if the property controller component is attached if (osd_property_controller_.try_get() != std::nullopt) { osd_property_controller = osd_property_controller_.try_get().value(); osd_property_controller->set_display_text(true); // Start a thread which periodically toggles the display-text property } ... } gxf_result_t stop() override { // Signal samplethread to stop } }; } // namespace deepstream } // namespace nvidia Triggering Actions ~~~~~~~~~~~~~~~~~~~~ This sample code demonstrates the usage of an ``INvDsAction`` (``NvDsSourceManipulationAction`` in this case) component by a custom component to trigger actions of an ``INvDsElement`` based component (``NvDsMultiSrcInput`` in this case). :: // Provided as part of the NvDsSourceExt extension which contains the // NvDsMultiSrcInput component. This header contains the definition of // NvDsSourceManipulationAction component which provides methods to trigger // add/remove source actions of NvDsMultiSrcInput component. #include "nvdsinputsrc_signals.hpp" namespace nvidia { namespace deepstream { class NvDsSampleSourceManipulator : public INvDsComponent { nvidia::gxf::Parameter> source_manip_action_; NvDsSourceManipulationAction *source_manip_action = nullptr; bool add = true; gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override { ... // Must register a handle parameter of type NvDsSourceManipulationAction result &= registrar->parameter( source_manip_action_, "source-manip-action", "Source Manipulation Action", "Handle to a nvidia::deepstream::NvDsSourceManipulationAction " "component", std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL); ... } void sample_thread_func() { while (1) { // Call methods of the action component via it's handle to trigger the add/remove actions. if (add) { if (!source_manip_action->add_source("file:///some/file/path.mp4", 2)) { GXF_LOG_WARNING("Failed to add source"); } } else { if (!source_manip_action->remove_source("file:///some/file/path.mp4", 2)) { GXF_LOG_WARNING("Failed to remove source"); } } add = !add; sleep(10); // Break when signalled to stop } } gxf_result_t start() override { ... // Check if the action component is attached if (source_manip_action_.try_get() != std::nullopt) { source_manip_action = source_manip_action_.try_get().value(); // Start a thread to periodically add / remove a source } ... } gxf_result_t stop() override { // Stop the thread } }; } // namespace deepstream } // namespace nvidia The component ``NvDsSampleSourceManipulator`` provided as part of the sample extension is a complete sample implementation demonstrating this. Handling signal callbacks ~~~~~~~~~~~~~~~~~~~~~~~~~~~ This sample code demonstrates the usage of an ``INvDsSignal`` (``NvDsModelUpdatedSignal`` in this case) component by a custom component to handle signals emitted by an ``INvDsElement`` based component (``NvDsInferVideo`` in this case). :: // Provided as part of the NvDsInferenceExt extension which contains the // NvDsInferVideo component. This header contains the definition of // NvDsModelUpdatedSignal component which provides the Handler interface and // callback method prototype and a way to set the Handler instance. #include "nvdsinference_signals.hpp" namespace nvidia { namespace deepstream { class NvDsSampleSignalHandler : public INvDsComponent, public NvDsModelUpdatedSignal::Handler { nvidia::gxf::Parameter> model_update_signal_; gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override { ... // Must register a handle parameter of type NvDsModelUpdatedSignal result &= registrar->parameter( model_update_signal_, "model-update-signal", "Model Updated Signal", "Handle to a nvidia::deepstream::NvDsModelUpdatedSignal " "component.", std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL); ... }; // Implement the methods of the NvDsModelUpdatedSignal::Handler interface void on_model_updated(int errorCode, gchar *configFilePath) override { // Handle the on-the-fly model update status. } gxf_result_t start() override { // Start the component ... // Check if the signal component is attached, set the signal handler to self if (model_update_signal_.try_get() != std::nullopt) { // The pointer to self (this) can be passed since the component // implements the NvDsModelUpdatedSignal::Handler interface model_update_signal_.try_get().value()->set_handler(this); } ... } gxf_result_t stop() override { // Stop the component } }; } // namespace deepstream } // namespace nvidia Implementation of an Configuration Provider component ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This sample code demonstrates the implementation of a configuration provider interface (``INvDsInferModelConfigComponent`` in this case) component by a custom component to provide a configuration (to ``NvDsInferVideo`` in this case). :: // Provided as part of the NvDsInferenceExt extension which contains the // NvDsInferVideo component. This header contains the definition of // INvDsInferModelConfigComponent interface which acts as a model configuration // provider to the NvDsInferVideo component. #include "nvdsinference_config.hpp" namespace nvidia { namespace deepstream { class SampleModel : public INvDsInferModelConfigComponent { gxf_result_t fill_config(NvDsInferModelConfig *config) override { // config_infer_primary.txt is packaged alongside the extension binary, // along with caffemodel, prototxt and labels file. Use get_absolute_path() // method to convert a path relative to the extension binary to an absolute // path at runtime. config->config_file_path = get_absolute_path("config_infer_primary.txt"); if (engine_file_.try_get() != std::nullopt) { // The configuration provider component itself can have parameters which // can be used to populate the configuration. config->model_engine_path = get_absolute_path(engine_file_.try_get().value()); } return GXF_SUCCESS; } gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override { nvidia::gxf::Expected result; result &= registrar->parameter(engine_file_, "model-engine-file", "Model Engine File", "Path to the model engine file. Absolute or " "relative to the extension directoy.", std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL); return nvidia::gxf::ToResultCode(result); } nvidia::gxf::Parameter engine_file_; }; } // namespace deepstream } // namespace nvidia ``NvDsSampleAudioTemplateLib`` and ``NvDsSampleVideoTemplateLib`` are other examples of components acting as configuration providers. These components are part of the sample extension.