DeepStream 3D Custom Manual ======================================== New ``ds3d`` framework, interfaces and custom-libs are defined for DS 3D processing. These interfaces are capble of different types of data fusion and can implement different types of custom libraries for ``dataloader``, ``datafilter`` and ``datarender``. The interface has ABI compatible layers and modern C++ interface layers. You'll only need to focus on the modern interface for application or custom lib development. DeepStream 3D ``dataloader`` is loaded by ``GstAppSrc``. It could be used for depth camera such as stereo cameras, Time-of-Flight cameras to capture image/depth data or 2D/3D data-load from the source file. ``datafilter`` is loaded by the ``nvds3dfilter`` `Gst-plugin`. It could be used for 2D depth data processing , 3D point-cloud data extraction from depth and other 2D-depth or 3D-points data filters. ``datarender`` is loaded by ``GstAppSink``. It could be used for 2D depth rendering and 3D point-cloud data rendering. ``deepstream-3d-depth-camera`` has the sample code to load these custom libs and to connect these components together in simple ways. Besides that, DS3D has a simple C++ safe pointer for `Gstreamer` components. The interfaces are found in header files located at ``/opt/nvidia/deepstream/deepstream/sources/libs/ds3d/gst/``. The image below shows the overview of depth to 3D point processing pipeline in ``deepstream-3d-depth-camera``. .. image:: /content/DS_3D_depth_camera_3d_points_process.png :align: center :alt: DeepStream Depth Camera for 3D point cloud processing overview All the components are configured in YAML format. They are loaded by `Gst-plugins`.There are 3 major components, they may all be loaded into the deepstream pipeline. ``ds3d::dataloader`` - Load Custom Lib for Data Capture --------------------------------------------------------- Load and Manage Dataloader ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader config_body: streams: [color, depth] A custom `dataloader` must have ``type: ds3d::dataloader``. It is created by explicit call of ``NvDs3D_CreateDataLoaderSrc(srcConfig, loaderSrc, start)`` with the full compoment YAML content. During this call, the ``custom_lib_path`` is loaded and a specific data loader is created via ``custom_create_function``. A `GstAppsrc` object is also created into ``loaderSrc.gstElement``. `GstAppsrc` manages the `ds3d::dataloader` dataflows. This ``ds3d::dataloader`` component could be started automatically by `gst-pipeline` or manually by the application call. :: GuardDataLoader dataloader = loaderSrc.customProcessor; ErrCode c = dataloader.start(); To stop the `dataloader`, user can set `GstAppsrc` states to ``GST_STATE_READY`` or stop it manually. :: GuardDataLoader dataloader = loaderSrc.customProcessor; ErrCode c = dataloader.stop(); Dataloader in User Application ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: #include GuardDataLoader dataLoader; dataLoader.reset(createTestTimeDataloader()); # create a specific ABI ErrCode c = dataloader.setErrorCallback([](ErrCode c, const char*){}); c = dataloader.start(yamlconfig); // check result. while(isGood(c)) { GuardDataMap data; c = dataloader.readData(data); // process data } c = dataloader.flush(); c = dataloader.stop(); ``GuardDataLoader`` provides safe access to ``abiDataLoader``. Once it's created, it will maintain the reference pointer to the `dataloader`. Implement a Custom Dataloader ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: #include class TestTimeDataLoader : public ds3d::impl::SyncImplDataLoader { public: TestTimeDataLoader() = default; protected: ErrCode startImpl(const std::string& content, const std::string& path) override { setOutputCaps("ds3d/datamap"); return ErrCode::kGood; } ErrCode readDataImpl(GuardDataMap& datamap) override { datamap.reset(NvDs3d_CreateDataHashMap()); static uint64_t iTime = 0; TimeStamp t{iTime++, 0, 0}; datamap.setData("time", t); emitError(ErrCode::kGood, "timstamp added"); return ErrCode::kGood; } ErrCode stopImpl() override { return ErrCode::kGood; } ErrCode flushImpl() override { return ErrCode::kGood; } }; DS3D_EXTERN_C_BEGIN DS3D_EXPORT_API abiRefDataLoader* createTestTimeDataloader() { return NewAbiRef(new TestTimeDataLoader); } DS3D_EXTERN_C_END A shown in the example above, You'll need to derive `dataloader` from the ``ds3d::impl::SyncImplDataLoader`` class, and implement interfaces for the following: :: ErrCode startImpl(const std::string& content, const std::string& path) override; ErrCode readDataImpl(GuardDataMap& datamap) override; ErrCode stopImpl() override; ErrCode flushImpl() override; To load this custom lib through ``NvDs3D_CreateDataLoaderSrc``, you'll also need to export ``createTestTimeDataloader``. ``ds3d::datafilter``- Loads Custom Lib for Input and Output Processing ----------------------------------------------------------------------- Load And Manage Datafilter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: name: point2cloud_datafilter type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_3d_depth2point_datafilter.so custom_create_function: createDepth2PointFilter config_body: A custom `datafilter` must have ``type: ds3d::datafilter``. It is loaded through the ``nvds3dfilter`` `Gst-plugin`. It is started by ``gst_element_set_state(GST_STATE_READY)``. During this call, the ``custom_lib_path`` is loaded and a specific data filter is created by ``custom_create_function``. ``nvds3dfilter`` `Gst-plugin` has ``config-content`` and ``config-file`` properties. One of them must be set to create a `datafilter` object. Datafilter in User Application ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: #include GuardDataFilter datafilter; datafilter.reset(createTestFakeDatafilter()); # create a specific ABI ErrCode c = datafilter.setErrorCallback([](ErrCode c, const char*){}); c = datafilter.start(yamlconfig); // check result. int consumedNum = 0; auto dataConsumed = [&consumedNum](ErrCode c, const abiRefDataMap* data) { if (isGood(c)) { ++consumedNum; } }; for (int i = 0; i < 100; ++i) { TimeStamp t{i, 0, 0}; GuardDataMap dataIn; dataIn.reset(NvDs3d_CreateDataHashMap()); dataIn.setData("time", t); GuardDataMap dataOut; ErrCode cbCode = ErrCode::kGood; auto outputCB = [&dataOut, &cbCode](ErrCode c, const abiRefDataMap* data) { cbCode = c; if (data) { GuardDataMap newData(*data); dataOut = newData; } }; c = dataFilter.process(dataIn, dataConsumed, outputCB); } c = datafilter.flush(); c = datafilter.stop(); ``GuardDataFilter`` provides safe access to the ``abiDataFilter``. Once it's created, it will maintain the reference pointer to `datafilter`. Implement a Custom Datafilter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: #include class TestFakeDataFilter : public impl::BaseImplDataFilter { public: TestFakeDataFilter() = default; protected: ErrCode startImpl(const std::string& content, const std::string& path) override { setInputCaps(kFakeCapsMetaName); setOutputCaps(kFakeCapsMetaName); return ErrCode::kGood; } ErrCode processImpl( GuardDataMap datamap, OnGuardDataCBImpl outputDataCb, OnGuardDataCBImpl inputConsumedCb) override { DS_ASSERT(datamap); TimeStamp t; ErrCode c = datamap.getData("time", t); if (!isGood(c)) { return c; } t.t0 += 1; inputConsumedCb(ErrCode::kGood, datamap); c = datamap.setData("time", t); if (!isGood(c)) { return c; } outputDataCb(ErrCode::kGood, datamap); return ErrCode::kGood; } ErrCode flushImpl() override { return ErrCode::kGood; } ErrCode stopImpl() override { return ErrCode::kGood; } }; DS3D_EXTERN_C_BEGIN DS3D_EXPORT_API abiRefdatafilter* createTestFakeDatafilter() { return NewAbiRef(new TestFakeDataFilter); } DS3D_EXTERN_C_END As shown in the example above, you'll need to derive the `datafilter` from the ``ds3d::impl::BaseImplDataFilter`` class, and implement interfaces for the following: :: ErrCode startImpl(const std::string& content, const std::string& path) override; ErrCode processImpl( GuardDataMap datamap, OnGuardDataCBImpl outputDataCb, OnGuardDataCBImpl inputConsumedCb) override; ErrCode stopImpl() override; ErrCode flushImpl() override; To load this custom lib through ``nvds3dfilter`` `Gst-plugin`, you'll also need to export a specific symbol ``createTestFakeDatafilter``. ``ds3d::datarender`` - Loads Custom Lib for Data Rendering ------------------------------------------------------------------ Load And Manage Datarender ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: name: point-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createPointCloudDataRender config_body: title: ds3d-point-cloud-test A custom `datarender` must have ``type: ds3d::datarender``. It is created by explicit call of ``NvDs3D_CreateDataRenderSink(sinkConfig, renderSink, start)`` with the full compoment YAML content. During this call, the ``custom_lib_path`` is loaded and a specific data loader is created via ``custom_create_function``. A `GstAppsink` object is also created into ``renderSink.gstElement``. `GstAppsink` manages the `ds3d::datarender` dataflows. This `ds3d::datarender` component could be automatically started by the `gst-pipeline`, or manually by the application call. :: GuardDataRender datarender = renderSink.customProcessor; ErrCode c = datarender.start(); To stop the `datarender`, you can set `GstAppsink` states to ``GST_STATE_READY``, or stop manually. :: GuardDataRender datarender = renderSink.customProcessor; ErrCode c = datarender.stop(); Datarender in User Application ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: #include GuardDataRender datarender; datarender.reset(createTestFakedatarender()); ErrCode c = datarender.setErrorCallback([](ErrCode c, const char*){}); c = datarender.start(yamlconfig); // check result. for (int i = 0; i < 100; ++i) { static uint64_t iTime = 0; TimeStamp t{iTime++, 0, 0}; GuardDataMap datamap; datamap.reset(NvDs3d_CreateDataHashMap()); ASSERT_TRUE(datamap); datamap.setData("time", t); if (i == 0) { c = dataRender.preroll(datamap); } ErrCode c = dataRender.render(datamap, dataRenderd); } c = datarender.flush(); c = datarender.stop(); ``GuardDataRender`` provides safe access to ``abidatarender``. Once it's created, it will maintain the reference pointer to datarender. ``preroll`` is called only once to initialize some resources. Implement a Custom Datarender ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: :: #include class TestFakeDataRender : public impl::BaseImplDataRender { public: TestFakeDataRender() = default; protected: ErrCode startImpl(const std::string& content, const std::string& path) override { setInputCaps("ds3d/datamap"); return ErrCode::kGood; } ErrCode prerollImpl(GuardDataMap datamap) override { return ErrCode::kGood; } ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override { DS_ASSERT(datamap); emitError(ErrCode::kGood, "data rendered"); dataDoneCb(ErrCode::kGood, datamap); return ErrCode::kGood; } ErrCode flushImpl() override { return ErrCode::kGood; } ErrCode stopImpl() override { return ErrCode::kGood; } }; DS3D_EXTERN_C_BEGIN DS3D_EXPORT_API abiRefdatarender* createTestFakedatarender() { return NewAbiRef(new TestFakeDataRender()); } DS3D_EXTERN_C_END As shown in the example above, you'll need to derive `datarender` from the ``ds3d::impl::BaseImplDataRender`` class, and implement interfaces for the following: :: ErrCode startImpl(const std::string& content, const std::string& path) override; ErrCode prerollImpl(GuardDataMap datamap) override; ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override; ErrCode stopImpl() override; ErrCode flushImpl() override; To load this custom lib through ``NvDs3D_CreateDataRenderSink``, you'll also need to export a specific symbol ``createTestFakedatarender``. DS3D ``GuardDataMap`` Buffer Management ------------------------------------------------- DS3D Data Map Read ^^^^^^^^^^^^^^^^^^^^^^^^^ DS3D defines class objects ``ds3d::abiRefDataMap``. All internal data are hash and stored into this data map. ``NvDs3DBuffer`` is defined to store the 3D datamap into `GstBuffer`. Header file is ``nvds3d_meta.h``. :: struct NvDs3DBuffer { uint32_t magicID; // must be 'DS3D' ds3d::abiRefDataMap* datamap; }; .. Warning:: Do not use the `datamap` directly. The easy and safe way to access that is through ``GuardDataMap``. See sample code in :doc: `DS_3D_Depth_Camera`. Example: :: #include #include if (NvDs3D_IsDs3DBuf(gstBuf)) { const abiRefDataMap* refDataMap = nullptr; ErrCode c = NvDs3D_Find1stDataMap(gstBuf, refDataMap); ... // check error code if (refDataMap) { GuardDataMap dataMap(*refDataMap); FrameGuard pointFrame; c = dataMap.getGuardData(kPointXYZ, pointFrame); // get 3D points reference. ... // check error code FrameGuard uvCoord; c = dataMap.getGuardData(kPointCoordUV, uvCoord); // get 3D points UV coordinates reference. ... // check error code Frame2DGuard depthFrame; c = dataMap.getGuardData(kDepthFrame, depthFrame); // get depth frame reference. ... // check error code DepthScale scale; c = dataMap.getData(kDepthScaleUnit, scale); // copy depth scale ... // check error code } } DS3D Data Map Write ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create an empty `datamap`, store some frames into this `datamap`. Example: :: #include #include #include GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // set true to take the reference ownership. /* Create depth frame and store them into ds3d datamap. */ // Assume depth datatype: Uint16 { Frame2DPlane depthPlane = {640, 480, 640 * sizeof(uint16_t) , sizeof(uint16_t), 0}; uint32_t depthBytesPerFrame = depthPlane.pitchInBytes * depthPlane.height; std::vector data(depthBytesPerFrame); // Depth data void* dataPtr = &data[0]; // create depth 2D frame Frame2DGuard depthFrame = Wrap2DFrame( dataPtr, depthPlane}, depthBytesPerFrame, MemType::kCpu, 0, [data = std::move(data)](void*) {}); c = datamap.setGuardData(kDepthFrame, depthFrame); // store depthFrame reference into datamap. ... // check error code DepthScale scale{0.001, {nullptr}}; // c = datamap.setData(kDepthScaleUnit, scale); // copy depth scale into datamap. ... // check error code } /* Create color image frame and store them into ds3d datamap. */ // Assume format is RGBA { Frame2DPlane colorPlane = {1920, 1080, 1920 * sizeof(uint8_t) , sizeof(uint8_t), 0}; uint32_t colorBytesPerFrame = colorPlane.pitchInBytes * colorPlane.height; std::vector data(colorBytesPerFrame); // Image data void* dataPtr = &data[0]; // create color 2D frame Frame2DGuard frame = Wrap2DFrame( dataPtr, {_config.colorPlane}, bytesPerFrame, MemType::kCpu, 0, [data = std::move(data)](void*) {}); c = datamap.setGuardData(kColorFrame, colorFrame); // store colorFrame reference into datamap. ... // check error code } /* Create 3D points frame and store them into ds3d datamap. */ { uint32_t pointNum = 640 * 480; std::vector points(pointNum); // 3D points data vec3f* pointPtr = &points[0]; FrameGuard pointsFrame = wrapPointXYZFrame( (void*)pointPtr, pointNum, MemType::kCpu, 0, [points = std::move(points)](void*) {}); c = datamap.setGuardData(kPointXYZ, pointXyzFrame); // store 3d-points XYZ data reference into datamap. ... // check error code std::vector uvData(pointNum); // 3D points data vec3f* uvPtr = &data[0]; FrameGuard pointUvCoord = wrapPointCoordUVFrame( (void*)uvPtr, pointNum, MemType::kCpu, 0, [uvData = std::move(uvData)](void*) {}); c = datamap.setGuardData(kPointCoordUV, pointUvCoord); // store 3d-points UV coordinate data reference into datamap. ... // check error code } The example below shows how to Create a new `GstBuffer` with ``ds3d`` `datamap`. :: // Assume ``GuardDataMap datamap`` is ready GstBuffer* gstBuf = nullptr; ErrCode c = NvDs3D_CreateGstBuf(gstBuf, datamap.abiRef(), false); // set false to increase reference count. ... // check error code Example below shows how to update an existing ``DS3D`` `GstBuffer` with new ``ds3d`` `datamap`. :: // Assume ``GuardDataMap datamap`` is ready // Assume ``GstBuffer* gstBuf`` is created by another compoment ErrCode c = NvDs3D_UpdateDataMap(gstBuf, datamap.abiRef(), false); // set false to increase reference count. ... // check error code Custom Libs Configuration Specifications --------------------------------------------------------- Components Common Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. csv-table:: ds3d common configuration specifications :file: ../text/tables/DS_3D tables/DS_3D_common_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 libnvds_3d_dataloader_realsense Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Configuration for `Realsense` `Dataloader` Header: :: name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader ``libnvds_3d_dataloader_realsense.so`` requires you to install `librealsense2` SDK. For x86, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md. For Jetson platform, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md. .. csv-table:: libnvds_3d_dataloader_realsense ``config_body`` fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_dataloader_realsense_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 libnvds_3d_depth2point_datafilter Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Configuration for Depth to Points Header: :: name: depth2points type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_3d_depth2point_datafilter.so custom_create_function: createDepth2PointFilter .. csv-table:: libnvds_3d_depth2point_datafilter ``config_body`` fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_depth2point_datafilter_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 libnvds_3d_gl_datarender Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Configuration for Common header: :: name: depth-point-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so Configuration for Body Common Part: .. csv-table:: libnvds_3d_gl_datarender ``config_body`` common fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_gl_datarender_common_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 Configuration for Point Render Header: :: name: point-3D-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createPointCloudDataRender # specific function for 3D point rendering Configuration for Body 3D Point Render Specific Part: For more details on 3D coordinate system, refer to https://learnopengl.com/Getting-started/Coordinate-Systems. To know the value meanings for ``view_position``, ``view_target`` and ``view_up``,refer to the ``gluLookAt`` here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluLookAt.xml. To know the value meanings for ``near``, ``far`` and ``fov``, refer to the ``gluPerspective`` here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluPerspective.xml. .. csv-table:: libnvds_3d_gl_datarender Point Render ``config_body`` Fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_gl_datarender_point_cloud_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 Configuration for Depth and Color 2D Render Header: :: name: depth-2D-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createDepthStreamDataRender # specific function for 2D depth rendering Configuration for Body Depth and Color 2D Specific Part: .. csv-table:: libnvds_3d_gl_datarender 2D Depth Render ``config_body`` Fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_gl_datarender_depth_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 libnvds_3d_depth_datasource Depth file source Specific Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Configuration header: :: name: depthfilesource type: ds3d::dataloader out_caps: ds3d/datamap, framerate=30/1 custom_lib_path: libnvds_3d_depth_datasource.so custom_create_function: createDepthColorLoader Configuration body: .. csv-table:: libnvds_3d_depth_datasource Depth file source ``config_body`` Fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_depth_datasource_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 Intrinsic Parameters in Configuration body: .. csv-table:: libnvds_3d_depth_datasource Intrinsic Parameters in Depth file source ``config_body`` Fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_depth_datasource_intrinsic_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 Extrinsic Parameters in Configuration body: .. csv-table:: libnvds_3d_depth_datasource Extrinsic Parameters in Depth file source ``config_body`` Fields :file: ../text/tables/DS_3D tables/DS_3D_libnvds_3d_depth_datasource_extrinsic_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1