DeepStream-3D Custom Apps and Libs Tutorials

ds3d framework, interfaces and custom-libs are designed for DeepStream-3D processing. ds3d is agnostic from Gstreamer/Glib frameworks. These interfaces are capable of different types of data fusion. Developers can implement different types of custom libraries for dataloader, datafilter and datarender. The interface has ABI compatible layers and modern C++ interface. Developers only need to focus on the modern C++ interface for application or custom lib development.

DS3D dataloader is loaded by GstAppSrc, enabling its utilization for depth cameras like stereo cameras and Time-of-Flight cameras to capture image/depth data or load data from the file system. Additionally, it can be employed for capturing lidar data from sensors or lidar data files. datafilter is loaded by the nvds3dfilter Gst-plugin. It could be used for 2D depth data processing , 3D point-cloud data extraction from depth, other 2D-depth or 3D-points data filters and lidar or 3D data inference. datarender is loaded by GstAppSink. It could be used for 2D depth rendering and 3D point-cloud and lidar data rendering. It also could be used for file dump.

DS3D Application Examples

• DeepStream-3D Multi-Modal Lidar and Camera Sensor Fusion App sample application showcases a multi-modal sensor fusion pipeline for LiDAR and camera data using the DS3D framework. There are 2 examples of multi-modal sensor fusion pipelines inside the samples. The sample app is located at /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion.

  • Refer to DeepStream 3D Multi-Modal Lidar and Camera Sensor Fusion App to see more details of deepstream-3d-lidar-sensor-fusion

  • DS3D BEVFusion pipeline shows the overview of 6-camera plus 1-LiDAR data fusion inference and rendering pipeline through deepstream-3d-lidar-sensor-fusion.

The image below shows the overview of lidar 3D data inference and rendering pipeline in deepstream-lidar-inference.

• deepstream-3d-depth-camera is another example for ds3d pipeline.

The image below shows the overview of depth to 3D point processing pipeline in deepstream-3d-depth-camera.

All the components are configured in YAML format. They are loaded by Gst-plugins.There are 3 major components, they may all be loaded into the deepstream pipeline.

DS3D data format ds3d/datamap

DS3D data format ds3d/datamap

ds3d/datamap recognizes the data format used in DS3D framework. Data buffers flowing between GStreamer plugins will be of this data-type. ds3d/datamap are key-value pairs where key is a string and value, a pointer, structure or tensor frame to data. All of the operations on the buffer is managed by GuardDataMap.

Examples:

• Create GuardDataMap datamap and set scaler key-values.

  #include <ds3d/common/hpp/datamap.hpp>
  using namespace ds3d;
  GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // creat a empty datamap
  TimeStamp ts{0};
  datamap.setData("DS3D::Timestamp", ts); // set timestamp
  float score = 0.1;
  datamap.setData("DS3D::Score", score); // copy score into datamap

• User-defined structure added into ds3d datamap.

  ds3d datamap requires each data type having a typeid: uint64_t before adding the data into datamap. This could prevent any wrong datatype casting in runtime. To achieve that, there are 2 ways to enable a typeid for a structure.

  Example 1, use REGISTER_TYPE_ID directly in custom data structure definition. this is mostly used when users define a new structure.

  #define DS3D_TYPEID_TIMESTAMP 0x20002
  struct TimeStamp {
    uint64_t t0 = 0;
    uint64_t t1 = 0;
    uint64_t t2 = 0;
    REGISTER_TYPE_ID(DS3D_TYPEID_TIMESTAMP)
  };
  // Add the shared_ptr into datamap
  std::shared_ptr<TimeStamp> timePtr(new TimeStamp);
  datamap.setPtrData("DS3D::Timestamp0", timePtr);
  // Add a copy of the TimeStamp into datamap
  TimeStamp time1;
  datamap.setData("DS3D::Timestamp1", time1);

  Example 2, Instantiate template struct TpId<> { static constexpr TIdType __typeid(); }, this is helpful when adding a 3rdparty data structure into datamap and not able to modify the 3rd-party existing DataStructure.

  // this is the 3rd-party structure
  struct Existing3rdpartData {
    float v0;
    int v1;
  };
  // derive ds3d::__TypeID for any 3rdparty data structure.
  #include "ds3d/common/type_trait.h"
  #define DS3D_TYPEID_EXISTING_3RDPART_DATA 0x80001
  namespace ds3d {
  template <>
  struct TpId<Existing3rdpartData>: __TypeID<DS3D_TYPEID_EXISTING_3RDPART_DATA> {};
  }
  // Add the shared_ptr into datamap
  std::shared_ptr<Existing3rdpartData> dataPtr(new Existing3rdpartData);
  datamap.setPtrData("3rdpartyData0", dataPtr);
  // Add a copy of the Existing3rdpartData into datamap
  Existing3rdpartData data3rdparty{0.0f, 0};
  datamap.setData("3rdpartyData1", data3rdparty);

• Create LiDAR tensor frame and add into GuardDataMap datamap.

  std::vector<vec4f> lidardata = {
      {{-5.0f, -5.0f, -3.0f, 0.6f}}, {{-5.0f, -5.0f, -3.0f, 0.85f}},
      {{1.0f, 0.5f, -1.0f, 0.82f}}, {{-5.0f, -5.0f, -3.0f, 0.8f}},
  };
  void* pointPtr = (void*)(&lidardata[0]);
  uint32_t pointsN = lidardata.size();
  FrameGuard lidarFrame = wrapLidarXYZIFrame<float>(
      (void*)pointPtr, pointsN, MemType::kCpu, 0,
  [holder = std::move(lidardata)](void*) {}); // lambda deleter
  const std::string keyName = "DS3D::LidarXYZI"; // user define a key name.
  // setGuardData holds a reference_count on the lidarFrame without deep-copy
  datamap.setGuardData(keyName, lidarFrame);

• Create 2D-Image tensor frame and add into GuardDataMap datamap.

  std::vector<vec4b> imagedata = {
      {{255, 0, 0, 255}}, {{0, 255, 0, 255}}, {{0, 255, 0, 255}},
      {{0, 0, 255, 255}}, {{255, 255, 0, 255}}, {{0, 255, 0, 255}},
  };
  uint32_t width = 3, height = 2;
  Frame2DPlane colorPlane = {width, height, (uint32_t)sizeof(vec4b) * width, sizeof(vec4b), 0};
  void* colorPtr = (void*)(&imagedata[0]);
  uint32_t bytes = colorPlane.pitchInBytes * colorPlane.height;
  Frame2DGuard imageFrame = impl::Wrap2DFrame<uint8_t, FrameType::kColorRGBA>(
    colorPtr, {colorPlane}, bytes, MemType::kCpu, 0,
    [holder = std::move(imagedata)](void*) {}); // lambda deleter
  const std::string keyName = "DS3D::ColorFrame"; // user define a key name.
  // setGuardData holds a reference_count on the imageFrame without deep-copy
  datamap.setGuardData(keyName, imageFrame);

• Create custom shaped tensor frame and add into GuardDataMap datamap.

  std::vector<float> tensordata = {
      0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11.0f
  };
  void* tensorPtr = (void*)(&tensordata[0]);
  Shape tensorShape{3, {1, 3, 4}}; // tensor shape (1, 3, 4)
  uint32_t bytes = tensordata.size() * sizeof(tensordata[0]);
  FrameGuard tensorFrame = impl::WrapFrame<float, FrameType::kCustom>(
          tensorPtr, bytes, tensorShape, MemType::kCpu, 0,
          [holder = std::move(imagedata)](void*) {}); // lambda deleter
  const std::string keyName = "DS3D::dataarray"; // user define a key name.
  // setGuardData holds a reference_count on the tensorFrame without deep-copy
  datamap.setGuardData(keyName, tensorFrame);

• Query data values from GuardDataMap datamap

  GuardDataMap::getGuardData, return a safe reference counted guard data GuardDataT<DataStructure> e.g. Frame2DGuard, FrameGuard, GuardDataT<TimeStamp> GuardDataMap::getPtrData, return a reference counted std::shared_ptr<DataStructure> e.g. std::shared_ptr<TimeStamp> GuardDataMap::getData, return a copy of the DataStructure. e.g. TimeStamp

  // example to use different ways to get timestamp.
  std::string timeKey = "DS3D::Timestamp";
  if (datamap.hasData(timeKey)) {
      // get a COPY of Timestamp
      TimeStamp t0;
      DS_ASSERT(isGood(datamap.getData(timeKey, t0));
      // get a std::shared_ptr of Timestamp
      std::shared_ptr<TimeStamp> tPtr;
      DS_ASSERT(isGood(datamap.getPtrData(timeKey, tPtr));
      // get a reference-counted GuardDataT<Timestamp>
      GuardDataT<TimeStamp> timeGuard;
      DS_ASSERT(isGood(datamap.getPtrData(timeKey, timeGuard));
  // example to get a lidar tensor frame
  FrameGuard pointFrame;
  std::string lidarKey = "DS3D::LidarXYZI";
  if (datamap.hasData(lidarKey)) { // get lidar tensor frame
      // pointFrame holds a reference count
      DS_ASSERT(isGood(dataMap.getGuardData(lidarKey, pointFrame)));
      DataType dType = pointFrame->dataType();
      FrameType frameType = pointFrame->frameType();
      MemType memType = pointFrame->memType();
      Shape pShape = pointFrame->shape();
      void *dataPtr = pointFrame->base();
      size_t dataBytes = pointFrame->bytes();
  }
  // example to get a RGBA image tensor frame
  Frame2DGuard rgbaImage;
  std::string imageKey = "DS3D::ColorFrame";
  if (datamap.hasData(imageKey)) { // get 2D image tensor frame
      // rgbaImage holds a reference count
      DS_ASSERT(isGood(dataMap.getGuardData(imageKey, rgbaImage)));
      DataType dType = rgbaImage->dataType();
      FrameType frameType = rgbaImage->frameType();
      MemType memType = rgbaImage->memType();
      Shape pShape = rgbaImage->shape();
      void *dataPtr = rgbaImage->base();
      size_t dataBytes = rgbaImage->bytes();
      DS_ASSERT(rgbaImage->planes() == 1); // RGBA image has 1 plane.
      Frame2DPlane plane = rgbaImage->getPlane(0);
  }
  // example to get a custom shaped tensor frame, e.g. data_array
  FrameGuard tensorFrame;
  std::string tensorKey = "DS3D::dataarray";
  if (datamap.hasData(tensorKey)) { // get any tensor frame
      // tensorFrame holds a reference count
      DS_ASSERT(isGood(dataMap.getGuardData(tensorKey, tensorFrame)));
      DataType dType = tensorFrame->dataType();
      FrameType frameType = tensorFrame->frameType();
      MemType memType = tensorFrame->memType();
      Shape pShape = tensorFrame->shape();
      void *dataPtr = tensorFrame->base();
      size_t dataBytes = tensorFrame->bytes();
  }

DS3D DataMap interoperate with GstBuffer

DS3D DataMap has ABI-compatible class ds3d::abiRefDataMap. With that, NvDs3DBuffer is defined for storing DS3D datamap along with GstBuffer. Header file is ds3d/gst/nvds3d_meta.h.

struct NvDs3DBuffer {
    uint32_t magicID;  // must be 'DS3D'
    ds3d::abiRefDataMap* datamap;
};

• Example to get datamap from GstBuffer

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
using namespace ds3d;
GstBuffer *gstBuf = ; // get the gstBuf from probe function or Gstreamer plugins
if (NvDs3D_IsDs3DBuf(gstBuf)) {
  const abiRefDataMap* refDataMap = nullptr;
  ErrCode c = NvDs3D_Find1stDataMap(gstBuf, refDataMap);
  if (refDataMap) {
    GuardDataMap dataMap(*refDataMap);
    FrameGuard lidarFrame;
    c = dataMap.getGuardData("DS3D::LidarXYZI", pointFrame); // get lidar points reference.
    FrameGuard uvCoord;
    c = dataMap.getGuardData("DS3D::TextureCoordKey", uvCoord); // get 3D points UV coordinates reference.
    Frame2DGuard depthFrame;
    c = dataMap.getGuardData("DS3D::DepthFrame", depthFrame); // get depth frame reference.
    DepthScale scale;
    c = dataMap.getData("DS3D::DepthScaleUnit", scale); // copy depth scale
  }
}

• Example to create a ds3d::datamap into a new GstBuffer

  #include <ds3d/common/hpp/datamap.hpp>
  #include <ds3d/common/hpp/frame.hpp>
  #include <ds3d/common/impl/impl_frames.h>
  GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // set true to take the reference ownership.
  /* Create color image frame and store them into ds3d datamap. */
  // Assume format is RGBA
  {
    Frame2DPlane colorPlane = {1920, 1080, 1920 * sizeof(uint8_t) , sizeof(uint8_t), 0};
    uint32_t colorBytesPerFrame = colorPlane.pitchInBytes * colorPlane.height;
    std::vector<uint8_t> data(colorBytesPerFrame); // Image data
    void* dataPtr = &data[0];
    // create color 2D frame
    Frame2DGuard frame = Wrap2DFrame<uint8_t, FrameType::kColorRGBA>(
          dataPtr, {_config.colorPlane}, bytesPerFrame, MemType::kCpu, 0,
          [data = std::move(data)](void*) {});
    c = datamap.setGuardData(kColorFrame, colorFrame); // store colorFrame reference into datamap.
    ... // check error code
  }

  Once datamap is ready, you can create a new GstBuffer with DS3D datamap.

  // `GuardDataMap datamap` is ready
  GstBuffer* gstBuf = nullptr;
  ErrCode c = NvDs3D_CreateGstBuf(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
  ... // check error code

• Example to update an existing DS3D GstBuffer with a new ds3d datamap.

  // Assume `GuardDataMap datamap` is ready
  // Assume `GstBuffer* gstBuf` is created by another component
  ErrCode c = NvDs3D_UpdateDataMap(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
    ... // check error code

ds3d::dataloader - Load Custom Lib for Data Capture

ds3d::dataloader - Load Custom Lib for Data Capture

Load and Manage DS3D Dataloader

Examples:

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
config_body:
  streams: [color, depth]

A custom dataloader must have type: ds3d::dataloader. It is created by explicit call of NvDs3D_CreateDataLoaderSrc(srcConfig, loaderSrc, start) with the full component YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsrc object is also created into loaderSrc.gstElement.

GstAppsrc manages the ds3d::dataloader dataflows. This ds3d::dataloader component could be started automatically by gst-pipeline or manually by the application call.

GuardDataLoader dataloader = loaderSrc.customProcessor;
ErrCode c = dataloader.start();

To stop the dataloader, user can set GstAppsrc states to GST_STATE_READY or stop it manually.

GuardDataLoader dataloader = loaderSrc.customProcessor;
ErrCode c = dataloader.stop();

DS3D Dataloader in DeepStream User Application

DS3D Custom Dataloaders are agnostic from Gstreamer/Glib Framework. But it could be working with Gstramer as well. The ds3d::dataloader could interactively work with GstAppSrc together. They could be created by NvDs3D_CreateDataLoaderSrc.

Examples:

#include <ds3d/common/config.h>
#include <ds3d/gst/nvds3d_gst_plugin.h>
std::string yamlStr = R"(
  name: ds3d_lidar_file_source
  type: ds3d::dataloader
  out_caps: ds3d/datamap
  custom_lib_path: libnvds_lidarfileread.so
  custom_create_function: createLidarFileLoader
  config_body:
    data_config_file: lidar_data_list.yaml
    points_num: 242180
    mem_type: gpu # choose [cpu gpu]
    gpu_id: 0
    mem_pool_size: 6
    element_size: 4
    output_datamap_key: DS3D::LidarXYZI
)";
ErrCode c = ErrCode::kGood;
ComponentConfig config;
c = parseComponentConfig(yamlStr.c_str(), "./config_lidar_loader.yaml", config);
DS_ASSERT(config.type == ComponentType::kDataLoader);
DS_ASSERT(config.customLibPath == "libnvds_lidarfileread.so");
DS_ASSERT(config.customCreateFunction == "createLidarFileLoader");
gst::DataLoaderSrc appLoader;
c = NvDs3D_CreateDataLoaderSrc(c, appLoader, true);
// get DS3D custom dataloader.
GuardDataLoader dataLoader = appLoader.customProcessor;
// get GstAppSrc from this appLoader;
GstElement* appsrc = appLoader.gstElement.get();
// gstreamer pipeline setup and running.
// during each read_data from GstAppSrc callback. it would simultaneously reading data from dataloader
// GuardDataMap datamap;
// dataloader.read_data(datamap);
// DS3D dataloader would stop qutomatically when Gstpipeline is stopped.
// But in the case if user want to stop it early or manually.
// Obtain DS3D custom dataloader and stop manually.
c = dataloader.flush();
c = dataloader.stop();

GuardDataLoader provides safe access to abiDataLoader. Once it’s created, it will maintain the reference pointer to the dataloader.

Implement a DS3D Custom Dataloader

Examples:

#include <ds3d/common/impl/impl_dataloader.h>
class TestTimeDataLoader : public ds3d::impl::SyncImplDataLoader {
public:
    TestTimeDataLoader() = default;
protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setOutputCaps("ds3d/datamap");
        return ErrCode::kGood;
    }
    ErrCode readDataImpl(GuardDataMap& datamap) override
    {
        datamap.reset(NvDs3d_CreateDataHashMap());
        static uint64_t iTime = 0;
        TimeStamp t{iTime++, 0, 0};
        datamap.setData("time", t);
        emitError(ErrCode::kGood, "timestamp added");
        return ErrCode::kGood;
    }
    ErrCode stopImpl() override { return ErrCode::kGood; }
    ErrCode flushImpl() override { return ErrCode::kGood; }
};
DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefDataLoader*
createTestTimeDataloader()
{
    return NewAbiRef<abiDataLoader>(new TestTimeDataLoader);
}
DS3D_EXTERN_C_END

A shown in the example above, You’ll need to derive dataloader from the ds3d::impl::SyncImplDataLoader class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode readDataImpl(GuardDataMap& datamap) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

ds3d::databridge - Loads Custom Lib for data conversion to and from DS3D.

ds3d::databridge - Loads Custom Lib for data conversion to and from DS3D.

This plugin and custom lib helps convert data type to and from ds3d/datamap

More details: Gst-nvds3dbridge.

ds3d::datafilter- DS3D Custom DataFilter

ds3d::datafilter- DS3D Custom DataFilter

DS3D DataFilter is processing inputs from ds3d::datamap and producing outputs input new ds3d::datamap. Users can implement custom datafilter lib for their own use case.

Create And Manage DS3D Datafilter in DeepStream App

Examples:

#include <ds3d/common/config.h>
#include <ds3d/gst/nvds3d_gst_ptr.h>
#include <ds3d/gst/nvds3d_gst_plugin.h>
std::string yamlStr = R"(
  name: fusion_inference
  type: ds3d::datafilter
  in_caps: ds3d/datamap
  out_caps: ds3d/datamap
  custom_lib_path: libnvds_tritoninferfilter.so
  custom_create_function: createLidarInferenceFilter
  config_body:
    mem_pool_size: 2
    model_inputs:
    config_file: model_config_files/config_triton_bev_fusion_infer_grpc.pbtxt
    - name: input_image_0
      datatype: UINT8
      shape: [1, 900, 1600, 4]
      from: DS3D::ColorFrame_0+1
      is_2d_frame: true
    - name: input_lidar
      datatype: FP32
      shape: [242180, 4]
      from: DS3D::LidarXYZI+0
)";
gst::ElePtr gstPlugin = gst::elementMake("nvds3dfilter", "ds3d-custom-filter");
g_object_set(G_OBJECT(gstPlugin.get()), "config-content", yamlStr.c_str(), nullptr);
GstElement* ele = gstPlugin.get();

A custom datafilter must have type: ds3d::datafilter. It is loaded through the nvds3dfilter Gst-plugin. It is started by gst_element_set_state(GST_STATE_READY). During this call, the custom_lib_path is loaded and a specific data filter is created by custom_create_function. nvds3dfilter Gst-plugin has config-content and config-file properties. One of them must be set to create a datafilter object.

Implement a Custom DS3D Datafilter

Examples:

#include <ds3d/common/impl/impl_datafilter.h>
class TestFakeDataFilter : public impl::BaseImplDataFilter {
public:
    TestFakeDataFilter() = default;
protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setInputCaps(kFakeCapsMetaName);
        setOutputCaps(kFakeCapsMetaName);
        return ErrCode::kGood;
    }
    ErrCode processImpl(
        GuardDataMap datamap, OnGuardDataCBImpl outputDataCb,
        OnGuardDataCBImpl inputConsumedCb) override
    {
        DS_ASSERT(datamap);
        TimeStamp t;
        ErrCode c = datamap.getData("time", t);
        if (!isGood(c)) {
            return c;
        }
        t.t0 += 1;
        inputConsumedCb(ErrCode::kGood, datamap);
        c = datamap.setData("time", t);
        if (!isGood(c)) {
            return c;
        }
        outputDataCb(ErrCode::kGood, datamap);
        return ErrCode::kGood;
    }
    ErrCode flushImpl() override { return ErrCode::kGood; }
    ErrCode stopImpl() override { return ErrCode::kGood; }
};
DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefdatafilter*
createTestFakeDatafilter()
{
    return NewAbiRef<abidatafilter>(new TestFakeDataFilter);
}
DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive the datafilter from the ds3d::impl::BaseImplDataFilter class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode processImpl(
        GuardDataMap datamap, OnGuardDataCBImpl outputDataCb,
        OnGuardDataCBImpl inputConsumedCb) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through nvds3dfilter Gst-plugin, you’ll also need to export a specific symbol createTestFakeDatafilter.

ds3d::datarender - Loads DS3D Custom DataRender

ds3d::datarender - Loads DS3D Custom DataRender

DS3D Custom DataRenders are agnostic from Gstreamer/Glib Framework. But it could be working with Gstramer as well. The ds3d::datarender could interactively work with GstAppSink together. They could be created by NvDs3D_CreateDataRenderSink.

Examples:

Load And Manage DS3D Datarender

Examples:

#include <ds3d/common/config.h>
#include <ds3d/gst/nvds3d_gst_plugin.h>
std::string yamlStr = R"(

name: lidar_render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createLidarDataRender gst_properties: sync: True async: False drop: False config_body: title: ds3d-lidar-render streams: [lidardata] width: 1280 height: 720 block: True view_position: [0, 0, 60] view_target: [0, 0, 0] view_up: [0, 1, 0] near: 0.3 far: 100 fov: 50 lidar_color: [0, 255, 0] lidar_data_key: DS3D::LidarXYZI element_size: 4 lidar_bbox_key: DS3D::Lidar3DBboxRawData enable_label: True )”;

ErrCode c = ErrCode::kGood; ComponentConfig config; c = parseComponentConfig(yamlStr.c_str(), ”./config_lidar_loader.yaml”, config); DS_ASSERT(config.type == ComponentType::kDataRender); DS_ASSERT(config.customLibPath == “libnvds_3d_gl_datarender.so”); DS_ASSERT(config.customCreateFunction == “createLidarDataRender”);

gst::DataRenderSink appRender; c = NvDs3D_CreateDataRenderSink(config, appRender, true); // get DS3D custom datarender. GuardDataRender datarender = appRender.customProcessor; // get GstAppSink from this appRender; GstElement* appsink = appRender.gstElement.get();

// gstreamer pipeline setup and running. // during each render_data from GstAppSink callback. it would simultaneously reading data from datarender // datarender.render(datamap, consumed_callback);

// DS3D datarender would stop qutomatically when Gstpipeline is stopped. // But in the case if user want to stop it early or manually. // Obtain DS3D custom datarender and stop manually. c = datarender.stop();

A custom datarender must have type: ds3d::datarender. It is created by explicit call of NvDs3D_CreateDataRenderSink(sinkConfig, renderSink, start) with the full component YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsink object is also created into renderSink.gstElement.

GstAppsink manages the ds3d::datarender dataflows. This ds3d::datarender component could be automatically started by the gst-pipeline, or manually by the application call.

GuardDataRender datarender = renderSink.customProcessor;
ErrCode c = datarender.start();

To stop the datarender, you can set GstAppsink states to GST_STATE_READY, or stop manually.

GuardDataRender datarender = renderSink.customProcessor;
ErrCode c = datarender.stop();

GuardDataRender provides safe access to abidatarender. Once it’s created, it will maintain the reference pointer to datarender. preroll is called only once to initialize some resources.

Implement a DS3D Custom Datarender

Examples:

#include <ds3d/common/impl/impl_datarender.h>
class TestFakeDataRender : public impl::BaseImplDataRender {
public:
    TestFakeDataRender() = default;
protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setInputCaps("ds3d/datamap");
        return ErrCode::kGood;
    }
    ErrCode prerollImpl(GuardDataMap datamap) override { return ErrCode::kGood; }
    ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override
    {
        DS_ASSERT(datamap);
        emitError(ErrCode::kGood, "data rendered");
        dataDoneCb(ErrCode::kGood, datamap);
        return ErrCode::kGood;
    }
    ErrCode flushImpl() override { return ErrCode::kGood; }
    ErrCode stopImpl() override { return ErrCode::kGood; }
};
DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefdatarender*
createTestFakedatarender()
{
    return NewAbiRef<abiDataRender>(new TestFakeDataRender());
}
DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive datarender from the ds3d::impl::BaseImplDataRender class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode prerollImpl(GuardDataMap datamap) override;
ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through NvDs3D_CreateDataRenderSink, you’ll also need to export a specific symbol createTestFakedatarender.

Custom Libs Configuration Specifications

Components Common Configuration Specifications

ds3d common configuration specifications

These custom libs are part of DeepStream release package.

Supported DS3D Custom Process libraries

libnvds_tritoninferfilter Configuration Specifications

Multi-modal tensors Triton inference with key-value pairs in ds3d::datamap, Supports user defined custom preprocess and postprocess.

• Configuration for multi-modal triton inference Header:

  name: multimodal_triton_infer
  type: ds3d::datafilter
  in_caps: ds3d/datamap
  out_caps: ds3d/datamap
  custom_lib_path: libnvds_tritoninferfilter.so
  custom_create_function: createLidarInferenceFilter

• Config body specifications

libnvds_tritoninferfilter config_body fields

model_inputs has all input layers information. If there is custom preprocess function/lib specified inside config body. The lib would search from keyname of the input ds3d::datamap, and forward the frame/tensor data into Triton server’s input tensors. The from keyname must be a frame, 2D-frame or tensor inside the ds3d::datamap

• Configuration Specifications of the model_inputs

• Custom preprocess before triton inference

If custom_preprocess_lib_path and custom_preprocess_func_name are specified, Custom processing will be loaded and parse the config body and get all common and user defined information. then process each input ds3d::datamap and generate batchArray for triton inference inputs.

Users can derive interface IInferCustomPreprocessor from sources/includes/ds3d/common/hpp/lidar_custom_process.hpp See examples in sources/libs/ds3d/inference_custom_lib/ds3d_v2x_infer_custom_preprocess/nvinferserver_custom_preprocess.cpp

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
#include <ds3d/common/hpp/lidar_custom_process.hpp>
using namespace ds3d;
using namespace nvdsinferserver;
class NvInferServerCustomPreProcess : public IInferCustomPreprocessor {
public:
  // process key-values from datamap and generate into model inputs batchArray.
  NvDsInferStatus preproc(GuardDataMap &datamap, SharedIBatchArray batchArray, cudaStream_t stream) override {...}
};
extern "C" {
IInferCustomPreprocessor *CreateInferServerCustomPreprocess() {
  return new NvInferServerCustomPreProcess();
} }

• Custom postprocess after triton inference

Postprocessing is quite specific for each model. You’ll need to implement your own postprocessing functions based on inference output tensors results. Custom postprocess libs and functions are specified from nvdsinferserver’s config. for example in apps/sample_apps/deepstream-3d-lidar-sensor-fusion/model_config_files/config_triton_bev_fusion_infer_grpc.pbtxt

infer_config {
  backend {
    triton {
      model_name: "bevfusion"
      grpc {...}
  } }
  extra {
    output_buffer_pool_size: 4
    # specify custom postprocess function
    custom_process_funcion: "Nvds3d_CreateLidarDetectionPostprocess"
  }
  custom_lib {
    # specify custom postprocess library
    path: "libnvds_3d_infer_postprocess_lidar_detection.so"
  }
}

An example of custom postprocess implementation is provided here: sources/libs/ds3d/inference_custom_lib/ds3d_lidar_detection_postprocess/ds3d_infer_postprocess_lidar_detection.cpp

#include "infer_custom_process.h"
#include <ds3d/common/hpp/frame.hpp>
#include <ds3d/common/hpp/datamap.hpp>
using namespace ds3d;
using namespace nvdsinferserver;
class DS3DTritonLidarInferCustomPostProcess : public IInferCustomProcessor {
public:
  // process key-values from datamap and generate into model inputs batchArray.
  NvDsInferStatus inferenceDone(const IBatchArray* batchArray, const IOptions* inOptions) override {
   ...
    // get ``ds3d::datamap`` from ``inOptions``
    abiRefDataMap* refDataMap = nullptr;
    if (inOptions->hasValue(kLidarRefDataMap)) {
        INFER_ASSERT(inOptions->getObj(kLidarRefDataMap, refDataMap) == NVDSINFER_SUCCESS);
    }
    GuardDataMap dataMap(*refDataMap);
    ...
    // parsing output tensors from batchArray
    TensorMap outTensors;
    for (uint32_t i = 0; i < batchArray->getSize(); ++i) {
        auto buf = batchArray->getSafeBuf(i);
        outTensors[buf->getBufDesc().name] = buf;
    }
    std::vector<Lidar3DBbox> bboxes;
    ret = parseLidar3Dbbox(outTensors, bboxes);
    // warp data into ds3d frame ``bboxFrame``
    size_t bufBytes = sizeof(Lidar3DBbox) * bboxes.size();
    void* bufBase = (void*)bboxes.data();
    Shape shape{3, {1, (int)bboxes.size(), sizeof(Lidar3DBbox)}};
    FrameGuard bboxFrame = impl::WrapFrame<uint8_t, FrameType::kCustom>(
        bufBase, bufBytes, shape, MemType::kCpu, 0, [outdata = std::move(bboxes)](void*) {});
    // add key-value frame into ds3d::datamap
    ErrCode code = dataMap.setGuardData(_3dBboxKey, bboxFrame);
    ...
    return ret;
  }
};
extern "C" {
IInferCustomProcessor *Nvds3d_CreateLidarDetectionPostprocess() {
  return new DS3DTritonLidarInferCustomPostProcess();
} }

Custom Datafilter libnvds_3d_alignment_datafilter Specifications

Data alignment for lidar and video data can be done using a custom library provided with DeepStreamSDK.

More details on this is here: Custom ds3d::datafilter library: libnvds_3d_alignment_datafilter.so.

Custom Datafilter libnvds_3d_lidar_preprocess_datafilter Specifications

LiDAR data preprocess into voxel data format before V2X model sensor fusion inference. Source files located at /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/datafilter/lidar_preprocess

• Configuration for Lidar data preprocess ds3d::datafilter Header:

  name: lidarpreprocess
  type: ds3d::datafilter
  out_caps: ds3d/datamap
  custom_lib_path: libnvds_3d_lidar_preprocess_datafilter.so
  custom_create_function: createLidarPreprocessFilter

• Config Body Specifications

For example:

config_body:
mem_pool_size: 4
filter_input_datamap_key: DS3D::LidarXYZI_0
model_inputs:
- name: feats
  datatype: FP16
  shape: [4, 8000, 10, 9]
- name: coords
  datatype: INT32
  shape: [4, 8000, 4]
- name: N
  datatype: INT32
  shape: [4, 1]
gpu_id: 0
input_tensor_mem_type: GpuCuda
lidar_data_from: [DS3D::LidarXYZI_0, DS3D::LidarXYZI_1, DS3D::LidarXYZI_2, DS3D::LidarXYZI_3]
output_features_tensor_key: DS3D::LidarFeatureTensor
output_coords_tensor_key: DS3D::LidarCoordTensor
output_num_tensor_key: DS3D::LidarPointNumTensor

Custom Dataloader libnvds_lidarfileread Configuration Specifications

The lib reads lidar data file frame by frame, it creates a new ds3d::datamap per frame and deliver to next component. Source files located at /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/dataloader/lidarsource

• Configuration for Lidar file reader ds3d::dataloader Header:

  name: ds3d_lidar_file_source
  type: ds3d::dataloader
  out_caps: ds3d/datamap
  custom_lib_path: libnvds_lidarfileread.so
  custom_create_function: createLidarFileLoader

• Config Body Specifications

For example:

config_body:
  data_config_file: lidar_nuscene_data_list.yaml
  points_num: 242180
  fixed_points_num: True
  lidar_datatype: FP32
  mem_type: gpu
  gpu_id: 0
  mem_pool_size: 6
  element_size: 4
  output_datamap_key: DS3D::LidarXYZI
  file_loop: True

• lidar data config file

The lib reads frames from data_config_file which is a separate file containers multiple lidar files. each lidar file is a single lidar frame file which are listed inside of source-list. Each item’s keyname is the timestamp(millisecond) of the lidar frame.

An example of data_config_file.

source-list:
  - 0: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/data/nuscene/LIDAR_TOP/000000-LIDAR_TOP.bin
  - 50: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/data/nuscene/LIDAR_TOP/000001-LIDAR_TOP.bin
  - 100: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/data/nuscene/LIDAR_TOP/000002-LIDAR_TOP.bin

Custom Dataloader libnvds_3d_dataloader_realsense Configuration Specifications

Configuration for Realsense Dataloader Header:

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

libnvds_3d_dataloader_realsense.so requires you to install librealsense2 SDK. For x86, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md. For Jetson platform, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md.

libnvds_3d_dataloader_realsense config_body fields

Custom datafilter libnvds_3d_depth2point_datafilter Configuration Specifications

Convert 2D depth data into 3D point-cloud(XYZ) data into ds3d::datamap

• Configuration for Depth to Points Header:

name: depth2points
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter

libnvds_3d_depth2point_datafilter config_body fields

Custom datarender libnvds_3d_gles_ensemble_render Configuration Specifications

Renders 3D scene using OpenGL ES (GLES) with various elements (textures, LiDAR points, bounding boxes) within a single window, allowing for flexible layout customization. Users can split the windows into multiple views, and project each tensor data or frame from ds3d::datamap into a separate view or into multiple views at the same time. It could also support overlay if multi frames rendered into a single view position overlap, it depends on the render graph order.

• Configuration Header

name: ds3d_sensor_fusion_render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gles_ensemble_render.so
custom_create_function: NvDs3D_CreateGlesEnsembleRender

• Configuration Body

An example of a multi-view config with render_graph. There are 2 views rendered with different frames inside the same ds3d::datamap. The 2D color image is rendered by texture3d_render into view area [0, 0, 640, 360]; The Lidar data is rendered by lidar3d_render into view area [640, 0, 1280, 360].

config_body:
  window_width: 1280 # window size
  window_height: 360 # window size
  color_clear: true
  window_title: DS3D-Lidar-6-Cameras-BEVFusion
  render_graph:
    - texture3d_render: # 2D texture view
        layout: [0, 0, 640, 360] # layout [x0, y0, x1, y1]
        max_vertex_num: 6
        color_clear: false
        texture_frame_key: DS3D::ColorFrame_2 # image data key
    - lidar3d_render: # lidar top view
        layout: [640, 0, 1280, 360] # layout [x0, y0, x1, y1]
        color_clear: false
        view_position: [0, 0, 30]
        view_target: [0, 0, 0]
        view_up: [0, 1, 0]
        lidar_color: [0, 0, 255]
        lidar_data_key: DS3D::LidarXYZI
        lidar_bbox_key: DS3D::Lidar3DBboxRawData
        element_size: 4

libnvds_3d_gles_ensemble_render config_body fields

• Render graph render_graph

It supports 2 different modes of views

• texture3d_render, it renders 2D image data into a specified view area.

• lidar3d_render, it renders lidar data and 3D bbox data into a specified view area.

Each mode could be configured multiple times in the same render_graph. Multiple camera data inside a single ds3d::datamap could be configured into different view area. Similarly, multiple lidar data insdie the same ds3d::datamap could also be configured at different viewpoint(topview, front-view, side-view) into different view area.

• Configuration Specifications of texture3d_render

libnvds_3d_gles_ensemble_render texture3d_render fields

• Configuration Specifications of lidar3d_render

libnvds_3d_gles_ensemble_render lidar3d_render fields

Custom datarender libnvds_3d_gl_datarender Configuration Specifications

Configuration Common header for libnvds_3d_gl_datarender:

name: depth-point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so

Configuration Body for Common Part:

libnvds_3d_gl_datarender config_body common fields

• Configuration Header for Point Cloud Render:

name: point-3D-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender # specific function for 3D point rendering

• Configuration Header for Lidar data Render:

name: lidar-data-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createLidarDataRender # specific function for Lidar point cloud rendering

• Configuration Body for 3D Point Cloud and Lidar Render Part:

For more details on 3D coordinate system, refer to https://learnopengl.com/Getting-started/Coordinate-Systems. To know the value meanings for view_position, view_target and view_up,refer to the gluLookAt here: https://registry.khronos.org/OpenGL-Refpages/gl2.1/xhtml/gluLookAt.xml. To know the value meanings for near, far and fov, refer to the gluPerspective here: https://registry.khronos.org/OpenGL-Refpages/gl2.1/xhtml/gluPerspective.xml.

libnvds_3d_gl_datarender Point Cloud Render config_body Fields

• Configuration Body for Lidar Render Specific Part:

libnvds_3d_gl_datarender Lidar Render extra config_body Fields

• Configuration Header for Depth and Color 2D Render:

name: depth-2D-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender # specific function for 2D depth rendering

• Configuration Body for Depth and Color 2D Specific Part:

libnvds_3d_gl_datarender 2D Depth Render config_body Fields

libnvds_3d_depth_datasource Depth file source Specific Configuration Specifications

Configuration header:

name: depthfilesource
type: ds3d::dataloader
out_caps: ds3d/datamap, framerate=30/1
custom_lib_path: libnvds_3d_depth_datasource.so
custom_create_function: createDepthColorLoader

Configuration body:

libnvds_3d_depth_datasource Depth file source config_body Fields

Configuration Body for Intrinsic Parameters :

libnvds_3d_depth_datasource Intrinsic Parameters in Depth file source config_body Fields

Configuration Body for Extrinsic Parameters:

libnvds_3d_depth_datasource Extrinsic Parameters in Depth file source config_body Fields