DeepStream 3D Custom Manual¶

New ds3d framework, interfaces and custom-libs are defined for DS 3D processing. These interfaces are capble of different types of data fusion and can implement different types of custom libraries for dataloader, datafilter and datarender. The interface has ABI compatible layers and modern C++ interface layers. You’ll only need to focus on the modern interface for application or custom lib development.

DeepStream 3D dataloader is loaded by GstAppSrc. It could be used for depth camera such as stereo cameras, Time-of-Flight cameras to capture image/depth data or 2D/3D data-load from the source file., It also could be used for lidar data sensor or lidar data file capture. datafilter is loaded by the nvds3dfilter Gst-plugin. It could be used for 2D depth data processing , 3D point-cloud data extraction from depth, other 2D-depth or 3D-points data filters and lidar or 3D data inference. datarender is loaded by GstAppSink. It could be used for 2D depth rendering and 3D point-cloud and lidar data rendering. It also could be used for file dump.

DS3D Application Examples¶

deepstream-lidar-inference has the sample code to load these custom libs and to connect these components together in simple ways. Besides that, DS3D has a simple C++ safe pointer for Gstreamer components. The interfaces are found in header files located at /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/gst/.

The image below shows the overview of lidar 3D data inference and rendering pipeline in deepstream-lidar-inference.

See more details in the DeepStream Lidar Inference App (Alpha).

deepstream-3d-depth-camera is another example for ds3d pipepline.

The image below shows the overview of depth to 3D point processing pipeline in deepstream-3d-depth-camera.

See more details in the DeepStream 3D Depth Camera App.

All the components are configured in YAML format. They are loaded by Gst-plugins.There are 3 major components, they may all be loaded into the deepstream pipeline.

`ds3d::dataloader` - Load Custom Lib for Data Capture¶

Load and Manage Dataloader¶

Examples:

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
config_body:
  streams: [color, depth]

A custom dataloader must have type: ds3d::dataloader. It is created by explicit call of NvDs3D_CreateDataLoaderSrc(srcConfig, loaderSrc, start) with the full compoment YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsrc object is also created into loaderSrc.gstElement.

GstAppsrc manages the ds3d::dataloader dataflows. This ds3d::dataloader component could be started automatically by gst-pipeline or manually by the application call.

GuardDataLoader dataloader = loaderSrc.customProcessor;
ErrCode c = dataloader.start();

To stop the dataloader, user can set GstAppsrc states to GST_STATE_READY or stop it manually.

GuardDataLoader dataloader = loaderSrc.customProcessor;
ErrCode c = dataloader.stop();

Dataloader in User Application¶

Examples:

#include <ds3d/common/hpp/dataloader.hpp>
GuardDataLoader dataLoader;
dataLoader.reset(createTestTimeDataloader()); # create a specific ABI
ErrCode c = dataloader.setErrorCallback([](ErrCode c, const char*){});
c = dataloader.start(yamlconfig); // check result.
while(isGood(c)) {
  GuardDataMap data;
  c = dataloader.readData(data);
  // process data
}
c = dataloader.flush();
c = dataloader.stop();

GuardDataLoader provides safe access to abiDataLoader. Once it’s created, it will maintain the reference pointer to the dataloader.

Implement a Custom Dataloader¶

Examples:

#include <ds3d/common/impl/impl_dataloader.h>
class TestTimeDataLoader : public ds3d::impl::SyncImplDataLoader {
public:
    TestTimeDataLoader() = default;

protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setOutputCaps("ds3d/datamap");
        return ErrCode::kGood;
    }
    ErrCode readDataImpl(GuardDataMap& datamap) override
    {
        datamap.reset(NvDs3d_CreateDataHashMap());
        static uint64_t iTime = 0;
        TimeStamp t{iTime++, 0, 0};
        datamap.setData("time", t);
        emitError(ErrCode::kGood, "timstamp added");
        return ErrCode::kGood;
    }
    ErrCode stopImpl() override { return ErrCode::kGood; }
    ErrCode flushImpl() override { return ErrCode::kGood; }
};

DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefDataLoader*
createTestTimeDataloader()
{
    return NewAbiRef<abiDataLoader>(new TestTimeDataLoader);
}
DS3D_EXTERN_C_END

A shown in the example above, You’ll need to derive dataloader from the ds3d::impl::SyncImplDataLoader class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode readDataImpl(GuardDataMap& datamap) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through NvDs3D_CreateDataLoaderSrc, you’ll also need to export createTestTimeDataloader.

`ds3d::datafilter`- Loads Custom Lib for Input and Output Processing¶

Load And Manage Datafilter¶

Examples:

name: point2cloud_datafilter
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter
config_body:

A custom datafilter must have type: ds3d::datafilter. It is loaded through the nvds3dfilter Gst-plugin. It is started by gst_element_set_state(GST_STATE_READY). During this call, the custom_lib_path is loaded and a specific data filter is created by custom_create_function. nvds3dfilter Gst-plugin has config-content and config-file properties. One of them must be set to create a datafilter object.

Datafilter in User Application¶

Examples:

#include <ds3d/common/hpp/datafilter.hpp>
GuardDataFilter datafilter;
datafilter.reset(createTestFakeDatafilter()); # create a specific ABI
ErrCode c = datafilter.setErrorCallback([](ErrCode c, const char*){});
c = datafilter.start(yamlconfig); // check result.
int consumedNum = 0;
auto dataConsumed = [&consumedNum](ErrCode c, const abiRefDataMap* data) {
    if (isGood(c)) {
        ++consumedNum;
    }
};
for (int i = 0; i < 100; ++i) {
  TimeStamp t{i, 0, 0};
  GuardDataMap dataIn;
  dataIn.reset(NvDs3d_CreateDataHashMap());
  dataIn.setData("time", t);

  GuardDataMap dataOut;
  ErrCode cbCode = ErrCode::kGood;
  auto outputCB = [&dataOut, &cbCode](ErrCode c, const abiRefDataMap* data) {
      cbCode = c;
      if (data) {
          GuardDataMap newData(*data);
          dataOut = newData;
      }
  };
  c = dataFilter.process(dataIn, dataConsumed, outputCB);
}
c = datafilter.flush();
c = datafilter.stop();

GuardDataFilter provides safe access to the abiDataFilter. Once it’s created, it will maintain the reference pointer to datafilter.

Implement a Custom Datafilter¶

Examples:

#include <ds3d/common/impl/impl_datafilter.h>
class TestFakeDataFilter : public impl::BaseImplDataFilter {
public:
    TestFakeDataFilter() = default;

protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setInputCaps(kFakeCapsMetaName);
        setOutputCaps(kFakeCapsMetaName);
        return ErrCode::kGood;
    }
    ErrCode processImpl(
        GuardDataMap datamap, OnGuardDataCBImpl outputDataCb,
        OnGuardDataCBImpl inputConsumedCb) override
    {
        DS_ASSERT(datamap);
        TimeStamp t;
        ErrCode c = datamap.getData("time", t);
        if (!isGood(c)) {
            return c;
        }
        t.t0 += 1;
        inputConsumedCb(ErrCode::kGood, datamap);
        c = datamap.setData("time", t);
        if (!isGood(c)) {
            return c;
        }
        outputDataCb(ErrCode::kGood, datamap);
        return ErrCode::kGood;
    }

    ErrCode flushImpl() override { return ErrCode::kGood; }
    ErrCode stopImpl() override { return ErrCode::kGood; }
};

DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefdatafilter*
createTestFakeDatafilter()
{
    return NewAbiRef<abidatafilter>(new TestFakeDataFilter);
}
DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive the datafilter from the ds3d::impl::BaseImplDataFilter class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode processImpl(
        GuardDataMap datamap, OnGuardDataCBImpl outputDataCb,
        OnGuardDataCBImpl inputConsumedCb) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through nvds3dfilter Gst-plugin, you’ll also need to export a specific symbol createTestFakeDatafilter.

`ds3d::datarender` - Loads Custom Lib for Data Rendering¶

Load And Manage Datarender¶

Examples:

name: point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender
config_body:
  title: ds3d-point-cloud-test

A custom datarender must have type: ds3d::datarender. It is created by explicit call of NvDs3D_CreateDataRenderSink(sinkConfig, renderSink, start) with the full compoment YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsink object is also created into renderSink.gstElement.

GstAppsink manages the ds3d::datarender dataflows. This ds3d::datarender component could be automatically started by the gst-pipeline, or manually by the application call.

GuardDataRender datarender = renderSink.customProcessor;
ErrCode c = datarender.start();

To stop the datarender, you can set GstAppsink states to GST_STATE_READY, or stop manually.

GuardDataRender datarender = renderSink.customProcessor;
ErrCode c = datarender.stop();

Datarender in User Application¶

Examples:

#include <ds3d/common/hpp/datarender.hpp>
GuardDataRender datarender;
datarender.reset(createTestFakedatarender());
ErrCode c = datarender.setErrorCallback([](ErrCode c, const char*){});
c = datarender.start(yamlconfig); // check result.
for (int i = 0; i < 100; ++i) {
  static uint64_t iTime = 0;
  TimeStamp t{iTime++, 0, 0};
  GuardDataMap datamap;
  datamap.reset(NvDs3d_CreateDataHashMap());
  ASSERT_TRUE(datamap);
  datamap.setData("time", t);
  if (i == 0) {
      c = dataRender.preroll(datamap);
  }
  ErrCode c = dataRender.render(datamap, dataRenderd);
}
c = datarender.flush();
c = datarender.stop();

GuardDataRender provides safe access to abidatarender. Once it’s created, it will maintain the reference pointer to datarender. preroll is called only once to initialize some resources.

Implement a Custom Datarender¶

Examples:

#include <ds3d/common/impl/impl_datarender.h>
class TestFakeDataRender : public impl::BaseImplDataRender {
public:
    TestFakeDataRender() = default;

protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setInputCaps("ds3d/datamap");
        return ErrCode::kGood;
    }
    ErrCode prerollImpl(GuardDataMap datamap) override { return ErrCode::kGood; }
    ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override
    {
        DS_ASSERT(datamap);
        emitError(ErrCode::kGood, "data rendered");
        dataDoneCb(ErrCode::kGood, datamap);
        return ErrCode::kGood;
    }
    ErrCode flushImpl() override { return ErrCode::kGood; }
    ErrCode stopImpl() override { return ErrCode::kGood; }
};

DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefdatarender*
createTestFakedatarender()
{
    return NewAbiRef<abiDataRender>(new TestFakeDataRender());
}
DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive datarender from the ds3d::impl::BaseImplDataRender class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode prerollImpl(GuardDataMap datamap) override;
ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through NvDs3D_CreateDataRenderSink, you’ll also need to export a specific symbol createTestFakedatarender.

DS3D `GuardDataMap` Buffer Management¶

DS3D Data Map Read¶

DS3D defines class objects ds3d::abiRefDataMap. All internal data are hash and stored into this data map. NvDs3DBuffer is defined to store the 3D datamap into GstBuffer. Header file is nvds3d_meta.h.

struct NvDs3DBuffer {
    uint32_t magicID;  // must be 'DS3D'
    ds3d::abiRefDataMap* datamap;
};

Warning

Do not use the datamap directly. The easy and safe way to access that is through GuardDataMap. See sample code in :doc: DS_3D_Depth_Camera.

Example:

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
if (NvDs3D_IsDs3DBuf(gstBuf)) {
  const abiRefDataMap* refDataMap = nullptr;
  ErrCode c = NvDs3D_Find1stDataMap(gstBuf, refDataMap);
  ... // check error code
  if (refDataMap) {
    GuardDataMap dataMap(*refDataMap);
    FrameGuard pointFrame;
    c = dataMap.getGuardData(kPointXYZ, pointFrame); // get 3D points reference.
    ... // check error code

    FrameGuard uvCoord;
    c = dataMap.getGuardData(kPointCoordUV, uvCoord); // get 3D points UV coordinates reference.
    ... // check error code

    Frame2DGuard depthFrame;
    c = dataMap.getGuardData(kDepthFrame, depthFrame); // get depth frame reference.
    ... // check error code

    DepthScale scale;
    c = dataMap.getData(kDepthScaleUnit, scale); // copy depth scale
    ... // check error code
  }

}

DS3D Data Map Write¶

Create an empty datamap, store some frames into this datamap. Example:

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
#include <ds3d/common/impl/impl_frames.h>
GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // set true to take the reference ownership.

/* Create depth frame and store them into ds3d datamap. */
// Assume depth datatype: Uint16
{
  Frame2DPlane depthPlane = {640, 480, 640 * sizeof(uint16_t) , sizeof(uint16_t), 0};
  uint32_t depthBytesPerFrame = depthPlane.pitchInBytes * depthPlane.height;
  std::vector<uint8_t> data(depthBytesPerFrame); // Depth data
  void* dataPtr = &data[0];
  // create depth 2D frame
  Frame2DGuard depthFrame = Wrap2DFrame<uint16_t, FrameType::kDepth>(
          dataPtr, depthPlane}, depthBytesPerFrame, MemType::kCpu, 0,
          [data = std::move(data)](void*) {});
  c = datamap.setGuardData(kDepthFrame, depthFrame); // store depthFrame reference into datamap.
  ... // check error code

  DepthScale scale{0.001, {nullptr}}; //
  c = datamap.setData(kDepthScaleUnit, scale); // copy depth scale into datamap.
  ... // check error code
}

/* Create color image frame and store them into ds3d datamap. */
// Assume format is RGBA
{
  Frame2DPlane colorPlane = {1920, 1080, 1920 * sizeof(uint8_t) , sizeof(uint8_t), 0};
  uint32_t colorBytesPerFrame = colorPlane.pitchInBytes * colorPlane.height;
  std::vector<uint8_t> data(colorBytesPerFrame); // Image data
  void* dataPtr = &data[0];
  // create color 2D frame
  Frame2DGuard frame = Wrap2DFrame<uint8_t, FrameType::kColorRGBA>(
        dataPtr, {_config.colorPlane}, bytesPerFrame, MemType::kCpu, 0,
        [data = std::move(data)](void*) {});
  c = datamap.setGuardData(kColorFrame, colorFrame); // store colorFrame reference into datamap.
  ... // check error code
}

/* Create 3D points frame and store them into ds3d datamap. */
{
  uint32_t pointNum = 640 * 480;
  std::vector<vec3f> points(pointNum); // 3D points data
  vec3f* pointPtr = &points[0];
  FrameGuard pointsFrame = wrapPointXYZFrame<float>(
    (void*)pointPtr, pointNum, MemType::kCpu, 0, [points = std::move(points)](void*) {});
  c = datamap.setGuardData(kPointXYZ, pointXyzFrame); // store 3d-points XYZ data reference into datamap.
  ... // check error code


  std::vector<vec3f> uvData(pointNum); // 3D points data
  vec3f* uvPtr = &data[0];
  FrameGuard pointUvCoord = wrapPointCoordUVFrame<float>(
    (void*)uvPtr, pointNum, MemType::kCpu, 0, [uvData = std::move(uvData)](void*) {});
  c = datamap.setGuardData(kPointCoordUV, pointUvCoord); // store 3d-points UV coordinate data reference into datamap.
  ... // check error code
}

The example below shows how to Create a new GstBuffer with ds3d datamap.

// Assume ``GuardDataMap datamap`` is ready
GstBuffer* gstBuf = nullptr;
ErrCode c = NvDs3D_CreateGstBuf(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
... // check error code

Example below shows how to update an existing DS3D GstBuffer with new ds3d datamap.

// Assume ``GuardDataMap datamap`` is ready
// Assume ``GstBuffer* gstBuf`` is created by another compoment
ErrCode c = NvDs3D_UpdateDataMap(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
... // check error code

Custom Libs Configuration Specifications¶

Components Common Configuration Specifications¶

ds3d common configuration specifications¶
Property	Meaning	Type and Range	Example
type	Custom processor type	String, [ds3d::dataloader, ds3d::datafilter, ds3d::datarender]	type: ds3d::dataloader
name	Indicate user-defined component name	String	name: depthloader
in_caps	Indicate Gst sink caps for the component	String	in_caps: ds3d/datamap
out_caps	Indicate Gst sink caps for the component	String	out_caps: ds3d/datamap
custom_lib_path	Indicate custom lib path	String	custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function	Indicate custom function to create the specific ds3d processing component	String	custom_create_function: createPointCloudDataRender
config_body	Indicate YAML specific content for the custom comonent	String	config_body: in_streams: [color, depth] max_points: 407040

libnvds_3d_dataloader_realsense Configuration Specifications¶

Configuration for Realsense Dataloader Header:

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

libnvds_3d_dataloader_realsense.so requires you to install librealsense2 SDK. For x86, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md. For Jetson platform, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md.

libnvds_3d_dataloader_realsense `config_body` fields¶
Property	Meaning	Type and Range	Example
streams	Specify which streams to enable	List[String], select from [color, depth]	streams: [color, depth]
aligned_image_to_depth	Indicate whether color image is aligned to depth	Boolean	aligned_image_to_depth: False

libnvds_3d_depth2point_datafilter Configuration Specifications¶

Configuration for Depth to Points Header:

name: depth2points
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter

libnvds_3d_depth2point_datafilter `config_body` fields¶
Property	Meaning	Type and Range	Example
streams	Specify which streams to enable	List[String], select from [color, depth]	streams: [color, depth]
max_points	Indicate maximum 3d points to allocate	Uint32	max_points: 407040
mem_pool_size	Indicate max buffer pool size	Uint32	mem_pool_size: 8

libnvds_3d_gl_datarender Configuration Specifications¶

Configuration Common header for libnvds_3d_gl_datarender:

name: depth-point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so

Configuration Body for Common Part:

libnvds_3d_gl_datarender `config_body` common fields¶
Property	Meaning	Type and Range	Example
title	Specify window title	String	title: ds3d-point-cloud-test
streams	Indicate which streams to render. depth render must have [depth], 3D points render must have [points]	List[String], select from [color, depth, points]	streams: [color, depth]
width	Specify window width	UINT32	width: 1280
height	Specify window height	UINT32	height: 720
block	Indicate rendering thread as block mode	Boolean	block: True

Configuration Header for Point Cloud Render:

name: point-3D-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender # specific function for 3D point rendering

Configuration Header for Lidar data Render:

name: lidar-data-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createLidarDataRender # specific function for Lidar point cloud rendering

Configuration Body for 3D Point Cloud and Lidar Render Part:

For more details on 3D coordinate system, refer to https://learnopengl.com/Getting-started/Coordinate-Systems. To know the value meanings for view_position, view_target and view_up,refer to the gluLookAt here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluLookAt.xml. To know the value meanings for near, far and fov, refer to the gluPerspective here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluPerspective.xml.

libnvds_3d_gl_datarender Point Cloud Render `config_body` Fields¶
Property	Meaning	Type and Range	Example
view_position	Specify view position [x, y, z]coordinates	List[Float]	view_position: [0, 0, -1]
view_target	Specify view target [x, y, z]coordinates	List[Float]	view_target: [0, 0, 1]
view_up	Specify view up direction [x, y, z]coordinates	List[Float]	view_up: [0, -1.0, 0]
near	Specify perspective projection near plane	Float	near: 0.01
far	Specify perspective projection far plane	Float	far: 10.0
fov	Specify perspective projection field of view, degree angle	Float	fov: 40.0
coord_y_opposite	Specify texture map V direction, Realsense coordinates is different from GLES default coordinates	Boolean	coord_y_opposite: False
positive_z_only	Specify whether display negtive depth values	Boolean	positive_z_only: False

Configuration Body for Lidar Render Specific Part:

libnvds_3d_gl_datarender Lidar Render extra `config_body` Fields¶
Property	Meaning	Type and Range	Example
view_position	Specify view position [x, y, z]coordinates	List[Float]	view_position: [0, 0, -1]
view_target	Specify view target [x, y, z]coordinates	List[Float]	view_target: [0, 0, 1]
view_up	Specify view up direction [x, y, z]coordinates	List[Float]	view_up: [0, -1.0, 0]
near	Specify perspective projection near plane	Float	near: 0.01
far	Specify perspective projection far plane	Float	far: 10.0
fov	Specify perspective projection field of view, degree angle	Float	fov: 40.0
lidar_color	Specify lidar data color for display	List[Uint32]	lidar_color: [0, 255, 0]
element_size	Specify lidar data element size. e.g. 4 for XYZI or 3 for XYZ	Uint32	element_size: 4
lidar_data_key	Specify lidar data frame in datamap, default value is DS3D::LidarXYZI	String	lidar_data_key: DS3D::LidarXYZI
lidar_bbox_key	Specify lidar 3D bounding box data in datamap, default value is DS3D::Lidar3DBboxRawData	String	lidar_bbox_key: DS3D::Lidar3DBboxRawData

Configuration Header for Depth and Color 2D Render:

name: depth-2D-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender # specific function for 2D depth rendering

Configuration Body for Depth and Color 2D Specific Part:

libnvds_3d_gl_datarender 2D Depth Render `config_body` Fields¶
Property	Meaning	Type and Range	Example
min_depth	Specify minimum depth value. other values less that it will be removed in rendering	Float	min_depth: 0.3
max_depth	Specify maximum depth value. other values less that it will be removed in rendering	Float	max_depth: 2.0
min_depth_color	Specify minimum depth rendering color in [R, G, B]	List[Uint32]	min_depth_color: [255, 128, 0]
max_depth_color	Specify maximum depth rendering color in [R, G, B]	Float	max_depth_color: [0, 128, 255]

libnvds_3d_depth_datasource Depth file source Specific Configuration Specifications¶

Configuration header:

name: depthfilesource
type: ds3d::dataloader
out_caps: ds3d/datamap, framerate=30/1
custom_lib_path: libnvds_3d_depth_datasource.so
custom_create_function: createDepthColorLoader

Configuration body:

libnvds_3d_depth_datasource Depth file source `config_body` Fields¶
Property	Meaning	Type and Range	Example
depth_source	Specify file path for depth source	String	depth_source: depth_uint16_640x480.bin
color_source	Specify file path for color image source	String	color_source: color_rgba_1920x1080.bin
depth_scale	Indicate depth unit in meters per each depth value	Float	depth_scale: 0.0010
depth_datatype	Indicate depth datatype, only [uint16] is supported for this version	String, Values must be uint16	depth_datatype: uint16
depth_size	Indicate depth resolutions in [width, height]	List[Uint32], must be [width, height]	depth_size: [640, 480]
color	Indicate color format. only rgba is supported	String. Value must be rgba	color: rgba
color_size	Indicate color resolutions in [width, height]	List[Uint32], must be [width, height]	color_size: [1920, 1080]
depth_intrinsic	Indicate depth sensor intrinsic parameter groups	Intrinsic Configuration Group	depth_intrinsic: width: 848 height: 480 centerX: 424.06073 centerY: 237.75032 fx: 422.513062 fy: 422.513062
color_intrinsic	Indicate color sensor intrinsic parameter groups	Intrinsic Configuration Group	color_intrinsic: width: 1920 height: 1080 centerX: 964.288086 centerY: 533.287354 fx: 1358.21423 fy: 1358.2533
depth_to_color_extrinsic	Indicate extrinsic parameters from depth sensor to color sensor	Extrinsic Configuration Group	depth_to_color_extrinsic: rotation: [1, -0.0068, 0.0010, 0.0068, 1, 0, -0.0010, 0, 1] translation: [0.01481, -0.0001, 0.0002]

Configuration Body for Intrinsic Parameters :

libnvds_3d_depth_datasource Intrinsic Parameters in Depth file source `config_body` Fields¶
Property	Meaning	Type and Range	Example
width	Specify sensor width in pixels	Uint32	width: 848
height	Specify sensor height in pixels	Uint32	height: 480
centerX	Specify coordinate axis position in pixels in horizontal direction	Float	centerX: 424.06
centerY	Specify coordinate axis position in pixels in vertical direction	Float	centerY: 533.28
fx	Specify focal length in pixels in X direction	Float	fx: 1358.21
fy	Specify focal length in pixels in Y direction	Float	fy: 1358.25

Configuration Body for Extrinsic Parameters:

libnvds_3d_depth_datasource Extrinsic Parameters in Depth file source `config_body` Fields¶
Property	Meaning	Type and Range	Example
rotation	Specify an extrinsic 3x3 matrix for rotation. Values in Column-major order	List[Float], Values in Column-major order	rotation: [1, -0.0068, 0.0010, 0.0068, 1, 0, -0.0010, 0, 1]
translation	Specify an extrinsic 3x1 matrix for translation. Values in Column-major order	List[Float], Values in Column-major order	translation: [0.01481, -0.0001, 0.0002]

DeepStream 3D Custom Manual¶

DS3D Application Examples¶

ds3d::dataloader - Load Custom Lib for Data Capture¶

Load and Manage Dataloader¶

Dataloader in User Application¶

Implement a Custom Dataloader¶

ds3d::datafilter- Loads Custom Lib for Input and Output Processing¶

Load And Manage Datafilter¶

Datafilter in User Application¶

Implement a Custom Datafilter¶

ds3d::datarender - Loads Custom Lib for Data Rendering¶

Load And Manage Datarender¶

Datarender in User Application¶

Implement a Custom Datarender¶

DS3D GuardDataMap Buffer Management¶

DS3D Data Map Read¶

DS3D Data Map Write¶

Custom Libs Configuration Specifications¶

Components Common Configuration Specifications¶

libnvds_3d_dataloader_realsense Configuration Specifications¶

libnvds_3d_depth2point_datafilter Configuration Specifications¶

libnvds_3d_gl_datarender Configuration Specifications¶

libnvds_3d_depth_datasource Depth file source Specific Configuration Specifications¶

`ds3d::dataloader` - Load Custom Lib for Data Capture¶

`ds3d::datafilter`- Loads Custom Lib for Input and Output Processing¶

`ds3d::datarender` - Loads Custom Lib for Data Rendering¶

DS3D `GuardDataMap` Buffer Management¶