DeepStream 3D Custom Manual

New ds3d framework, interfaces and custom-libs are defined for DS 3D processing. These interfaces are capble of different types of data fusion and can implement different types of custom libraries for dataloader, datafilter and datarender. The interface has ABI compatible layers and modern C++ interface layers. You’ll only need to focus on the modern interface for application or custom lib development.

DeepStream 3D dataloader is loaded by GstAppSrc. It could be used for depth camera such as stereo cameras, Time-of-Flight cameras to capture image/depth data or 2D/3D data-load from the source file., It also could be used for lidar data sensor or lidar data file capture. datafilter is loaded by the nvds3dfilter Gst-plugin. It could be used for 2D depth data processing , 3D point-cloud data extraction from depth, other 2D-depth or 3D-points data filters and lidar or 3D data inference. datarender is loaded by GstAppSink. It could be used for 2D depth rendering and 3D point-cloud and lidar data rendering. It also could be used for file dump.

DS3D Application Examples

  • deepstream-lidar-inference has the sample code to load these custom libs and to connect these components together in simple ways. Besides that, DS3D has a simple C++ safe pointer for Gstreamer components. The interfaces are found in header files located at /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/gst/.

The image below shows the overview of lidar 3D data inference and rendering pipeline in deepstream-lidar-inference.

DeepStream Lidar point cloud inference and rendering overview

See more details in the DeepStream Lidar Inference App (Alpha).

  • deepstream-3d-depth-camera is another example for ds3d pipeline.

The image below shows the overview of depth to 3D point processing pipeline in deepstream-3d-depth-camera.

DeepStream Depth Camera for 3D point cloud processing overview

See more details in the DeepStream 3D Depth Camera App.

All the components are configured in YAML format. They are loaded by Gst-plugins.There are 3 major components, they may all be loaded into the deepstream pipeline.

ds3d::dataloader - Load Custom Lib for Data Capture

Load and Manage Dataloader

Examples:

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
config_body:
  streams: [color, depth]

A custom dataloader must have type: ds3d::dataloader. It is created by explicit call of NvDs3D_CreateDataLoaderSrc(srcConfig, loaderSrc, start) with the full compoment YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsrc object is also created into loaderSrc.gstElement.

GstAppsrc manages the ds3d::dataloader dataflows. This ds3d::dataloader component could be started automatically by gst-pipeline or manually by the application call.
GuardDataLoader dataloader = loaderSrc.customProcessor;
ErrCode c = dataloader.start();
To stop the dataloader, user can set GstAppsrc states to GST_STATE_READY or stop it manually.
GuardDataLoader dataloader = loaderSrc.customProcessor;
ErrCode c = dataloader.stop();

Dataloader in User Application

Examples:

#include <ds3d/common/hpp/dataloader.hpp>
GuardDataLoader dataLoader;
dataLoader.reset(createTestTimeDataloader()); # create a specific ABI
ErrCode c = dataloader.setErrorCallback([](ErrCode c, const char*){});
c = dataloader.start(yamlconfig); // check result.
while(isGood(c)) {
  GuardDataMap data;
  c = dataloader.readData(data);
  // process data
}
c = dataloader.flush();
c = dataloader.stop();

GuardDataLoader provides safe access to abiDataLoader. Once it’s created, it will maintain the reference pointer to the dataloader.

Implement a Custom Dataloader

Examples:

#include <ds3d/common/impl/impl_dataloader.h>
class TestTimeDataLoader : public ds3d::impl::SyncImplDataLoader {
public:
    TestTimeDataLoader() = default;

protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setOutputCaps("ds3d/datamap");
        return ErrCode::kGood;
    }
    ErrCode readDataImpl(GuardDataMap& datamap) override
    {
        datamap.reset(NvDs3d_CreateDataHashMap());
        static uint64_t iTime = 0;
        TimeStamp t{iTime++, 0, 0};
        datamap.setData("time", t);
        emitError(ErrCode::kGood, "timstamp added");
        return ErrCode::kGood;
    }
    ErrCode stopImpl() override { return ErrCode::kGood; }
    ErrCode flushImpl() override { return ErrCode::kGood; }
};

DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefDataLoader*
createTestTimeDataloader()
{
    return NewAbiRef<abiDataLoader>(new TestTimeDataLoader);
}
DS3D_EXTERN_C_END

A shown in the example above, You’ll need to derive dataloader from the ds3d::impl::SyncImplDataLoader class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode readDataImpl(GuardDataMap& datamap) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through NvDs3D_CreateDataLoaderSrc, you’ll also need to export createTestTimeDataloader.

ds3d::datafilter- Loads Custom Lib for Input and Output Processing

Load And Manage Datafilter

Examples:

name: point2cloud_datafilter
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter
config_body:

A custom datafilter must have type: ds3d::datafilter. It is loaded through the nvds3dfilter Gst-plugin. It is started by gst_element_set_state(GST_STATE_READY). During this call, the custom_lib_path is loaded and a specific data filter is created by custom_create_function. nvds3dfilter Gst-plugin has config-content and config-file properties. One of them must be set to create a datafilter object.

Datafilter in User Application

Examples:

#include <ds3d/common/hpp/datafilter.hpp>
GuardDataFilter datafilter;
datafilter.reset(createTestFakeDatafilter()); # create a specific ABI
ErrCode c = datafilter.setErrorCallback([](ErrCode c, const char*){});
c = datafilter.start(yamlconfig); // check result.
int consumedNum = 0;
auto dataConsumed = [&consumedNum](ErrCode c, const abiRefDataMap* data) {
    if (isGood(c)) {
        ++consumedNum;
    }
};
for (int i = 0; i < 100; ++i) {
  TimeStamp t{i, 0, 0};
  GuardDataMap dataIn;
  dataIn.reset(NvDs3d_CreateDataHashMap());
  dataIn.setData("time", t);

  GuardDataMap dataOut;
  ErrCode cbCode = ErrCode::kGood;
  auto outputCB = [&dataOut, &cbCode](ErrCode c, const abiRefDataMap* data) {
      cbCode = c;
      if (data) {
          GuardDataMap newData(*data);
          dataOut = newData;
      }
  };
  c = dataFilter.process(dataIn, dataConsumed, outputCB);
}
c = datafilter.flush();
c = datafilter.stop();

GuardDataFilter provides safe access to the abiDataFilter. Once it’s created, it will maintain the reference pointer to datafilter.

Implement a Custom Datafilter

Examples:

#include <ds3d/common/impl/impl_datafilter.h>
class TestFakeDataFilter : public impl::BaseImplDataFilter {
public:
    TestFakeDataFilter() = default;

protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setInputCaps(kFakeCapsMetaName);
        setOutputCaps(kFakeCapsMetaName);
        return ErrCode::kGood;
    }
    ErrCode processImpl(
        GuardDataMap datamap, OnGuardDataCBImpl outputDataCb,
        OnGuardDataCBImpl inputConsumedCb) override
    {
        DS_ASSERT(datamap);
        TimeStamp t;
        ErrCode c = datamap.getData("time", t);
        if (!isGood(c)) {
            return c;
        }
        t.t0 += 1;
        inputConsumedCb(ErrCode::kGood, datamap);
        c = datamap.setData("time", t);
        if (!isGood(c)) {
            return c;
        }
        outputDataCb(ErrCode::kGood, datamap);
        return ErrCode::kGood;
    }

    ErrCode flushImpl() override { return ErrCode::kGood; }
    ErrCode stopImpl() override { return ErrCode::kGood; }
};

DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefdatafilter*
createTestFakeDatafilter()
{
    return NewAbiRef<abidatafilter>(new TestFakeDataFilter);
}
DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive the datafilter from the ds3d::impl::BaseImplDataFilter class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode processImpl(
        GuardDataMap datamap, OnGuardDataCBImpl outputDataCb,
        OnGuardDataCBImpl inputConsumedCb) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through nvds3dfilter Gst-plugin, you’ll also need to export a specific symbol createTestFakeDatafilter.

ds3d::datarender - Loads Custom Lib for Data Rendering

Load And Manage Datarender

Examples:

name: point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender
config_body:
  title: ds3d-point-cloud-test

A custom datarender must have type: ds3d::datarender. It is created by explicit call of NvDs3D_CreateDataRenderSink(sinkConfig, renderSink, start) with the full compoment YAML content. During this call, the custom_lib_path is loaded and a specific data loader is created via custom_create_function. A GstAppsink object is also created into renderSink.gstElement.

GstAppsink manages the ds3d::datarender dataflows. This ds3d::datarender component could be automatically started by the gst-pipeline, or manually by the application call.

GuardDataRender datarender = renderSink.customProcessor;
ErrCode c = datarender.start();

To stop the datarender, you can set GstAppsink states to GST_STATE_READY, or stop manually. .. code-block:: text

GuardDataRender datarender = renderSink.customProcessor; ErrCode c = datarender.stop();

Datarender in User Application

Examples:

#include <ds3d/common/hpp/datarender.hpp>
GuardDataRender datarender;
datarender.reset(createTestFakedatarender());
ErrCode c = datarender.setErrorCallback([](ErrCode c, const char*){});
c = datarender.start(yamlconfig); // check result.
for (int i = 0; i < 100; ++i) {
  static uint64_t iTime = 0;
  TimeStamp t{iTime++, 0, 0};
  GuardDataMap datamap;
  datamap.reset(NvDs3d_CreateDataHashMap());
  ASSERT_TRUE(datamap);
  datamap.setData("time", t);
  if (i == 0) {
      c = dataRender.preroll(datamap);
  }
  ErrCode c = dataRender.render(datamap, dataRenderd);
}
c = datarender.flush();
c = datarender.stop();

GuardDataRender provides safe access to abidatarender. Once it’s created, it will maintain the reference pointer to datarender. preroll is called only once to initialize some resources.

Implement a Custom Datarender

Examples:

#include <ds3d/common/impl/impl_datarender.h>
class TestFakeDataRender : public impl::BaseImplDataRender {
public:
    TestFakeDataRender() = default;

protected:
    ErrCode startImpl(const std::string& content, const std::string& path) override
    {
        setInputCaps("ds3d/datamap");
        return ErrCode::kGood;
    }
    ErrCode prerollImpl(GuardDataMap datamap) override { return ErrCode::kGood; }
    ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override
    {
        DS_ASSERT(datamap);
        emitError(ErrCode::kGood, "data rendered");
        dataDoneCb(ErrCode::kGood, datamap);
        return ErrCode::kGood;
    }
    ErrCode flushImpl() override { return ErrCode::kGood; }
    ErrCode stopImpl() override { return ErrCode::kGood; }
};

DS3D_EXTERN_C_BEGIN
DS3D_EXPORT_API abiRefdatarender*
createTestFakedatarender()
{
    return NewAbiRef<abiDataRender>(new TestFakeDataRender());
}
DS3D_EXTERN_C_END

As shown in the example above, you’ll need to derive datarender from the ds3d::impl::BaseImplDataRender class, and implement interfaces for the following:

ErrCode startImpl(const std::string& content, const std::string& path) override;
ErrCode prerollImpl(GuardDataMap datamap) override;
ErrCode renderImpl(GuardDataMap datamap, OnGuardDataCBImpl dataDoneCb) override;
ErrCode stopImpl() override;
ErrCode flushImpl() override;

To load this custom lib through NvDs3D_CreateDataRenderSink, you’ll also need to export a specific symbol createTestFakedatarender.

DS3D GuardDataMap Buffer Management

DS3D Data Map Read

DS3D defines class objects ds3d::abiRefDataMap. All internal data are hash and stored into this data map. NvDs3DBuffer is defined to store the 3D datamap into GstBuffer. Header file is nvds3d_meta.h.

struct NvDs3DBuffer {
    uint32_t magicID;  // must be 'DS3D'
    ds3d::abiRefDataMap* datamap;
};

Warning

Do not use the datamap directly. The easy and safe way to access that is through GuardDataMap. See sample code in :doc: DS_3D_Depth_Camera.

Example:

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
if (NvDs3D_IsDs3DBuf(gstBuf)) {
  const abiRefDataMap* refDataMap = nullptr;
  ErrCode c = NvDs3D_Find1stDataMap(gstBuf, refDataMap);
  ... // check error code
  if (refDataMap) {
    GuardDataMap dataMap(*refDataMap);
    FrameGuard pointFrame;
    c = dataMap.getGuardData(kPointXYZ, pointFrame); // get 3D points reference.
    ... // check error code

    FrameGuard uvCoord;
    c = dataMap.getGuardData(kPointCoordUV, uvCoord); // get 3D points UV coordinates reference.
    ... // check error code

    Frame2DGuard depthFrame;
    c = dataMap.getGuardData(kDepthFrame, depthFrame); // get depth frame reference.
    ... // check error code

    DepthScale scale;
    c = dataMap.getData(kDepthScaleUnit, scale); // copy depth scale
    ... // check error code
  }

}

DS3D Data Map Write

Create an empty datamap, store some frames into this datamap. Example:

#include <ds3d/common/hpp/datamap.hpp>
#include <ds3d/common/hpp/frame.hpp>
#include <ds3d/common/impl/impl_frames.h>
GuardDataMap datamap(NvDs3d_CreateDataHashMap(), true); // set true to take the reference ownership.

/* Create depth frame and store them into ds3d datamap. */
// Assume depth datatype: Uint16
{
  Frame2DPlane depthPlane = {640, 480, 640 * sizeof(uint16_t) , sizeof(uint16_t), 0};
  uint32_t depthBytesPerFrame = depthPlane.pitchInBytes * depthPlane.height;
  std::vector<uint8_t> data(depthBytesPerFrame); // Depth data
  void* dataPtr = &data[0];
  // create depth 2D frame
  Frame2DGuard depthFrame = Wrap2DFrame<uint16_t, FrameType::kDepth>(
          dataPtr, depthPlane}, depthBytesPerFrame, MemType::kCpu, 0,
          [data = std::move(data)](void*) {});
  c = datamap.setGuardData(kDepthFrame, depthFrame); // store depthFrame reference into datamap.
  ... // check error code

  DepthScale scale{0.001, {nullptr}}; //
  c = datamap.setData(kDepthScaleUnit, scale); // copy depth scale into datamap.
  ... // check error code
}

/* Create color image frame and store them into ds3d datamap. */
// Assume format is RGBA
{
  Frame2DPlane colorPlane = {1920, 1080, 1920 * sizeof(uint8_t) , sizeof(uint8_t), 0};
  uint32_t colorBytesPerFrame = colorPlane.pitchInBytes * colorPlane.height;
  std::vector<uint8_t> data(colorBytesPerFrame); // Image data
  void* dataPtr = &data[0];
  // create color 2D frame
  Frame2DGuard frame = Wrap2DFrame<uint8_t, FrameType::kColorRGBA>(
        dataPtr, {_config.colorPlane}, bytesPerFrame, MemType::kCpu, 0,
        [data = std::move(data)](void*) {});
  c = datamap.setGuardData(kColorFrame, colorFrame); // store colorFrame reference into datamap.
  ... // check error code
}

/* Create 3D points frame and store them into ds3d datamap. */
{
  uint32_t pointNum = 640 * 480;
  std::vector<vec3f> points(pointNum); // 3D points data
  vec3f* pointPtr = &points[0];
  FrameGuard pointsFrame = wrapPointXYZFrame<float>(
    (void*)pointPtr, pointNum, MemType::kCpu, 0, [points = std::move(points)](void*) {});
  c = datamap.setGuardData(kPointXYZ, pointXyzFrame); // store 3d-points XYZ data reference into datamap.
  ... // check error code


  std::vector<vec3f> uvData(pointNum); // 3D points data
  vec3f* uvPtr = &data[0];
  FrameGuard pointUvCoord = wrapPointCoordUVFrame<float>(
    (void*)uvPtr, pointNum, MemType::kCpu, 0, [uvData = std::move(uvData)](void*) {});
  c = datamap.setGuardData(kPointCoordUV, pointUvCoord); // store 3d-points UV coordinate data reference into datamap.
  ... // check error code
}

The example below shows how to Create a new GstBuffer with ds3d datamap.

// Assume ``GuardDataMap datamap`` is ready
GstBuffer* gstBuf = nullptr;
ErrCode c = NvDs3D_CreateGstBuf(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
... // check error code

Example below shows how to update an existing DS3D GstBuffer with new ds3d datamap.

// Assume ``GuardDataMap datamap`` is ready
// Assume ``GstBuffer* gstBuf`` is created by another compoment
ErrCode c = NvDs3D_UpdateDataMap(gstBuf, datamap.abiRef(), false); // set false to increase reference count.
... // check error code

Custom Libs Configuration Specifications

Components Common Configuration Specifications

ds3d common configuration specifications

Property

Meaning

Type and Range

Example

type

Custom processor type

String, [ds3d::dataloader, ds3d::datafilter, ds3d::datarender]

type: ds3d::dataloader

name

Indicate user-defined component name

String

name: depthloader

in_caps

Indicate Gst sink caps for the component

String

in_caps: ds3d/datamap

out_caps

Indicate Gst sink caps for the component

String

out_caps: ds3d/datamap

custom_lib_path

Indicate custom lib path

String

custom_lib_path: libnvds_3d_gl_datarender.so

custom_create_function

Indicate custom function to create the specific ds3d processing component

String

custom_create_function: createPointCloudDataRender

config_body

Indicate YAML specific content for the custom comonent

String

config_body:

in_streams: [color, depth] max_points: 407040

libnvds_3d_dataloader_realsense Configuration Specifications

Configuration for Realsense Dataloader Header:

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

libnvds_3d_dataloader_realsense.so requires you to install librealsense2 SDK. For x86, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md. For Jetson platform, follow the instructions from https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md.

libnvds_3d_dataloader_realsense config_body fields

Property

Meaning

Type and Range

Example

streams

Specify which streams to enable

List[String], select from [color, depth]

streams: [color, depth]

aligned_image_to_depth

Indicate whether color image is aligned to depth

Boolean

aligned_image_to_depth: False

libnvds_3d_depth2point_datafilter Configuration Specifications

Configuration for Depth to Points Header:

name: depth2points
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter
libnvds_3d_depth2point_datafilter config_body fields

Property

Meaning

Type and Range

Example

streams

Specify which streams to enable

List[String], select from [color, depth]

streams: [color, depth]

max_points

Indicate maximum 3d points to allocate

Uint32

max_points: 407040

mem_pool_size

Indicate max buffer pool size

Uint32

mem_pool_size: 8

libnvds_3d_gl_datarender Configuration Specifications

Configuration Common header for libnvds_3d_gl_datarender:

name: depth-point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so

Configuration Body for Common Part:

libnvds_3d_gl_datarender config_body common fields

Property

Meaning

Type and Range

Example

title

Specify window title

String

title: ds3d-point-cloud-test

streams

Indicate which streams to render. depth render must have [depth], 3D points render must have [points]

List[String], select from [color, depth, points]

streams: [color, depth]

width

Specify window width

UINT32

width: 1280

height

Specify window height

UINT32

height: 720

block

Indicate rendering thread as block mode

Boolean

block: True

Configuration Header for Point Cloud Render:

name: point-3D-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender # specific function for 3D point rendering

Configuration Header for Lidar data Render:

name: lidar-data-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createLidarDataRender # specific function for Lidar point cloud rendering

Configuration Body for 3D Point Cloud and Lidar Render Part:

For more details on 3D coordinate system, refer to https://learnopengl.com/Getting-started/Coordinate-Systems. To know the value meanings for view_position, view_target and view_up,refer to the gluLookAt here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluLookAt.xml. To know the value meanings for near, far and fov, refer to the gluPerspective here: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluPerspective.xml.

libnvds_3d_gl_datarender Point Cloud Render config_body Fields

Property

Meaning

Type and Range

Example

view_position

Specify view position [x, y, z]coordinates

List[Float]

view_position: [0, 0, -1]

view_target

Specify view target [x, y, z]coordinates

List[Float]

view_target: [0, 0, 1]

view_up

Specify view up direction [x, y, z]coordinates

List[Float]

view_up: [0, -1.0, 0]

near

Specify perspective projection near plane

Float

near: 0.01

far

Specify perspective projection far plane

Float

far: 10.0

fov

Specify perspective projection field of view, degree angle

Float

fov: 40.0

coord_y_opposite

Specify texture map V direction, Realsense coordinates is different from GLES default coordinates

Boolean

coord_y_opposite: False

positive_z_only

Specify whether display negtive depth values

Boolean

positive_z_only: False

Configuration Body for Lidar Render Specific Part:

libnvds_3d_gl_datarender Lidar Render extra config_body Fields

Property

Meaning

Type and Range

Example

view_position

Specify view position [x, y, z]coordinates

List[Float]

view_position: [0, 0, -1]

view_target

Specify view target [x, y, z]coordinates

List[Float]

view_target: [0, 0, 1]

view_up

Specify view up direction [x, y, z]coordinates

List[Float]

view_up: [0, -1.0, 0]

near

Specify perspective projection near plane

Float

near: 0.01

far

Specify perspective projection far plane

Float

far: 10.0

fov

Specify perspective projection field of view, degree angle

Float

fov: 40.0

lidar_color

Specify lidar data color for display

List[Uint32]

lidar_color: [0, 255, 0]

element_size

Specify lidar data element size. e.g. 4 for XYZI or 3 for XYZ

Uint32

element_size: 4

lidar_data_key

Specify lidar data frame in datamap, default value is DS3D::LidarXYZI

String

lidar_data_key: DS3D::LidarXYZI

lidar_bbox_key

Specify lidar 3D bounding box data in datamap, default value is DS3D::Lidar3DBboxRawData

String

lidar_bbox_key: DS3D::Lidar3DBboxRawData

Configuration Header for Depth and Color 2D Render:

name: depth-2D-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender # specific function for 2D depth rendering

Configuration Body for Depth and Color 2D Specific Part:

libnvds_3d_gl_datarender 2D Depth Render config_body Fields

Property

Meaning

Type and Range

Example

min_depth

Specify minimum depth value. other values less that it will be removed in rendering

Float

min_depth: 0.3

max_depth

Specify maximum depth value. other values less that it will be removed in rendering

Float

max_depth: 2.0

min_depth_color

Specify minimum depth rendering color in [R, G, B]

List[Uint32]

min_depth_color: [255, 128, 0]

max_depth_color

Specify maximum depth rendering color in [R, G, B]

Float

max_depth_color: [0, 128, 255]

libnvds_3d_depth_datasource Depth file source Specific Configuration Specifications

Configuration header:

name: depthfilesource
type: ds3d::dataloader
out_caps: ds3d/datamap, framerate=30/1
custom_lib_path: libnvds_3d_depth_datasource.so
custom_create_function: createDepthColorLoader

Configuration body:

libnvds_3d_depth_datasource Depth file source config_body Fields

Property

Meaning

Type and Range

Example

depth_source

Specify file path for depth source

String

depth_source: depth_uint16_640x480.bin

color_source

Specify file path for color image source

String

color_source: color_rgba_1920x1080.bin

depth_scale

Indicate depth unit in meters per each depth value

Float

depth_scale: 0.0010

depth_datatype

Indicate depth datatype, only [uint16] is supported for this version

String, Values must be uint16

depth_datatype: uint16

depth_size

Indicate depth resolutions in [width, height]

List[Uint32], must be [width, height]

depth_size: [640, 480]

color

Indicate color format. only rgba is supported

String. Value must be rgba

color: rgba

color_size

Indicate color resolutions in [width, height]

List[Uint32], must be [width, height]

color_size: [1920, 1080]

depth_intrinsic

Indicate depth sensor intrinsic parameter groups

Intrinsic Configuration Group

depth_intrinsic:

width: 848 height: 480 centerX: 424.06073 centerY: 237.75032 fx: 422.513062 fy: 422.513062

color_intrinsic

Indicate color sensor intrinsic parameter groups

Intrinsic Configuration Group

color_intrinsic:

width: 1920 height: 1080 centerX: 964.288086 centerY: 533.287354 fx: 1358.21423 fy: 1358.2533

depth_to_color_extrinsic

Indicate extrinsic parameters from depth sensor to color sensor

Extrinsic Configuration Group

depth_to_color_extrinsic:

rotation: [1, -0.0068, 0.0010, 0.0068, 1, 0, -0.0010, 0, 1] translation: [0.01481, -0.0001, 0.0002]

Configuration Body for Intrinsic Parameters :

libnvds_3d_depth_datasource Intrinsic Parameters in Depth file source config_body Fields

Property

Meaning

Type and Range

Example

width

Specify sensor width in pixels

Uint32

width: 848

height

Specify sensor height in pixels

Uint32

height: 480

centerX

Specify coordinate axis position in pixels in horizontal direction

Float

centerX: 424.06

centerY

Specify coordinate axis position in pixels in vertical direction

Float

centerY: 533.28

fx

Specify focal length in pixels in X direction

Float

fx: 1358.21

fy

Specify focal length in pixels in Y direction

Float

fy: 1358.25

Configuration Body for Extrinsic Parameters:

libnvds_3d_depth_datasource Extrinsic Parameters in Depth file source config_body Fields

Property

Meaning

Type and Range

Example

rotation

Specify an extrinsic 3x3 matrix for rotation. Values in Column-major order

List[Float], Values in Column-major order

rotation: [1, -0.0068, 0.0010, 0.0068, 1, 0, -0.0010, 0, 1]

translation

Specify an extrinsic 3x1 matrix for translation. Values in Column-major order

List[Float], Values in Column-major order

translation: [0.01481, -0.0001, 0.0002]