DeepStream 3D Depth Camera App¶

The deepstream-3d-depth-camera sample application is provided at app/sample_apps/deepstream-3d-depth-camera for your reference. This example demonstrates two different pipelines which bring depth camera from 2D into 3D world. The depth and color data capture component is produced by Intel^® RealSense^™ Depth Camera D435 series. For more details, see https://www.intelrealsense.com/depth-camera-d435/.

Prerequisites¶

You must have the following development packages installed:

GStreamer-1.0

GStreamer-1.0 Base Plugins

GStreamer-1.0 gstrtspserver

X11 client-side library

libyaml-cpp-dev

RealSense SDK

To install these packages, execute the following command:

sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \
libgstrtspserver-1.0-dev libx11-dev libyaml-cpp-dev

Follow the RealSense SDK documentation to add RealSense public key and list of repositories. For x86, refer to https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md. For Jetson platforms, refer to https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md. You may use Ubuntu 20.04 source path into apt-repository to install dependencies.
```
sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo bionic main" -u
```
Then install librealsense2 package to build and test.
```
sudo apt-get install librealsense2-utils librealsense2-dev
```
Make sure RealSense SDK version is not earlier than 2.48.
Plugin in Intel RealSense Depth D400 series USB camera. To verify the camera and upgrade firmware, run realsense-viewer and update from the UI. Navigate to the folder sources/apps/sample_apps/deepstream-3d-depth-camera.
If you are running these commands inside deepstream container, make sure camera devices are mounted in the docker run command. For example:
```
docker run --rm -it --gpus '"device=0"' --device /dev/video0 --device /dev/video1 --device /dev/video2 ...
```
Export DISPLAY environment to the correct display. for example, export DISPLAY=:0.0.

Depth Color Capture to 2D Rendering Pipeline Overview¶

This is the first pipeline from the depth capture to 2D based depthmap to direct rendering with linear colors configured by the user.

The pipeline is setup by ds_3d_realsense_depth_capture_render.yaml. It has 2 components, ds3d::dataloader for depth/color and datarender for GLES 2D render.

ds3d::dataloader loads custom lib libnvds_3d_dataloader_realsense.so and creates a RealSense dataloader through the createRealsenseDataloader function. This specific loader is configured for output streams of [color, depth]. Gst-appsrc connects the dataloader into the deepstream pipeline.
```
name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
```
ds3d::datarender loads custom lib libnvds_3d_gl_datarender.so and create a GLES render through the createDepthStreamDataRender function. This specific loader is configured for 2D depth and color images. Gst-appsink connects the datarender into the deepstream pipeline.
```
name: depth-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender
```
realsense_dataloader captures color and depth streams in ds3d/datamap, then delivers the data to ds3d::datarender component depth-render.

Depth Color Capture to 3D Point Cloud Processing and Rendering¶

The 2nd pipeline is from depth capture, 3D point cloud processing and alignment, to the 3D point cloud rendering with colors.

The 2nd pipeline is setup by the ds_3d_realsense_depth_to_point_cloud.yaml. It has 3 components: ds3d::dataloader for depth/color capture, ds3d::datafilter for depth-to-point-cloud processing, and datarender for GLES 3D points render.

ds3d::dataloader is the same as the 1st pipeline for depth and color capture. It also outputs the intrinsic parameters of the color/depth sensors, and extrinsic parameters from depth to color sensor module.
```
name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
```
ds3d::datafilter loads custom lib libnvds_3d_depth2point_datafilter.so and creates a depth-to-3D-point processing filter through the createDepth2PointFilter function. This specific filter generates the 3D point-cloud dataset from depth, and calculates a UV-coordinate map for correspondence between points and color position. It requires sensors’ intrinsic and extrinsic data from dataloader to make the alignment.
```
name: point2cloud_datafilter
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter
```
ds3d::datafilter is loaded by the nvds3dfilter Gst-plugin which accepts in_caps as sink_caps and out_caps as src_caps. It creates a custom ds3d::datafilter instance and processess data as ds3d/datamap.
ds3d::datarender loads custom lib libnvds_3d_gl_datarender.so and create a GLES render through the createPointCloudDataRender function . This specific loader is configured for 3D points and colors rendering. It also supports 3D scene rotation based on mouse-drag.
```
name: point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender
```
realsense_dataloader captures color and depth streams along with the intrinsic and extrinsic parameters of the sensor in ds3d/datamap. It then delivers the data to the next component point2cloud_datafilter. This component generates 3D point-cloud data and UV coordination map into new output data ds3d/datamap. Finally, the data is delivered to ds3d::datarender component point-render for 3D rendering.

Inside the configuration files, in_caps and out_caps correspond to Gstreamer’s sink_caps and src_caps.

Getting Started¶

Run RealSense Camera for Depth Capture and 2D Rendering Examples¶

Run the depth capture 2D render pipeline:

$ deepstream-3d-depth-camera -c ds_3d_realsense_depth_capture_render.yaml

This sets up a realsense dataloader.

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

It then streams ds3d/datamap to final the rendering component: depth-render.

name: depth-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender

The depth-render custom lib shall display the depth data and color data together in same window. Update min_depth/max_depth to remove foreground and background objects in the depth rendering. From RealSense specification, D435 Camera depth range is between 0.3 and 3 meters.

min_depth: 0.3 # in meters
max_depth: 2.0 # in meters

Update min_depth_color/min_depth_color [R, G, B] values to visualize color map of depth.

min_depth_color: [255, 128, 0] # RGB color value for mininum depth
max_depth_color: [0, 128, 255] # RGB color value for maximum depth
The other colors are linearly interpolated between min_depth_color and max_depth_color

Run 3D Depth Capture, Point Cloud filter, and 3D Points Rendering Examples¶

Run the depth capture and 3D processing pipeline:

$ deepstream-3d-depth-camera -c ds_3d_realsense_depth_to_point_cloud.yaml

This sets up a realsense dataloader (same as 2D depth). It then streams ds3d/datamap to the downstream datafilter component point2cloud_datafilter.

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

It streams ds3d/datamap to the nvds3dfilter Gst-plugin which loads point2cloud_datafilter to convert 2D depth into 3D points. For more details on nvds3dfilter Gst-plugin, See Gst-nvds3dfilter.

name: point2cloud_datafilter
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter

Finally the data stream as ds3d/datamap is delivered to the render component point-render.

name: point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender

Fields of view_* are the eyes of the view position, similar to the OpenGL gluLookAt().

view_position: [0, 0, -1] # view position in xyz coordinates
view_target: [0, 0, 1] # view target which is the direction pointing to
view_up: [0, -1.0, 0] # view up direction

Fields of near/far/fov are the perspective range of the eye, similar to the OpenGL gluPerspective().

near: 0.01 # nearest points of perspective
far: 10.0 # farmost points of perspective
fov: 40.0 # FOV of perspective