DeepStream 3D Depth Camera App ======================================== The ``deepstream-3d-depth-camera`` sample application is provided at `app/sample_apps/deepstream-3d-depth-camera` for your reference. This example demonstrates two different pipelines which bring depth camera from 2D into 3D world. The depth and color data capture component is produced by |Intel®| |RealSense™| Depth Camera D435 series. For more details, see https://www.intelrealsense.com/depth-camera-d435/. .. |Intel®| replace:: Intel\ :sup:`®` .. |RealSense™| replace:: RealSense\ :sup:`™` Prerequisites --------------- You must have the following development packages installed: * GStreamer-1.0 * GStreamer-1.0 Base Plugins * GStreamer-1.0 gstrtspserver * X11 client-side library * libyaml-cpp-dev * RealSense SDK 1. To install these packages, execute the following command: :: sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \ libgstrtspserver-1.0-dev libx11-dev libyaml-cpp-dev 2. Follow the RealSense SDK documentation to add RealSense public key and list of repositories. For x86, refer to https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md. For Jetson platforms, refer to https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md. You may use Ubuntu 18.04 source path into apt-repository to install dependencies, until Ubuntu 20.04 support on Jetson(JP5.0+) is available. :: sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo bionic main" -u 3. Then install ``librealsense2`` package to build and test. :: sudo apt-get install librealsense2-utils librealsense2-dev Make sure RealSense SDK version is not earlier than 2.48. 4. Plugin in Intel RealSense Depth D400 series USB camera. To verify the camera and upgrade firmware, run ``realsense-viewer`` and update from the UI. Navigate to the folder `sources/apps/sample_apps/deepstream-3d-depth-camera`. 5. If you are running these commands inside deepstream container, make sure camera devices are mounted in the ``docker run`` command. For example, :: docker run --rm -it --gpus '"device=0"' --device /dev/video0 --device /dev/video1 --device /dev/video2 ... 6. Export ``DISPLAY`` environment to the correct display. for example, ``export DISPLAY=:0.0``. Depth Color Capture to 2D Rendering Pipeline Overview ------------------------------------------------------------- This is the first pipeline from the depth capture to 2D based `depthmap` to direct rendering with linear colors configured by the user. .. image:: /content/DS_3D_depth_camera_2d_render.png :align: center :alt: DeepStream Depth Camera for 2D rendering stack overview The pipeline is setup by ``ds_3d_realsense_depth_capture_render.yaml``. It has 2 components, ``ds3d::dataloader`` for depth/color and ``datarender`` for `GLES 2D` render. * ``ds3d::dataloader`` loads custom lib ``libnvds_3d_dataloader_realsense.so`` and creates a RealSense `dataloader` through the ``createRealsenseDataloader`` function. This specific loader is configured for output streams of `[color, depth]`. `Gst-appsrc` connects the `dataloader` into the deepstream pipeline. :: name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader * ``ds3d::datarender`` loads custom lib ``libnvds_3d_gl_datarender.so`` and create a `GLES` render through the ``createDepthStreamDataRender`` function. This specific loader is configured for 2D depth and color images. `Gst-appsink` connects the `datarender` into the deepstream pipeline. :: name: depth-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createDepthStreamDataRender * ``realsense_dataloader`` captures color and depth streams in ``ds3d/datamap``, then delivers the data to ``ds3d::datarender`` component ``depth-render``. Depth Color Capture to 3D Point Cloud Processing and Rendering ---------------------------------------------------------------------- The 2nd pipeline is from depth capture, 3D point cloud processing and alignment, to the 3D point cloud rendering with colors. .. image:: /content/DS_3D_depth_camera_3d_points_process.png :align: center :alt: DeepStream Depth Camera for 3D point cloud processing overview The 2nd pipeline is setup by the ``ds_3d_realsense_depth_to_point_cloud.yaml``. It has 3 components: ``ds3d::dataloader`` for depth/color capture, ``ds3d::datafilter`` for depth-to-point-cloud processing, and ``datarender`` for `GLES 3D` points render. * ``ds3d::dataloader`` is the same as the 1st pipeline for depth and color capture. It also outputs the intrinsic parameters of the color/depth sensors, and extrinsic parameters from depth to color sensor module. :: name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader * ``ds3d::datafilter`` loads custom lib ``libnvds_3d_depth2point_datafilter.so`` and creates a depth-to-3D-point processing filter through the ``createDepth2PointFilter`` function. This specific filter generates the 3D point-cloud dataset from depth, and calculates a UV-coordinate map for correspondence between points and color position. It requires sensors' intrinsic and extrinsic data from ``dataloader`` to make the alignment. :: name: point2cloud_datafilter type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_3d_depth2point_datafilter.so custom_create_function: createDepth2PointFilter ``ds3d::datafilter`` is loaded by the ``nvds3dfilter`` `Gst-plugin` which accepts ``in_caps`` as `sink_caps` and ``out_caps`` as `src_caps`. It creates a custom ``ds3d::datafilter`` instance and processess data as ``ds3d/datamap``. * ``ds3d::datarender`` loads custom lib ``libnvds_3d_gl_datarender.so`` and create a `GLES` render through the ``createPointCloudDataRender`` function . This specific loader is configured for 3D points and colors rendering. It also supports 3D scene rotation based on mouse-drag. :: name: point-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createPointCloudDataRender * ``realsense_dataloader`` captures color and depth streams along with the intrinsic and extrinsic parameters of the sensor in ``ds3d/datamap``. It then delivers the data to the next component ``point2cloud_datafilter``. This component generates 3D point-cloud data and UV coordination map into new output data ``ds3d/datamap``. Finally, the data is delivered to ``ds3d::datarender`` component ``point-render`` for 3D rendering. Inside the configuration files, ``in_caps`` and ``out_caps`` correspond to Gstreamer's `sink_caps` and `src_caps`. Getting Started --------------------- Run RealSense Camera for Depth Capture and 2D Rendering Examples ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. Run the depth capture 2D render pipeline: :: $ deepstream-3d-depth-camera -c ds_3d_realsense_depth_capture_render.yaml 2. This sets up a `realsense` `dataloader`. :: name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader 3. It then streams `ds3d/datamap` to final the rendering component: `depth-render`. :: name: depth-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createDepthStreamDataRender 3. The `depth-render` custom lib shall display the depth data and color data together in same window. Update ``min_depth/max_depth`` to remove foreground and background objects in the depth rendering. From RealSense specification, `D435` Camera depth range is between ``0.3`` and ``3`` meters. :: min_depth: 0.3 # in meters max_depth: 2.0 # in meters 4. Update `min_depth_color/min_depth_color` [R, G, B] values to visualize color map of depth. :: min_depth_color: [255, 128, 0] # RGB color value for mininum depth max_depth_color: [0, 128, 255] # RGB color value for maximum depth The other colors are linearly interpolated between ``min_depth_color`` and ``max_depth_color`` Run 3D Depth Capture, Point Cloud filter, and 3D Points Rendering Examples ----------------------------------------------------------------------------- 1. Run the depth capture and 3D processing pipeline: :: $ deepstream-3d-depth-camera -c ds_3d_realsense_depth_to_point_cloud.yaml 2. This sets up a realsense `dataloader` (same as 2D depth). It then streams `ds3d/datamap` to the downstream `datafilter` component `point2cloud_datafilter`. :: name: realsense_dataloader type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_3d_dataloader_realsense.so custom_create_function: createRealsenseDataloader 3. It streams `ds3d/datamap` to the ``nvds3dfilter`` `Gst-plugin` which loads ``point2cloud_datafilter`` to convert 2D depth into 3D points. For more details on ``nvds3dfilter`` `Gst-plugin`, See :doc:`DS_plugin_gst-nvds3dfilter`. :: name: point2cloud_datafilter type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_3d_depth2point_datafilter.so custom_create_function: createDepth2PointFilter 4. Finally the data stream as `ds3d/datamap` is delivered to the render component ``point-render``. :: name: point-render type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createPointCloudDataRender * Fields of `view_*` are the eyes of the view position, similar to the `OpenGL` ``gluLookAt()``. :: view_position: [0, 0, -1] # view position in xyz coordinates view_target: [0, 0, 1] # view target which is the direction pointing to view_up: [0, -1.0, 0] # view up direction * Fields of `near/far/fov` are the perspective range of the eye, similar to the `OpenGL` ``gluPerspective()``. :: near: 0.01 # nearest points of perspective far: 10.0 # farmost points of perspective fov: 40.0 # FOV of perspective DeepStream 3D Depth Camera App Configuration Specifications ----------------------------------------------------------------- deepstream-3d-depth-camera ``[ds3d::userapp]`` group settings ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The table below demonstrates the group settings for `ds_3d_realsense_depth_to_point_cloud.yaml` as an example. .. csv-table:: 3D depth camera app user debug supported settings :file: ../text/tables/DS_3D tables/DS_3D_Depth_Camera_App_config_settings.csv :widths: 20, 20, 20, 40 :header-rows: 1 DS3D Custom Components Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ See more details in the `DS_3D supported custom components specifications` section in the :doc:`DS_3D_Custom_Manual` . Build application From Source ###################################### 1. Go to the folder `sources/apps/sample_apps/deepstream-3d-depth-camera`. 2. Run the following commands: :: $ Set CUDA_VER in the MakeFile as per platform. For x86, CUDA_VER=11.8 For Jetson, CUDA_VER=11.4 $ make $ make install .. note:: Check the source code for more details on how to load `dataloader/datarender` through ``Gst-appsrc`` and ``Gst-appsink``. `datafilter` is loaded by the ``nvds3dfilter`` `Gst-plugin`.