DeepStream Lidar Inference App (Alpha) ======================================== The ``deepstream-lidar-inference-app`` sample application is provided at `app/sample_apps/deepstream-lidar-inference-app/` for your reference. The ``deepstream_lidar_inference_app`` provides an end-to-end inference sample for lidar pointcloud data. The sample application reads the point cloud data from dataset files and send the data to Triton Inferencing filter with `PointPillarNet model `_, the inferencing result is the group of 3D bounding boxes of the objects. The sample application loads different pipelines based on different application config files. There are 2 pipelines configured in the sample app. 1. Lidar Triton inference for 3D objects detection and file dump. 2. Lidar Triton inference and 3D objects detection and GLES 3D rendering. Inside these sample configurations, the inference model is a 3D TAO model based on PointPillar. For more details about PointPillar, see https://arxiv.org/abs/1812.05784. .. note:: TensorRT 8.5 has a bug for FP16 mode when converting this specific TAO model to TensorRT engine file. DeepStream fallback to FP32 mode for this release. This is a snapshot for ``deepstream-lidar-inference-app`` running with lidar data inference objects detection and GLES 3D rendering with 3D bounding box display on screen. .. image:: /content/DS_3D_lidar_inference_app_snapshot.png :align: center :alt: DeepStream Lidar inference for 3D point cloud processing and rendering snapshot Prerequisites --------------- You must have the following development packages installed: * GStreamer-1.0 * GStreamer-1.0 Base Plugins * X11 client-side library * libyaml-cpp-dev To install these packages, execute the following command: :: sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \ libgstrtspserver-1.0-dev libx11-dev libyaml-cpp-dev Lidar Point Cloud to 3D Point Cloud Processing and Rendering ---------------------------------------------------------------------- The application can be configured as different pipelines according to the application configuration file. 1. Lidar data inference and 3D bounding box dump pipeline: This pipeline is from lidar point cloud, 3D point cloud inferencing, to the 3D objects data dump. .. image:: /content/DS_3D_lidar_inference_pointcloud_process.png :align: center :alt: DeepStream Lidar inference for 3D point cloud processing overview This pipeline is setup by the ``config_lidar_triton_infer.yaml``. It has 3 components: ``ds3d::dataloader`` for lidar pointcloud data file reading, ``ds3d::datafilter`` for point-cloud Triton inferencing, and ``ds3d::datarender`` for `3D Bounding Box` file dump. * ``ds3d::dataloader`` loads custom lib ``libnvds_lidarfileread.so`` and creates a `dataloader` through the ``createLidarFileLoader`` function. This specific loader is configured by lidar dataset file list `data_config_file`. `Gst-appsrc` connects the `dataloader` into the deepstream pipeline. :: name: lidarsource type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_lidarfileread.so custom_create_function: createLidarFileLoader * ``ds3d::datafilter`` loads custom lib ``libnvds_tritoninferfilter.so`` and creates a lidar point cloud Triton inferencing filter through the ``createLidarInferenceFilter`` function. For this specific configuration, The Lidar Triton filter inferences the point cloud data with TAO model `PointPillarNet model `_ and return the 3D bounding boxes around each object. :: name: lidarfilter type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_tritoninferfilter.so custom_create_function: createLidarInferenceFilter ``ds3d::datafilter`` is loaded by the ``nvds3dfilter`` `Gst-plugin` which accepts ``in_caps`` as `sink_caps` and ``out_caps`` as `src_caps`. It creates a custom ``ds3d::datafilter`` instance and processess data as ``ds3d/datamap``. * ``ds3d::datarender`` loads custom lib ``libnvds_lidarfilewrite.so`` to dump the detected 3D bounding boxes to a file. :: name: lidarrender type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_lidarfilewrite.so custom_create_function: createLidarFileDataRender 2. Lidar data inference and Lidar data rendering with 3D bounding box display pipeline: This pipeline is from lidar point cloud data file, 3D point cloud inferencing, to the 3D point cloud rendering with colors. .. image:: /content/DS_3D_lidar_inference_pointcloud_render.png :align: center :alt: DeepStream Lidar inference for 3D point cloud processing and rendering overview This pipeline is setup by the ``config_lidar_source_triton_render.yaml``. It has 3 components: ``ds3d::dataloader`` for lidar pointcloud data file reading, ``ds3d::datafilter`` for point-cloud Triton inferencing, and ``ds3d::datarender`` for Lidar 3D data `LidarXYZI` and `3D Bounding Box` rendering. * ``ds3d::dataloader`` loads custom lib ``libnvds_lidarfileread.so`` and creates a `dataloader` through the ``createLidarFileLoader`` function. This specific loader is configured by lidar dataset file list `data_config_file`. `Gst-appsrc` connects the `dataloader` into the deepstream pipeline. :: name: lidarsource type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_lidarfileread.so custom_create_function: createLidarFileLoader * ``ds3d::datafilter`` loads custom lib ``libnvds_tritoninferfilter.so`` and creates a lidar point cloud Triton inferencing filter through the ``createLidarInferenceFilter`` function. For this specific configuration, The Lidar Triton filter inferences the point cloud data with TAO model `PointPillarNet model `_ and return the 3D bounding boxes around each object. :: name: lidarfilter type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_tritoninferfilter.so custom_create_function: createLidarInferenceFilter ``ds3d::datafilter`` is loaded by the ``nvds3dfilter`` `Gst-plugin` which accepts ``in_caps`` as `sink_caps` and ``out_caps`` as `src_caps`. It creates a custom ``ds3d::datafilter`` instance and processess data as ``ds3d/datamap``. * ``ds3d::datarender`` loads custom lib ``libnvds_3d_gl_datarender.so`` and creates GLES Lidar point cloud rendering context to display XYZI or XYZ lidar data and 3D bounding boxes with ``custom_create_function: createLidarDataRender``. :: name: lidarrender type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createLidarDataRender Getting Started --------------------- Run Lidar Point Cloud Data File reader, Point Cloud Inferencing filter, and Point Cloud 3D rendering and data dump Examples ----------------------------------------------------------------------------------------------------------------------------- 1. Prepare `PointPillarNet model `_ and Triton environment, this app will use Triton to do inference, for more details of Triton Inferencing Server, refer to https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinferserver.html. There are more details in the deepstream-lidar-inference-app README file. 2. Run the lidar point cloud data inference pipeline in 2 modes. a. Run lidar data reader, point cloud 3D objects detection inference and 3D data GLES rendering pipeline: :: $ deepstream-lidar-inference-app -c configs/config_lidar_source_triton_render.yaml b. Run lidar data reader, point cloud 3D objects inference and 3D objects file dump pipeline: :: $ deepstream-lidar-inference-app -c configs/config_lidar_triton_infer.yaml 3. This part sets up a lidar point cloud loader `dataloader`. It then streams `ds3d/datamap` to the downstream `datafilter` component `lidarfilter`. :: name: lidarsource type: ds3d::dataloader out_caps: ds3d/datamap custom_lib_path: libnvds_lidarfileread.so custom_create_function: createLidarFileLoader 4. It streams `ds3d/datamap` to the ``nvds3dfilter`` `Gst-plugin` which loads ``lidarfilter`` to do Triton inferencing on point cloud. For more details on ``nvds3dfilter`` `Gst-plugin`, See :doc:`DS_plugin_gst-nvds3dfilter`. :: name: lidarfilter type: ds3d::datafilter in_caps: ds3d/datamap out_caps: ds3d/datamap custom_lib_path: libnvds_tritoninferfilter.so custom_create_function: createLidarInferenceFilter * Field of `model_inputs` is the description of the model input layers. Includes layer name, layer data type and layer dimensions. :: model_inputs: - name: points # name of the 1st layer datatype: FP32 # data type of the 1st layer shape: [1, 204800, 4] # data dimension of the 1st layer - name: num_points # name of the 2nd layer datatype: INT32 # data type of the 2nd layer shape: [1] # data dimension of the 2nd layer * Field of `model_outputs` is the description of the model output layers. Includes layer name, layer data type and layer dimensions. :: model_outputs: - name: output_boxes # name of the 1st layer datatype: FP32 # data type of the 1st layer shape: [1, 393216, 9] # data dimension of the 1st layer - name: num_boxes # name of the 2nd layer datatype: INT32 # data type of the 2nd layer shape: [1] # data dimension of the 2nd layer * Field of `labels` is the label list of the point cloud inferencing model. :: labels: # YAML list for object labels - Vehicle - Pedestrain - Cyclist 5. Finally the data stream as `ds3d/datamap` is delivered to the render component. a. 3D detection file dump component ``lidarfiledump``. :: name: lidarfiledump type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_lidarfilewrite.so custom_create_function: createLidarFileDataRender b. GLES lidar data rendering component ``lidardatarender``. :: name: lidardatarender type: ds3d::datarender in_caps: ds3d/datamap custom_lib_path: libnvds_3d_gl_datarender.so custom_create_function: createLidarDataRender DeepStream Lidar Inference App Configuration Specifications ----------------------------------------------------------------- deepstream-lidar-inference-app ``[ds3d::userapp]`` group settings ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The table below demonstrates the group settings for `config_lidar_triton_infer.yaml` and `config_lidar_source_triton_render.yaml` as the examples. .. csv-table:: point cloud inferencing app user debug supported settings :file: ../text/tables/DS_3D tables/DS_3D_Lidar_Inference_App_config_settings.csv :widths: 15, 20, 40, 20, 20 :header-rows: 1 DS3D Custom Components Configuration Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ See more details in the `DS_3D supported custom components specifications` section in the :doc:`DS_3D_Custom_Manual` . Build application From Source ###################################### 1. Go to the folder `sources/apps/sample_apps/deepstream-lidar-inference-app`. 2. Run the following commands: :: $ sudo make $ sudo make install .. note:: Check the source code for more details on how to load `dataloader/datarender` through ``Gst-appsrc`` and ``Gst-appsink``. `datafilter` is loaded by the ``nvds3dfilter`` `Gst-plugin`.