Skip to navigation
DeepStream Plugin GuidePlugins

Gst-nvds3dfilter

View as Markdown

The Gst-nvds3dfilter plugin is a customizable plugin which provides a custom library interface to load component ds3d::datafilter.

The Plugin is fully configured through YAML format content or file. sink_caps and src_caps is also configurable.

Gst-nvds3dfilter

Custom library Interfaces

The plugin parse the config file or content and set SinkCaps and SrcCaps. load a custom ds3d::datafilter from custom_lib_path. Create this datafilter by custom_create_function.

See more details of ds3d::datafilter in DS3D Manual ds3d::datafilter- DS3D Custom DataFilter.

Inputs and Outputs

  • Inputs

    • NvDs3DBuffer or
    • Metadata (NvDsBatchMeta) which has user meta NVDS_3D_DATAMAP_META_TYPE

  • Control parameters

    • config-file
    • config-content

  • Output

    • NvDs3DBuffer or
    • Metadata (NvDsBatchMeta) which has user meta NVDS_3D_DATAMAP_META_TYPE

  • sink_caps

    • parsed from filed of in_caps: ds3d/datamap or any user specified in_caps:.

  • src_caps

    • parsed from filed of out_caps: ds3d/datamap or any user specified out_caps:.

Configuration file

Example:

name: multimodal_triton_infer
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_tritoninferfilter.so
custom_create_function: createLidarInferenceFilter
config_body:
  in_streams: [color, depth]

Gst-nvds3dfilter properties Specifications

Gst-nvds3dfilter Properties.

PropertiesDescriptionRelease
config-contentyaml config contentDS 6.1
config-fileyaml config file pathDS 6.1

Supported DS3D filter custom libs

These custom libs are part of DeepStream release package.

DS3D Data filterData filter custom libraryCreating Instance FunctionDescription
Multi-Modal-triton-inferlibnvds_tritoninferfilter.socreateLidarInferenceFilterMulti-modal sensor data inference library, see details in libnvds_tritoninferfilter Configuration Specifications
Data-alignmentlibnvds_3d_alignment_datafilter.socreateLidarAlignmentFilterlidar/camera Sensor Intrinsic and Extrinsic parameters and alignment
Depth-2-point-cloudlibnvds_3d_depth2point_datafilter.socreateDepth2PointFilterConvert Image Depth data into 3D Point Cloud data, see details in Custom datafilter libnvds_3d_depth2point_datafilter Configuration Specifications

Custom ds3d::datafilter library: libnvds_3d_alignment_datafilter.so

Custom ds3d::datafilter library: libnvds_3d_alignment_datafilter.so

The input kLidarXYZI from ds3d/datamap buffer is converted into kLidarXYZIAligned. This is done by a series of transformations.

Transformations done inside the alignment filter are described below with the help of KITTI Data format and calibration data references.

Transform IDDetails
AL_TR_1Euclidean transformation from lidar to reference camera. Velodyne points convert to reference camera (cam0) coordinate. Output = matrixmultiply(Tr_velo_to_cam, input_lidar_point)
AL_TR_2Note: AL_TR_2 is done on top of AL_TR_1. Alignment filter expect users to set lidar_to_cam_extrisic with both Tr_velo_to_cam and R0_rect. Rotation to account for rectification for points in the reference camera. Output = matrixmultiply(matmul(R0_rect, Tr_velo_to_cam), input_lidar_point)
AL_TR_3[Optional and is configured to ON/OFF with config: align_to_intrinsic] Projective transformation from rectified reference camera frame (AL_TR2) to cam[2]. Note: Alignment filter expect users to set cam_intrinsic with P_rect[2]. Output = matrixmultiply(matmul(P_rect[2], R0_rect, Tr_velo_to_cam), input_lidar_point). Note: P_rect[2], R0_rect, Tr_velo_to_cam are all converted to 4X4 matrices.

In DeepStream world:

cam_intrinsicP0 OR P_rect[0]
lidar_to_cam_extrisicis a 3X4 matrix = matrix_multiply(R0_rect, Tr_velo_to_cam)

More info here: Alignment configuration Options.

Inputs and Outputs

  • Inputs

    • ds3d/datamap

  • Control parameters

    • config-file
    • config-content

  • Output

    • ds3d/datamap

Configuration file

Example:

name: lidar_alignment_filter
type: ds3d::datafilter
link_to: ds3d_aligned_render
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_alignment_datafilter.so
custom_create_function: createLidarAlignmentFilter
config_body:
  cam_width: 1242
  cam_height: 375
  cam_intrinsic: [7.215377000000e+02, 0.000000000000e+00, 0.000000000000e+00, 0.000000000000e+00, 7.215377000000e+02, 0.000000000000e+00, 6.095593000000e+02, 1.728540000000e+02, 1.000000000000e+00, 4.485728000000e+01, 2.163791000000e-01, 2.745884000000e-03]
  #lidar_to_cam_extrisic is a 3X4 matrix = matrix_multiply(R0_rect, Tr_velo_to_cam);
  #lidar_to_cam_extrisic is in column major order
  #For more details on R0_rect and Tr_velo_to_cam, please check: https://github.com/lkk688/3DDepth/blob/main/README.md
  lidar_to_cam_extrisic: [2.34773698e-04, 1.04494074e-02, 9.99945389e-01, -9.99944155e-01, 1.05653536e-02, 1.24365378e-04, -1.05634778e-02, -9.99889574e-01, 1.04513030e-02, -2.79681694e-03, -7.51087914e-02, -2.72132796e-01]
  #cam input keys to ratain in the output datamap
  cam_input_keys: [DS3D::ColorFrame+1]
  #lidar input and output/aligned keys to retain in the output datamap
  #note: lidar_input_keys and lidar_aligned_keys map the same sensor data
  lidar_input_keys: [DS3D::LidarXYZI+0]
  lidar_aligned_keys: [DS3D::LidarAlignedXYZIKey]
  max_points: 2073600
  mem_pool_size: 32
  align_to_intrinsic: false
  lidar_element_size: 4

Configuration options

config-keyDescriptionExample value
cam_widthThe width of camera image used for video input.1242
cam_heightThe height of camera image used for video input.375
cam_intrinsicIs a 3X4 matrix in column major. Projective transformation from rectified reference camera frame to cam in use for video input. More info here (see documentation for P_rect[i]; Note: we use P_rect[2] for the default config in ds3d_lidar_video_alignment_render.yaml).[3X4] matrix in column major.
lidar_to_cam_extrisicis a 3X4 matrix = matrix_multiply(R0_rect, Tr_velo_to_cam); lidar_to_cam_extrisic is in column major order For more details on R0_rect and Tr_velo_to_cam, please check: https://github.com/lkk688/3DDepth/blob/main/README.md[3X4] matrix in column major.
lidar_input_keyThe keyname to use for lidar input from the input datamap buffer.DS3D::LidarXYZI+0
lidar_aligned_keyThe keyname to use for lidar output newly added by alignment filter into the output datamap buffer.DS3D::LidarAlignedXYZIKey
max_pointsMaximum number of Lidar points in the input datamap at: {lidar_input_key: points}.2048000
mem_pool_sizeAlignment filter maintains an internal pool. This is the pool size the user want to allocate.4
align_to_intrinsicIf true: AL_TR_3 is done. If false: AL_TR_2 is done.True
lidar_element_sizeThe size of output aligned lidar data points. = 3 means XYZ = 4 means XYZI (I is the intensity) More details here.4

Custom ds3d::datafilter library: libnvds_3d_depth2point_datafilter.so

Custom ds3d::datafilter library: libnvds_3d_depth2point_datafilter.so