The PilotNetDetector module streams a RAW or H.264 encoded video and outputs a path that the vehicle should follow. The pipeline of this module includes loading an NVIDIA proprietary deep neural network called PilotNet, generating a processed frame from the camera image, and running inference using the loaded network on a region-of-interest (ROI) extracted from this frame.
The output of the network is visualized as a trajectory plot of the predicted path. This trajectory is rendered on an overlaid video of the region-of-interest (ROI) patches and a visualization of the network activations.
The module uses 60° Field of View Sekonix Camera Module (SS3323) with an AR0231 RCCB sensor as the primary sensor as an input with a requirement of additional sensors for specialized models.
The following is the list of cameras that may be requested by PilotNet along with their accepted extrensics and tolerances in rig coordinate system
| dwPilotNetCameraType | X (m) | Y (m) | Z (m) | roll (deg) | pitch (deg) | yaw (deg) |
|---|---|---|---|---|---|---|
| DW_PILOTNET_FRONT_CENTER_60FOV | 1.75 ±0.2 | 0 ±0.2 | 1.5 ±0.2 | 0 ±1.5° | 0 ±1.5° | 0 ±1.5° |
| DW_PILOTNET_FRONT_CENTER_120FOV | 1.75 ±0.2 | 0 ±0.2 | 1.5 ±0.2 | 0 ±1.5° | 0 ±1.5° | 0 ±1.5° |
| DW_PILOTNET_FRONT_LEFT_120FOV | 1.75 ±0.2 | 0.5 ±0.2 | 1.5 ±0.2 | 0 ±1.5° | 0 ±1.5° | 0 ±1.5° |
| DW_PILOTNET_FRONT_RIGHT_120FOV | 1.75 ±0.2 | -0.5 ±0.2 | 1.5 ±0.2 | 0 ±1.5° | 0 ±1.5° | 0 ±1.5° |
| DW_PILOTNET_CROSS_LEFT_120FOV | 1.65 ±0.2 | 0.6 ±0.2 | 1.5 ±0.2 | 0 ±1.5° | 0 ±1.5° | 45° ±1.5° |
| DW_PILOTNET_CROSS_RIGHT_120FOV | 1.65 ±0.2 | -0.6 ±0.2 | 1.5 ±0.2 | 0 ±1.5° | 0 ±1.5° | -45° ±1.5° |