PilotNet Workflow

Note

SW Release Applicability: This tutorial is applicable to modules in NVIDIA DRIVE Software releases.

Initialization

The PilotNetDetector parameters are initialized based on the dwPilotNetHandle_t using the following function call.

dwStatus dwPilotNetDetector_initParams(dwPilotNetDetectorParams* params,
                                       dwPilotNetHandle_t pilotnetDNN);

This call fills the params.numSensors and params.sensors[x].camera (where x is from 0 to params.numSensors) of the dwPilotNetDetectorParams struct. The rest need to be filled by the application The sensor parameters that need to be populated in the dwPilotNetDetectorParams include the sensor type, input CUDA stream, the designation of camera frame width and height, number of camera frames per second, camera model and camera extrensics.

dwRigHandle rigHandle
dwImageProperties cameraImageProperties;
dwCameraProperties cameraProperties;
for (uint32_t i = 0; i < params.numSensors; i++)
{
    uint32_t sensorId;
    dwRig_findSensorByName(&sensorId, "camera::front::center::60fov", rigHandle);
    params.sensors[i].sensorId = sensorId;

    params.sensors[i].stream = cudaStream;

    dwRect inputDimensions{};
    inputDimensions.width = cameraImageProperties.width;
    inputDimensions.height = cameraImageProperties.height;
    params.sensors[i].inDimensions = inputDimensions;

    params.sensors[i].fps = cameraProperties.framerate;

    dwCameraModelHandle_t cameraModel;
    dwCameraModel_initialize(&cameraModel, sensorId, rigHandle);
    params.sensors[i].cameraModel = cameraModel;

    dwTransformation3f transformation;
    dwRig_getSensorToRigTransformation(&transformation, sensorId, rigHandle);
    params.sensors[i].cameraExtrinsics = &transformation;
}

After populating all sensor information of each sensor in the params.sensor struct, the Pilotnet Detector module is initialized with the following function call:

dwStatus dwPilotNetDetector_initialize(dwPilotNetDetectorHandle_t* pilotnet,
                                       const dwPilotNetDetectorParams* params,
                                       dwPilotNetHandle_t pilotnetDNN,
                                       dwContextHandle_t ctx);

Finally a struct must be allocated and binded with the Pilotnet Detector module to store the output of the module with the function call

dwStatus dwPilotNetDetector_bindOutput(dwPilotNetDetectorOutput* output,
                                       dwPilotNetDetectorHandle_t pilotnet);

To get all the supported driving modes of the current model, use

dwStatus dwPilotNetDetector_getAvailableDrivingModes(bool* modes,
                                                     dwPilotNetDetectorHandle_t pilotnet);

Process

After you successfully initialize the Pilotnet Detector module, you must prepare two struct before inference:

• dwPilotNetDetectorState that describes status signals to the network. The status signals represent any non-image inputs such as lane change length, current speed, etc. The dwPilotNetDetector_setState() is used to pass this information to the module.
• A RAW camera frame that has been passed through the Software ISP pipeline to produce a quarter resolution (½ height x ½ width) de-bayered image, or an H.264 encoded YUV420 image. This image is passed using the dwPilotNetDetector_setCameraFrame() function. Note that this function must be called for every sesnsorId in dwPilotNetDetectorParams.

  for (uint32_t i = 0; i < params.numSensors; i++)
  {
      dwStatus dwPilotNetDetector_setCameraFrame(dwImageCudaFrame, params.sensors[i].sensorId, pilotnetHandle);
  }

Once all the structs are set, call this function to perform pre processing,

dwStatus dwPilotNetDetector_processFrames(dwPilotNetDetectorHandle_t pilotnet);

If insufficient images are passed using dwPilotNetDetector_setCameraFrame() before the call to dwPilotNetDetector_processFrames(), an error is thrown.

The region-of-interest extraction from the initial frame is performed internally by the Pilotnet Detector module as part of the dwPilotNetDetector_processFrames() function. The dimensions and position of the ROI can be queried from the module using dwPilotNetDetector_getROIParams() function. The ROI is in turn down-sampled to a patch of fixed dimension that is finally fed into the network. The patch for each input frame can be extracted from the module using the dwPilotNetDetector_getPatchU8() function.

Then call this function to runs the inference:

dwStatus dwPilotNetDetector_infer(dwPilotNetDetectorHandle_t pilotnet);

Finally perform post-processing on the inference output with

dwStatus dwPilotNetDetector_processOutput(dwPilotNetDetectorHandle_t pilotnet);

This is a blocking call, and the output is ready in the memory bound by the dwPilotNetDetector_bindOutput() function call.

To retrieve the predicted trajectory in image coordinates of a particular camera (denoted by sensorId) for rendering purposes, the following function is called:

dwStatus dwPilotNetDetector_convertPointsRigToSensor(dwVector2f* points,
                                                     uint16_t numPoints,
                                                     dwVector3f* trajPoints,
                                                     uint16_t numTrajPoints,
                                                     uint32_t sensorId,
                                                     dwPilotNetDetectorHandle_t pilotnet);

where trajPoints points to the trajectory of the desired driving mode index in the dwPilotNetDetectorOutput struct, e.g. output.trajectory[DW_PILOTNET_LANE_STABLE].

A visualization mask containing the overlay image of the network activations for the most recent inference can be obtained with the following function call:

dwStatus dwPilotNetDetector_getVisMaskU8(const dwImageCUDA** dwImg,
                                         uint32_t* numROI,
                                         uint32_t sensorId,
                                         dwPilotNetDetectorHandle_t pilotnet);

Reset and Release

To update/reset the extrensic calibration of a particluar sensor, use the following function call

dwStatus dwPilotNetDetector_setSensorExtrinsics(const dwTransformation3f* tx,
                                                const uint32_t sensorId,
                                                dwPilotNetDetectorHandle_t pilotnet);

If you need to reset the Pilotnet Detector module, e.g. to initialize it again with the same PilotNet model, the following function call can be used:

dwStatus dwPilotNetDetector_reset(dwPilotNetDetectorHandle_t pilotnet);

Finally, the Pilotnet Detector module is released with the function call:

dwStatus dwPilotNetDetector_release(dwPilotNetDetectorHandle_t *pilotnet);

Sample Application

The PilotNet sample application PilotNet Sample shows how to use the API calls mentioned above. It is designed to be a starting point for integrating the NVIDIA Pilotnet Detector module into more complex systems.