perception/path/camera/docs/usecase1_pilotnetdetector.md

Go to the documentation of this file.

# Copyright (c) 2019-2020 NVIDIA CORPORATION.  All rights reserved.

@page pathperception_usecase1_pilotnetdetector PilotNet Workflow

@note SW Release Applicability: This tutorial is applicable to modules in **NVIDIA DRIVE Software** releases.

#### Initialization
The PilotNetDetector parameters are initialized based on the dwPilotNetHandle_t using the following function call.
```{.cpp}
    dwStatus dwPilotNetDetector_initParams(dwPilotNetDetectorParams* params,
                                           dwPilotNetHandle_t pilotnetDNN);
```

This call fills the params.numSensors and params.sensors[x].camera (where x is from 0 to params.numSensors) of the ::dwPilotNetDetectorParams struct. The rest need to be filled by the application The sensor parameters that need to be populated in the ::dwPilotNetDetectorParams include the sensor type, input CUDA stream, the designation of camera frame width and height, number of camera frames per second, camera model and camera extrensics.
```{.cpp}
    dwRigHandle rigHandle
    dwImageProperties cameraImageProperties;
    dwCameraProperties cameraProperties;
    for (uint32_t i = 0; i < params.numSensors; i++)
    {
        uint32_t sensorId;
        dwRig_findSensorByName(&sensorId, "camera::front::center::60fov", rigHandle);
        params.sensors[i].sensorId = sensorId;

        params.sensors[i].stream = cudaStream;

        dwRect inputDimensions{};
        inputDimensions.width = cameraImageProperties.width;
        inputDimensions.height = cameraImageProperties.height;
        params.sensors[i].inDimensions = inputDimensions;

        params.sensors[i].fps = cameraProperties.framerate;

        dwCameraModelHandle_t cameraModel;
        dwCameraModel_initialize(&cameraModel, sensorId, rigHandle);
        params.sensors[i].cameraModel = cameraModel;

        dwTransformation3f transformation;
        dwRig_getSensorToRigTransformation(&transformation, sensorId, rigHandle);
        params.sensors[i].cameraExtrinsics = &transformation;
    }
```

After populating all sensor information of each sensor in the params.sensor struct, the @ref pathperception_mainsection_pilotnetdetector module is initialized with the following function call:
```{.cpp}
    dwStatus dwPilotNetDetector_initialize(dwPilotNetDetectorHandle_t* pilotnet,
                                           const dwPilotNetDetectorParams* params,
                                           dwPilotNetHandle_t pilotnetDNN,
                                           dwContextHandle_t ctx);
```

Finally a struct must be allocated and binded with the @ref pathperception_mainsection_pilotnetdetector module to store the output of the module with the function call

```{.cpp}
    dwStatus dwPilotNetDetector_bindOutput(dwPilotNetDetectorOutput* output,
                                           dwPilotNetDetectorHandle_t pilotnet);
```

To get all the supported driving modes of the current model, use

```{.cpp}
    dwStatus dwPilotNetDetector_getAvailableDrivingModes(bool* modes,
                                                         dwPilotNetDetectorHandle_t pilotnet);
```

#### Process

After you successfully initialize the @ref pathperception_mainsection_pilotnetdetector module, you must prepare two struct before inference:

- ::dwPilotNetDetectorState that describes status signals to the network. The status signals represent any non-image inputs such as lane change length, current speed, etc. The dwPilotNetDetector_setState() is used to pass this information to the module.
- A RAW camera frame that has been passed through the Software ISP pipeline to produce a quarter resolution (½ height x ½ width) de-bayered image, or an H.264 encoded YUV420 image. This image is passed using the dwPilotNetDetector_setCameraFrame() function. Note that this function must be called for every sesnsorId in ::dwPilotNetDetectorParams.
```{.cpp}
    for (uint32_t i = 0; i < params.numSensors; i++)
    {
        dwStatus dwPilotNetDetector_setCameraFrame(dwImageCudaFrame, params.sensors[i].sensorId, pilotnetHandle);
    }
```

Once all the structs are set, call this function to perform pre processing,

```{.cpp}
    dwStatus dwPilotNetDetector_processFrames(dwPilotNetDetectorHandle_t pilotnet);
```
If insufficient images are passed using dwPilotNetDetector_setCameraFrame() before the call to dwPilotNetDetector_processFrames(), an error is thrown.

The region-of-interest extraction from the initial frame is performed internally by the @ref pathperception_mainsection_pilotnetdetector module as part of the dwPilotNetDetector_processFrames() function. The dimensions and position of the ROI can be queried from the module using dwPilotNetDetector_getROIParams() function. The ROI is in turn down-sampled to a patch of fixed dimension that is finally fed into the network. The patch for each input frame can be extracted from the module using the dwPilotNetDetector_getPatchU8() function.

Then call this function to runs the inference:

```{.cpp}
    dwStatus dwPilotNetDetector_infer(dwPilotNetDetectorHandle_t pilotnet);
```

Finally perform post-processing on the inference output with
```{.cpp}
    dwStatus dwPilotNetDetector_processOutput(dwPilotNetDetectorHandle_t pilotnet);
```
This is a blocking call, and the output is ready in the memory bound by the dwPilotNetDetector_bindOutput() function call.

To retrieve the predicted trajectory in image coordinates of a particular camera (denoted by sensorId) for rendering purposes, the following function is called:

```{.cpp}
    dwStatus dwPilotNetDetector_convertPointsRigToSensor(dwVector2f* points,
                                                         uint16_t numPoints,
                                                         dwVector3f* trajPoints,
                                                         uint16_t numTrajPoints,
                                                         uint32_t sensorId,
                                                         dwPilotNetDetectorHandle_t pilotnet);
```

where trajPoints points to the trajectory of the desired driving mode index in the ::dwPilotNetDetectorOutput struct, e.g. output.trajectory[DW_PILOTNET_LANE_STABLE].

A visualization mask containing the overlay image of the network activations for the most recent inference can be obtained with the following function call:

```{.cpp}
    dwStatus dwPilotNetDetector_getVisMaskU8(const dwImageCUDA** dwImg,
                                             uint32_t* numROI,
                                             uint32_t sensorId,
                                             dwPilotNetDetectorHandle_t pilotnet);
```

#### Reset and Release

To update/reset the extrensic calibration of a particluar sensor, use the following function call

```{.cpp}
    dwStatus dwPilotNetDetector_setSensorExtrinsics(const dwTransformation3f* tx,
                                                    const uint32_t sensorId,
                                                    dwPilotNetDetectorHandle_t pilotnet);
```

If you need to reset the @ref pathperception_mainsection_pilotnetdetector module, e.g. to initialize it again with the same PilotNet model, the following function call can be used:

```{.cpp}
    dwStatus dwPilotNetDetector_reset(dwPilotNetDetectorHandle_t pilotnet);
```

Finally, the @ref pathperception_mainsection_pilotnetdetector module is released with the function call:

```{.cpp}
    dwStatus dwPilotNetDetector_release(dwPilotNetDetectorHandle_t *pilotnet);
```

#### Sample Application

The PilotNet sample application @ref dwx_pilotnet_sample shows how to use the API calls mentioned above. It is designed to be a starting point for integrating the NVIDIA @ref pathperception_mainsection_pilotnetdetector module into more complex systems.