waitcondition/camera/docs/usecase4.md

Go to the documentation of this file.

# Copyright (c) 2019-2020, NVIDIA CORPORATION.  All rights reserved.

@page waitcondition_usecase4 Wait Conditions Detector with Temporal Analysis Workflow

@note SW Release Applicability: This tutorial is applicable to modules in **NVIDIA DRIVE Software** releases.

## Initialization

TemporalAnalysis optionally can be used with WaitConditionsDetector to track and stabilize the detected intersection over the frames. Please refer to @ref waitcondition_usecase1 and @ref waitnet_usecase1 for more details about how to initialize and use the WaitNet DNN and WaitConditionsDetector modules first.

TemporalAnalysis module creates handle of type `dwTemporalAnalysisHandle_t`. This handle is owned by the application and must be released by the application when not needed anymore.

An instance of type `dwWaitConditionsDetectorTemporalAnalysisParams` encodes parameters that configures the TemporalAnalysis module before initialization. There is no function to initialize this data structure with default values. So all the values must be initialized manually. Here is an example:

```{.cpp}
dwWaitConditionsDetectorTemporalAnalysisParams taParams = {};

// fill parameters
taParams.gpuPostProc            = true;
taParams.scaleToCameraImage     = 2.0;
taParams.detectionNegativeCount = 15;   // number of frames of negative detections before stop seeing intersection (15 - 1/2 sec)
taParams.detectionPositiveCount = 15;   // number of frames of positive detections before start seeing intersection (15 - 1/2 sec)
taParams.directLineDistance     = 5.0;  // direct line distance (blend between euclidean distance and direct distance when within this range)
taParams.egoMotionOnlyDistance  = 12.0; // rig to intersection distance after which only egomotion is considered to track intersection's position
taParams.parzenWindowSize       = 20;
taParams.kdeSigma               = 1.0;
taParams.peakDetectionThreshold = 0.3;  // PDF peak detection threshold
taParams.validEvidenceThreshold = 0.3;  // 0-1 confidence level to accept bbox evidence into temporal filter

// range parameters: x - direct perpendicular distance, y - lateral distance (called "angle" in the code)
taParams.xRangeMin              = 0.0;
taParams.xRangeMax              = 200.0;
taParams.yRangeMin              = -100.0;
taParams.yRangeMax              = 100.0;

// roll/pitch active correction, filter coefficients
taParams.rollFilterK0           = 0.07;
taParams.rollFilterK1           = 0.3;
taParams.rollFilterK2           = 0.4;
taParams.pitchFilterK0          = 0.07;
taParams.pitchFilterK1          = 0.3;
taParams.pitchFilterK2          = 0.4;
```

`taParams` can now be used to initialize the TemporalAnalysis module:

```{.cpp}
dwTemporalAnalysisHandle_t m_temporalAnalysisHandle = DW_NULL_HANDLE;

CHECK_DW_ERROR(dwTemporalAnalysis_initializeSimple(&m_temporalAnalysisHandle,
                                                   cam2Rig,
                                                   cam,
                                                   m_egoMotion,
                                                   &taParams,
                                                   m_ctx));
```

The function `dwTemporalAnalysis_initializeSimple()` instantiates the TemporalAnalysis module and assigns the pointer to `m_temporalAnalysisHandle` handle. `m_ctx` is an initialized instance of `::dwContextHandle_t`.

`cam2Rig` and `cam` specify the extrinsic and intrinsic of the camera. `cam2Rig` is a pointer to "camera to rig" transformation matrix and `cam` is a handle to calibrated camera. `m_egoMotion` is a handle to an instance of EgoMotion. These parameters are necessary to map the pixels that are in image-space to real world coordinates and track intersection's position and distance.

## Temporal Analysis Process

After we get the detected intersection from WaitConditionsDetector module (@ref waitcondition_usecase1), we can process it using TemporalAnalysis:

```{.cpp}
CHECK_DW_ERROR(dwTemporalAnalysis_setTargetTimestamp(camFrameTime, m_temporalAnalysisHandle));
CHECK_DW_ERROR(dwTemporalAnalysis_addIntersectionEvidence(
               &intersection,
               requireTransformToGround,
               m_temporalAnalysisHandle));
CHECK_DW_ERROR(dwTemporalAnalysis_runTemporalAnalysis(m_temporalAnalysisHandle));
CHECK_DW_ERROR(dwTemporalAnalysis_getSmoothedIntersection(
               &intersection,
               m_temporalAnalysisHandle));
```

`intersection` which is an instance of `dwIntersectionDetection`, contains the intersection detected by WaitConditionsDetector module and returned by `dwWaitConditionsDetector_getDetectedIntersection()` function (@ref  waitcondition_usecase1).

`dwTemporalAnalysis_setTargetTimestamp()` sets the current time-stamp and associates it with the detected intersection. `camFrameTime` is the time-stamp of current camera frame. `dwTemporalAnalysis_addIntersectionEvidence()` adds detected intersection to list of evidences to be processed. `requireTransformToGround` is a boolean specifying whether a point in image space should be transformed into a 3-dimensional ground point or not (default is true). `dwTemporalAnalysis_runTemporalAnalysis()` uses the current evidence to perform the analysis. `dwTemporalAnalysis_getSmoothedIntersection()` can be used to get output of TemporalAnalysis which updates the `intersection` object with the tracked and temporally smoothed intersection after the call.

The following function can be used to set the CUDA stream:

```{.cpp}
dwStatus dwTemporalAnalysis_setCUDAStream(cudaStream_t stream,
    dwTemporalAnalysisHandle_t temporalAnalysisHandle);
```

The following function can be used to change extrinsic calibration during runtime if necessary:

```{.cpp}
dwStatus dwTemporalAnalysis_setCameraToRig(
    const dwTransformation3f* cam2Rig, dwTemporalAnalysisHandle_t temporalAnalysisHandle);
```

## Release

The following function can be used to release the module:

```{.cpp}
dwStatus dwTemporalAnalysis_release(
    dwTemporalAnalysisHandle_t temporalAnalysisHandle);
```