Wait Conditions Detector with Temporal Analysis Workflow

Note

SW Release Applicability: This tutorial is applicable to modules in NVIDIA DRIVE Software releases.

Initialization

TemporalAnalysis optionally can be used with WaitConditionsDetector to track and stabilize the detected intersection over the frames. Please refer to Wait Conditions Classification Workflow for Traffic Lights and WaitNet Workflow for more details about how to initialize and use the WaitNet DNN and WaitConditionsDetector modules first.

TemporalAnalysis module creates handle of type dwTemporalAnalysisHandle_t. This handle is owned by the application and must be released by the application when not needed anymore.

An instance of type dwWaitConditionsDetectorTemporalAnalysisParams encodes parameters that configures the TemporalAnalysis module before initialization. There is no function to initialize this data structure with default values. So all the values must be initialized manually. Here is an example:

dwWaitConditionsDetectorTemporalAnalysisParams taParams = {};

// fill parameters
taParams.gpuPostProc            = true;
taParams.scaleToCameraImage     = 2.0;
taParams.detectionNegativeCount = 15;   // number of frames of negative detections before stop seeing intersection (15 - 1/2 sec)
taParams.detectionPositiveCount = 15;   // number of frames of positive detections before start seeing intersection (15 - 1/2 sec)
taParams.directLineDistance     = 5.0;  // direct line distance (blend between euclidean distance and direct distance when within this range)
taParams.egoMotionOnlyDistance  = 12.0; // rig to intersection distance after which only egomotion is considered to track intersection's position
taParams.parzenWindowSize       = 20;
taParams.kdeSigma               = 1.0;
taParams.peakDetectionThreshold = 0.3;  // PDF peak detection threshold
taParams.validEvidenceThreshold = 0.3;  // 0-1 confidence level to accept bbox evidence into temporal filter

// range parameters: x - direct perpendicular distance, y - lateral distance (called "angle" in the code)
taParams.xRangeMin              = 0.0;
taParams.xRangeMax              = 200.0;
taParams.yRangeMin              = -100.0;
taParams.yRangeMax              = 100.0;

// roll/pitch active correction, filter coefficients
taParams.rollFilterK0           = 0.07;
taParams.rollFilterK1           = 0.3;
taParams.rollFilterK2           = 0.4;
taParams.pitchFilterK0          = 0.07;
taParams.pitchFilterK1          = 0.3;
taParams.pitchFilterK2          = 0.4;

taParams can now be used to initialize the TemporalAnalysis module:

dwTemporalAnalysisHandle_t m_temporalAnalysisHandle = DW_NULL_HANDLE;

CHECK_DW_ERROR(dwTemporalAnalysis_initializeSimple(&m_temporalAnalysisHandle,
                                                   cam2Rig,
                                                   cam,
                                                   m_egoMotion,
                                                   &taParams,
                                                   m_ctx));

The function dwTemporalAnalysis_initializeSimple() instantiates the TemporalAnalysis module and assigns the pointer to m_temporalAnalysisHandle handle. m_ctx is an initialized instance of dwContextHandle_t.

cam2Rig and cam specify the extrinsic and intrinsic of the camera. cam2Rig is a pointer to "camera to rig" transformation matrix and cam is a handle to calibrated camera. m_egoMotion is a handle to an instance of EgoMotion. These parameters are necessary to map the pixels that are in image-space to real world coordinates and track intersection's position and distance.

Temporal Analysis Process

After we get the detected intersection from WaitConditionsDetector module (Wait Conditions Classification Workflow for Traffic Lights), we can process it using TemporalAnalysis:

CHECK_DW_ERROR(dwTemporalAnalysis_setTargetTimestamp(camFrameTime, m_temporalAnalysisHandle));
CHECK_DW_ERROR(dwTemporalAnalysis_addIntersectionEvidence(
               &intersection,
               requireTransformToGround,
               m_temporalAnalysisHandle));
CHECK_DW_ERROR(dwTemporalAnalysis_runTemporalAnalysis(m_temporalAnalysisHandle));
CHECK_DW_ERROR(dwTemporalAnalysis_getSmoothedIntersection(
               &intersection,
               m_temporalAnalysisHandle));

intersection which is an instance of dwIntersectionDetection, contains the intersection detected by WaitConditionsDetector module and returned by dwWaitConditionsDetector_getDetectedIntersection() function (Wait Conditions Classification Workflow for Traffic Lights).

dwTemporalAnalysis_setTargetTimestamp() sets the current time-stamp and associates it with the detected intersection. camFrameTime is the time-stamp of current camera frame. dwTemporalAnalysis_addIntersectionEvidence() adds detected intersection to list of evidences to be processed. requireTransformToGround is a boolean specifying whether a point in image space should be transformed into a 3-dimensional ground point or not (default is true). dwTemporalAnalysis_runTemporalAnalysis() uses the current evidence to perform the analysis. dwTemporalAnalysis_getSmoothedIntersection() can be used to get output of TemporalAnalysis which updates the intersection object with the tracked and temporally smoothed intersection after the call.

The following function can be used to set the CUDA stream:

dwStatus dwTemporalAnalysis_setCUDAStream(cudaStream_t stream,
    dwTemporalAnalysisHandle_t temporalAnalysisHandle);

The following function can be used to change extrinsic calibration during runtime if necessary:

dwStatus dwTemporalAnalysis_setCameraToRig(
    const dwTransformation3f* cam2Rig, dwTemporalAnalysisHandle_t temporalAnalysisHandle);

Release

The following function can be used to release the module:

dwStatus dwTemporalAnalysis_release(
    dwTemporalAnalysisHandle_t temporalAnalysisHandle);