Freespace Perception Workflow

Note

SW Release Applicability: This tutorial is applicable to modules in NVIDIA DRIVE Software releases.

The following code snippet shows the general structure of a program that uses the free space detector to detect drive-able free space from a single camera. Note that error handling is left out for clarity.

Free Space Module Initialization

Free space detector module initialization takes an OpenRoadNet module handle as input:

dwStatus dwOpenRoadNet_initialize(dwOpenRoadNetHandle_t *openRoadNetHandle,
                                   dwContextHandle_t ctx,
                                   const dwOpenRoadNetParams *openRoadNetParams);

With the initialized OpenRoadNet module handle, free space module is initialized by:

dwStatus dwFreespaceDetector_initializeFromOpenRoadNet_new(dwFreespaceDetectorHandle_t* obj,
                                                           const dwFreespaceDetectorInitParams* initParams,
                                                           dwContextHandle_t ctx);

where dwFreespaceDetectorInitParams stores the initialization parameters used to initialize a free space detector module. Parameters such as the handle to a calibrated camera and cuda stream should be assigned here. Set calibrated camera handle to nullptr, to get free space boundary only in image space. You can also provide the runtime parameters via dwFreespaceDetectorRuntimeParams altogether by dwFreespaceDetector_setRuntimeParams() or individual set parameter functions described below.

To perform free space detection on a batch of images, you will need to set batchSize in dwOpenRoadNetParams to one of the dwOpenRoadNetBatchSize enum values before dwOpenRoadNet_initialize(). Next, the input image array needs to be bound by:

dwStatus dwFreespaceDetector_bindInput(const dwImageCUDA* const* frames, uint32_t frameCount, dwFreespaceDetectorHandle_t obj);

and output dwFreeSpaceDetection array is bound by:

dwStatus dwFreespaceDetector_bindOutput(dwFreespaceDetection* boundary, uint32_t frameIdx, dwFreespaceDetectorHandle_t obj);

where frameIdx corresponds to the frame index of the boundary to be bound.

Calibrated Camera Initialization (Optional)

To get free space boundary detected in top-view vehicle coordinate system, camera intrinsics can be set via the module initialization call. Note that dwFreespaceDetector_setCameraHandle(...) is deprecated. To set camera extrinsics:

dwFreespaceDetector_setCameraExtrinsics(dwTransformation3f cam2rig, dwFreespaceDetectorHandle_t obj);

Detector Parameter Setup (Optional)

Free space detector allows user to set detection Region of Interest (ROI) and how much each detected boundary is being spatially and/or temporally smoothed.

To set detection ROI in input image coordinates:

dwFreespaceDetector_setDetectionROI(const dwRect* roi,
                                    dwFreespaceDetectorHandle_t obj);

To spatially smooth each detected free space boundary:

dwFreespaceDetector_setSpatialSmoothFilterWidth(uint32_t width, dwFreespaceDetectorHandle_t obj);

To temporally smooth each detected free space boundary:

dwFreespaceDetector_setTemporalSmoothFactor(float32_t factor, dwFreespaceDetectorHandle_t obj);

To set all the aforementioned runtime parameters at once:

dwStatus dwFreespaceDetector_setRuntimeParams(const dwFreespaceDetectorRuntimeParams* rtParams,
                                              dwFreespaceDetectorHandle_t obj);

For more details, please refer to API header documentation.

Free Space Detection Execution

After module initialization and detector parameter setup, there are 2 options can be used to infer free space boundary on GPU and post-process on CPU: For batch or single image free space boundary computation, use the following 3-stage process calls in the order of:

dwFreespaceDetector_process(DW_FREESPACE_DETECTOR_STAGE_GPU_ASYNC_PREPROCESSING, dwFreespaceDetectorHandle_t obj));
dwFreespaceDetector_process(DW_FREESPACE_DETECTOR_STAGE_GPU_ASYNC_INFERENCE, dwFreespaceDetectorHandle_t obj));
dwFreespaceDetector_process(DW_FREESPACE_DETECTOR_STAGE_CPU_POSTPROCESSING, dwFreespaceDetectorHandle_t obj));

The process calls assume that input image array and output dwFreeSpaceDetection array are bound during initialization. Then, the result can be accessed from the output dwFreeSpaceDetection array.

Alternatively, to process only one image at a time, use:

dwStatus dwFreespaceDetector_detectBoundary(dwFreespaceDetection* boundary,
                                            const dwImageCUDA* frame,
                                            dwFreespaceDetectorHandle_t obj);

instead. Input and output bindings are not required here.

Note that the following APIs are already deprecated and will be removed in the next release:

dwStatus dwFreespaceDetector_processDeviceAsync(const dwImageCUDA* frame, dwFreespaceDetectorHandle_t obj);
dwStatus dwFreespaceDetector_interpretHost(dwFreespaceDetectorHandle_t obj);
dwStatus dwFreespaceDetector_getBoundaryDetection(dwFreespaceDetection* boundary, dwFreespaceDetectorHandle_t obj);

Free Space Module Release

To release the free space module, the module itself needs to be released first by:

dwFreespaceDetector_release(dwFreespaceDetectorHandle_t *obj);

then the OpenRoadNet module:

dwOpenRoadNet_release(dwOpenRoadNetHandle_t *obj);

For more detailed workflow, please refer to the free space sample: Freespace Detection Sample (OpenRoadNet)