DriveWorks SDK Reference
4.0.0 Release
For Test and Development only

Porting Guide from SDK 1.5 to SDK 2.0

This document will help you port your applications from DriveWorks SDK 1.5 to DriveWorks SDK 2.0.


Calibration

The module dwCalibratedCamera has been separated from dwRig renamed to dwCameraModel. It resides in the folder dw/calibration/cameramodel. It provides functions to handle calibrated intrinsic camera models.

  • Header file has been renamed from Camera.h to CameraModel.h.
  • The object handles have been renamed to dwCameraModelHandle_t and dwConstCameraModelHandle_t, respectively.
  • The prefixes for all functions declared in CameraModel.h have been updated from dwCalibratedCamera to dwCameraModel.
  • dwCameraModel_applyImageTransform() has been updated and now it takes the additional parameter bool updateMasks.
  • dwCameraModel_initialize has been moved here from dwRig.

For more information see Rig Configuration.


DNN

This release adds support for generating and using DLA (Deep-Learning-Accelerator) models. This change requires selection of the processor type at the time of initialization; therefore, the following functions' signatures have been updated accordingly:

  • dwDNN_initializeTensorRTFromFile() has been renamed to dwDNN_initializeTensorRTFromFileNew() and requires an additional argument of type dwProcessorType indicating on which processor the model shall run on. Note that the model must be generated on the corresponding processor.
  • dwDNN_initializeTensorRTFromMemory() has been renamed to dwDNN_initializeTensorRTFromMemoryNew() and requires an additional argument of type dwProcessorType indicating on which processor the model shall run on. Note that the model must be generated on the corresponding processor.
Note
The old APIs and the New suffix for the new APIs will be removed in the next major release.

For more information see DNN.


Egomotion

  • dwEgoMotionParameters has been renamed to dwEgomotionParameters.
  • dwEgoMotionSensorCharacteristics has been renamed to dwEgomotionSensorCharacteristics.
  • The dwEgomotionParameters.enableLocationFilter has been renamed to dwEgomotionParameters.disableLocationFilter.
  • dw/radarmotion has been moved to dw/egomotion/radar.

For more information see Egomotion.


Iterative Closest Point

ICP has been completelly moved to point cloud processing module.

For more information see Point Cloud ICP.


Image Processing

Port dwPyramidHandle_t to dwPyramidImage

Old name New name
dwPyramid_initialize() dwPyramid_create()
dwPyramid_release() dwPyramid_destroy()
dwPyramid_build() dwImageFilter_computePyramid()
dwPyramid_getLevelCount() dwPyramidImage::levelCount

dwPyramidImage::levelImages[level] provides an dwImageHandle_t for the (level)-th image, dwImage_getProperties() acquires the image size, and dwImage_getCUDA() acquires dwImageCUDA data.

Example: Old code for creating, building and destroying a pyramid:

// Old code
dwPyramidConfig config{};
config.width = width;
config.height = height;
config.levelCount = numLevel;
config.dataType = DW_TYPE_UINT8;
dwPyramid_initialize(&pyramid, &config, 0, context);
dwPyramid_build(&singlePlaneImage, pyramid);
dwPyramid_release(pyramid);

New code for creating, building and destroying a pyramid:

// New code
dwPyramid_create(&pyramid, numLevel, width, height, DW_TYPE_UINT8, context);
dwImageFilter_computePyramid(&pyramid, &singlePlaneImage, 0, context);

Port dwFeatureListHandle_t and dwFeatureListPointers to dwFeatureHistoryArray.

  • dwFeatureStatus has been renamed to dwFeature2DStatus.
  • DW_FEATURE_STATUS_XXX has been renamed to DW_FEATURE2D_STATUS_XXX.

The new dwFeatureHistoryArray struct provides pointers such as: *statuses, *ages, *ids, *locationHistory and *featureCount like the old dwFeatureListPointers struct did. It is also created and destroyed like the old handle dwFeatureListHandle_t. dwFeatureHistoryArray can be created on page-able/pinned CPU or CUDA memory.

Old name New name
dwFeatureListPointers::statuses dwFeatureHistoryArray::statuses
dwFeatureListPointers::ages dwFeatureHistoryArray::ages
dwFeatureHistoryArray::ids
dwFeatureListPointers::locationHistory dwFeatureHistoryArray::locationHistory
dwFeatureListPointers::featureCount dwFeatureHistoryArray::featureCount
dwFeatureList_initialize() dwFeatureHistoryArray_create()
dwFeatureList_release() dwFeatureHistoryArray_destroy()
dwFeatureList_reset() dwFeatureHistoryArray_reset()
dwFeatureList_getDataBasePointer() dwFeatureHistoryArray::data, dwFeatureHistoryArray::bytes
dwFeatureList_getCurrentTimeIdx() dwFeatureHistoryArray::currentTimeIdx
dwFeatureList_setCurrentTimeIdx() dwFeatureHistoryArray::currentTimeIdx
dwFeatureList_getFeatureListSize() dwFeatureHistoryArray::maxFeatures, dwFeatureHistoryArray::maxHistory

For new dw_imageprocessing APIs, there's no need to call selectValid(), compact() and proximityFilter() explicitly. They are automatically called within the DriveWorks module.

Example: In the old dw_feature APIs, the application needed to create a dwFeatureListHandle_t object and call dwFeatureList_getDataPointers() to get the interpretable dwFeatureListPointers.

// Old code
dwFeatureList_initialize(&featureList, maxFeatureCount, historyCapacity, width, height, 0, context);
void* d_database8;
size_t databytes;
dwFeatureListPointers featureDataGPU;
dwFeatureList_getDataBasePointer(&d_database8, &databytes, featureList);
dwFeatureList_getDataPointers(&featureDataGPU, d_database8, featureList));
// access the pointers of dwFeatureListPointers on GPU
featureDataGPU.statuses[...]
featureDataGPU.locationHistory[...]

If the application needed a CPU copy of the featureList handle, it first had to allocate CPU memory and call dwFeatureList_getDataPointers() to interpret it into dwFeatureListPointers.

// Old code
uint8_t* h_database8 = new uint8_t[databytes];
dwFeatureListPointers featureDataCPU;
dwFeatureList_getDataPointers(&featureDataCPU, h_database8, featureList));
// access the pointers of dwFeatureListPointers on CPU
featureDataCPU.statuses[...]
featureDataCPU.locationHistory[...]

It also needed the raw data pointer to copy data between the CPU and GPU.

// GPU -> CPU copy
cudaMemcpy(h_database8, d_database8, databytes, cudaMemcpyDeviceToHost);
// CPU -> GPU copy
cudaMemcpy(d_database8, h_database8, databytes, cudaMemcpyHostToDevice);

In the new dw_imageprocessing APIs, the application only needs to create the dwFeatureHistoryArray object on the CPU and GPU. It provides both raw pointers and interpretable pointers. It also provides the dwFeatureHistoryArray_copyAsync() API to easily copy data between two feature history arrays.

// New code
dwFeatureHistoryArray_create(&featureHistoryCPU, maxFeatureCount, historyCapacity, DW_MEMORY_TYPE_CPU, context);
dwFeatureHistoryArray_create(&featureHistoryGPU, maxFeatureCount, historyCapacity, DW_MEMORY_TYPE_CUDA, context);
// access the pointers on GPU
featureHistoryGPU.statuses[...]
featureHistoryGPU.locationHistory[...]
// access the pointers on CPU
featureHistoryCPU.statuses[...]
featureHistoryCPU.locationHistory[...]
// GPU -> CPU copy
dwFeatureHistoryArray_copyAsync(&featureHistoryCPU, &featureHistoryGPU, 0);
// CPU -> GPU copy
dwFeatureHistoryArray_copyAsync(&featureHistoryGPU, &featureHistoryCPU, 0);

The application can also create dwFeatureHistoryArray on pinned memory, it only needs to change DW_MEMORY_TYPE_CPU to DW_MEMORY_TYPE_PINNED during creation.

Like dwFeatureListHandle_t, there are also reset() and destroy() APIs. dwFeatureHistoryArray must be destroyed before the application exits.

// New code
dwFeatureHistoryArray_destroy(featureHistoryCPU);
dwFeatureHistoryArray_destroy(featureHistoryGPU);

Get particular timeIdx slice from dwFeatureHistoryArray

Similar to dwFeatureListPointers::locationHistory, dwFeatureHistoryArray::locationHistory stores historyCapacity time period of maxFeatureCount features. To get a particular feature[i] for timeIdx, it used to be:

// Old code
uint32_t currentTimeIdx;
dwFeatureList_getCurrentTimeIdx(&currentTimeIdx, featureList);
uint32_t histIdx = (currentTimeIdx + timeIdx) % historyCapacity;
dwVector2f* pLocation = featureDataCPU.locationHistory + histIdx * maxFeatureCount;
// get the particular feature[i]
pLocation[i]...

The new dw_imageprocessing modules provides a dwFeatureArray struct that contains features for a single time slice, so there's no need to compute histIdx any more.

// New code
dwFeatureArray features;
dwFeatureHistoryArray_get(&features, timeIdx, &featureHistoryCPU);
// get the particular feature[i]
features.locations[i]...

It can also get the most recent (top-most) time slice, or the slice that is 1 frame earlier than the top-most.

// New code
dwFeatureArray currentFeatures, previousFeatures;
dwFeatureHistoryArray_getCurrent(&currentFeatures, &featureHistoryCPU);
dwFeatureHistoryArray_getPrevious(&previousFeatures, &featureHistoryCPU);
// get the particular feature[i]
currentFeatures.locations[i]...
previousFeatures.locations[i]...

dwFeatureArray acquired by dwFeatureHistoryArray_getXXX APIs don't need to be destroyed as they are only shallow copies. Resources are automatically freed when the corresponding dwFeatureHistoryArray is destroyed. However, dwFeatureArray also provides creation APIs, where the usage is very similar to dwFeatureHistoryArray:

// New code
dwFeatureArray_create(&featureCPU, maxFeatureCount, DW_MEMORY_TYPE_CPU, context);
dwFeatureArray_create(&featureGPU, maxFeatureCount, DW_MEMORY_TYPE_CUDA, context);

dwFeatureArray created by dwFeatureArray_create() APIs must be destroyed by dwFeatureArray_destroy() before the application exits. Similarly, there is the dwFeatureArray_copyAsync() API to copy data between two feature array objects.

Port dwFeatureTrackerHandle_t to dwFeature2DDetectorHandle_t and dwFeature2DTrackerHandle_t

Harris corner detector handle dwFeature2DDetectorHandle_t

New imageprocessing module splits the Harris corner detector from dwFeatureTrackerHandle_t. The new Harris detector handle is dwFeature2DDetectorHandle_t.

Old name New name
dwFeatureTracker_initialize() dwFeature2DDetector_initialize()
dwFeatureTracker_initDefaultParams() dwFeature2DDetector_initDefaultParams()
dwFeatureTracker_reset() dwFeature2DDetector_reset()
dwFeatureTracker_release() dwFeature2DDetector_release()
dwFeatureTracker_setMask() dwFeature2DDetector_setMask()
dwFeatureTracker_detectNewFeatures() dwFeature2DDetector_detectFromPyramid()
dwFeature2DDetector_detectFromImage()

The new dwFeature2DDetectorConfig parameter not only covers all settings from the old dwFeatureTrackerConfig parameter, it also provides more flexible settings.

Old dwFeatureTrackerConfig New dwFeature2DDetectorConfig
type
dwFeatureTrackerConfig::imageWidth dwFeature2DDetectorConfig::imageWidth
dwFeatureTrackerConfig::imageHeight dwFeature2DDetectorConfig::imageHeight
dwFeatureTrackerConfig::maxFeatureCount dwFeature2DDetectorConfig::maxFeatureCount
dwFeature2DDetectorConfig::harrisK
dwFeatureTrackerConfig::cellSize dwFeature2DDetectorConfig::cellSize
dwFeatureTrackerConfig::detectorScoreThreshold dwFeature2DDetectorConfig::scoreThreshold
dwFeatureTrackerConfig::detectorDetailThreshold dwFeature2DDetectorConfig::detailThreshold
dwFeatureTrackerConfig::numEvenDistributionPerCell dwFeature2DDetectorConfig::numEvenDistributionPerCell
dwFeature2DDetectorConfig::detectionLevel
dwFeature2DDetectorConfig::harrisRadius
dwFeature2DDetectorConfig::NMSRadius
dwFeature2DDetectorConfig::maskType
dwFeature2DDetectorConfig::isMaskAdjustmentEnabled

Example of initializing a detector with the old code:

// Old code
// Detector and Tracker share the same dwFeatureTrackerHandle_t
dwFeatureTrackerConfig trackerConfig{};
dwFeatureTracker_initDefaultParams(&trackerConfig);
trackerConfig.imageWidth = imageWidth;
trackerConfig.imageHeight = imageHeight;
trackerConfig.maxFeatureCount = maxFeatureCount;
trackerConfig.cellSize = cellSize;
trackerConfig.detectorScoreThreshold = scoreThreshold;
trackerConfig.detectorDetailThreshold = detailThreshold;
trackerConfig.numEvenDistributionPerCell = numEvenDistributionPerCell;
dwFeatureTracker_initialize(&tracker, &trackerConfig, 0, context);

Example of initializing a detector with the new code:

// New code
dwFeature2DDetectorConfig detectorConfig = {};
detectorConfig.imageWidth = imageWidth;
detectorConfig.imageHeight = imageHeight;
detectorConfig.maxFeatureCount = maxFeatureCount;
detectorConfig.cellSize = cellSize;
detectorConfig.scoreThreshold = scoreThreshold;
detectorConfig.detailThreshold = detailThreshold;
detectorConfig.numEvenDistributionPerCell = numEvenDistributionPerCell;
dwFeature2DDetector_initialize(&detector, &detectorConfig, 0, context);
LK tracker handle dwFeature2DTrackerHandle_t

The new imageprocessing module splits LK tracker from dwFeatureTrackerHandle_t. The new LK tracker handle is dwFeature2DTrackerHandle_t.

Old name New name
dwFeatureTracker_initialize() dwFeature2DTracker_initialize()
dwFeatureTracker_initDefaultParams() dwFeature2DTracker_initDefaultParams()
dwFeatureTracker_reset() dwFeature2DTracker_reset()
dwFeatureTracker_release() dwFeature2DTracker_release()
dwFeatureTracker_trackFeatures() dwFeature2DTracker_trackFeatures()
dwFeature2DTracker_getNewToOldMap()

The new dwFeature2DTrackerConfig parameter not only covers all settings from the the old dwFeatureTrackerConfig parameter, it also provides more flexible settings.

Old dwFeatureTrackerConfig New dwFeature2DTrackerConfig
dwFeature2DTrackerConfig::algorithm
dwFeature2DTrackerConfig::detectorType
dwFeature2DTrackerConfig::isOutputCompact
dwFeatureTrackerConfig::imageWidth dwFeature2DTrackerConfig::imageWidth
dwFeatureTrackerConfig::imageHeight dwFeature2DTrackerConfig::imageHeight
dwFeatureTrackerConfig::maxFeatureCount dwFeature2DTrackerConfig::maxFeatureCount
dwFeature2DTrackerConfig::historyCapacity
dwFeatureTrackerConfig::windowSizeLK dwFeature2DTrackerConfig::windowSizeLK
dwFeatureTrackerConfig::iterationsLK dwFeature2DTrackerConfig::numIterTranslationOnly
dwFeature2DTrackerConfig::numIterScaling
dwFeature2DTrackerConfig::numLevelTranslationOnly
dwFeature2DTrackerConfig::maxScaleChange
dwFeature2DTrackerConfig::nccKillThreshold
dwFeature2DTrackerConfig::nccUpdateThreshold

Example of initializing a tracker with the old code:

// Old code
// Detector and Tracker share the same dwFeatureTrackerHandle_t
dwFeatureTrackerConfig trackerConfig{};
dwFeatureTracker_initDefaultParams(&trackerConfig);
trackerConfig.imageWidth = imageWidth;
trackerConfig.imageHeight = imageHeight;
trackerConfig.maxFeatureCount = maxFeatureCount;
trackerConfig.windowSizeLK = windowSize;
trackerConfig.iterationsLK = numIterTranslation;
dwFeatureTracker_initialize(&tracker, &trackerConfig, 0, context);

Example of initializing a tracker with the new code:

// New code
dwFeature2DTrackerConfig trackerConfig = {};
trackerConfig.detectorType = detectorConfig.type;
trackerConfig.maxFeatureCount = maxFeatureCount;
trackerConfig.historyCapacity = historyCapacity;
trackerConfig.imageWidth = imageWidth;
trackerConfig.imageHeight = imageHeight;
trackerConfig.windowSizeLK = windowSize;
trackerConfig.numIterTranslationOnly = numIterTranslation;
dwFeature2DTracker_initialize(&tracker, &trackerConfig, 0, context);
Use the new imageprocessing module to detect and track Harris corners

Here's a complete example showing the pipeline with the old dw_feature APIs and new dw_imageprocessing APIs.

// Old dw_features code
// Create feature list
dwFeatureListHandle_t featureList;
dwFeatureList_initialize(&featureList, maxFeatureCount, historyCapacity, width, height, 0, context);
// create pyramid
dwPyramidHandle_t pyramidCurrent, pyramidPrevious;
dwPyramidConfig config{};
config.width = width;
config.height = height;
config.levelCount = numLevel;
config.dataType = DW_TYPE_UINT8;
dwPyramid_initialize(&pyramidCurrent, &config, cudaStream, context);
dwPyramid_initialize(&pyramidPrevious, &config, cudaStream, context);
// create tracker and detector
dwFeatureTrackerConfig trackerConfig{};
dwFeatureTracker_initDefaultParams(&trackerConfig);
trackerConfig.imageWidth = width;
trackerConfig.imageHeight = height;
trackerConfig.maxFeatureCount = maxFeatureCount;
dwFeatureTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
// Old dw_feature APIs requires additional CUDA buffers for compact.
uint32_t *d_validCount, *d_validIndexes, *d_invalidCount, *d_invalidIndexes;
cudaMalloc(&d_validCount, sizeof(uint32_t));
cudaMalloc(&d_validIndexes, sizeof(uint32_t) * maxFeatureCount);
cudaMalloc(&d_invalidCount, sizeof(uint32_t));
cudaMalloc(&d_invalidIndexes, sizeof(uint32_t) * maxFeatureCount);
// Assuming sensor provides single channel image as input
while (image = camera->getFrame())
{
std::swap(pyramidCurrent, pyramidPrevious);
dwPyramid_build(&image, pyramidCurrent);
// Track
dwFeatureTracker_trackFeatures(featureList, pyramidPrevious, pyramidCurrent, nullptr, tracker);
// Apply proximity filters to make features uniformly distributed
dwFeatureList_proximityFilter(featureList);
// Determine which features to throw away
dwFeatureList_selectValid(d_validCount, d_validIndexes,
d_invalidCount, d_invalidIndexes, featureList);
// Compact list
dwFeatureList_compact(featureList, d_validCount, d_validIndexes,
d_invalidCount, d_invalidIndexes);
// Detect
dwFeatureTracker_detectNewFeatures(featureList, pyramidCurrent, tracker);
}
// release resources
dwFeatureTracker_release(tracker);
dwPyramid_release(pyramidCurrent);
dwPyramid_release(pyramidPrevious);
dwFeatureList_release(featureList);
cudaFree(d_validCount);
cudaFree(d_validIndexes);
cudaFree(d_invalidCount);
cudaFree(d_invalidIndexes);
// New dw_imageprocessing code
// New dw_imageprocessing APIs requires a GPU dwFeatureArray as the output of Harris corner detector.
// It also requires a GPU dwFeatureHistoryArray as output of LK tracker.
dwFeatureArray featureDetectedGPU;
dwFeatureArray_create(&featureDetectedGPU, maxFeatureCount, DW_MEMORY_TYPE_CUDA, context);
dwFeatureHistoryArray featureHistoryGPU;
dwFeatureHistoryArray_create(&featureHistoryGPU, maxFeatureCount, historyCapacity, DW_MEMORY_TYPE_CUDA, context);
// Create pyramid
dwPyramidImage pyramidCurrent, pyramidPrevious;
dwPyramid_create(&pyramidCurrent, numLevel, width, height, DW_TYPE_UINT8, context);
dwPyramid_create(&pyramidPrevious, numLevel, width, height, DW_TYPE_UINT8, context);
// create detector
dwFeature2DDetectorConfig detectorConfig = {};
detectorConfig.imageWidth = imageWidth;
detectorConfig.imageHeight = imageHeight;
detectorConfig.maxFeatureCount = maxFeatureCount;
dwFeature2DDetector_initialize(&detector, &detectorConfig, cudaStream, context);
// create tracker
dwFeature2DTrackerConfig trackerConfig = {};
trackerConfig.detectorType = detectorConfig.type;
trackerConfig.maxFeatureCount = maxFeatureCount;
trackerConfig.historyCapacity = historyCapacity;
trackerConfig.imageWidth = imageWidth;
trackerConfig.imageHeight = imageHeight;
dwFeature2DTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
// Assuming sensor provides single channel image as input, all handles and buffers are initialized.
while (image = camera->getFrame())
{
std::swap(pyramidCurrent, pyramidPrevious);
dwImageFilter_computePyramid(&pyramidCurrent, &imageY, cudaStream, context);
// Track
dwFeatureArray featurePredicted;
dwFeature2DTracker_trackFeatures(&featureHistoryGPU, &featurePredicted, nullptr,
&featureDetectedGPU, nullptr,
&pyramidPrevious, &pyramidCurrent, tracker);
// Detect
dwFeature2DDetector_detectFromPyramid(&featureDetectedGPU, &pyramidCurrent,
&featurePredicted, nullptr, detector);
}
// release resources
dwFeatureHistoryArray_destory(featureHistoryGPU);
dwFeatureArray_destroy(featureDetectedGPU);
dwPyramid_destroy(pyramidCurrent);
dwPyramid_destroy(pyramidPrevious);
Advanced detector and tracker

The new dwFeature2DDetectorHandle_t and dwFeature2DTrackerHandle_t provides advanced detecting/tracking options. To enable advanced mode, the application needs to set the detector type as DW_FEATURE2D_DETECTOR_TYPE_EX and the tracker algorithm as DW_FEATURE2D_TRACKER_ALGORITHM_EX:

dwFeature2DDetectorConfig detectorConfig = {};
// ... Other settings
dwFeature2DTrackerConfig trackerConfig = {};
trackerConfig.detectorType = detectorConfig.type;
// ... Other settings

Please refer to documentation for more details regarding available configuration.

Advanced mode also requires an additional device buffer nccScore. It can be left as nullptr, but a valid nccScore buffer is recommended for better tracking/detecting quality.

float32_t* nccScore = nullptr;
cudaMalloc(&nccScore, sizeof(float32_t) * maxFeatureCount);
// Initialization code
// ...
// Start detecting/tracking loop
while (image = camera->getFrame())
{
std::swap(pyramidCurrent, pyramidPrevious);
dwImageFilter_computePyramid(&pyramidCurrent, &imageY, cudaStream, context);
// Track
dwFeatureArray featurePredicted;
dwFeature2DTracker_trackFeatures(&featureHistoryGPU, &featurePredicted, nccScore,
&featureDetectedGPU, nullptr,
&pyramidPrevious, &pyramidCurrent, tracker);
// Detect
dwFeature2DDetector_detectFromPyramid(&featureDetectedGPU, &pyramidCurrent,
&featurePredicted, nccScore, detector);
}

The nccScore buffer is recommended only when both the detector and tracker are in advanced mode. It's valid to connect the DW_FEATURE2D_DETECTOR_TYPE_STD normal detector with the DW_FEATURE2D_TRACKER_ALGORITHM_EX advanced tracker, or to connect the DW_FEATURE2D_DETECTOR_TYPE_EX advanced detector with the DW_FEATURE2D_TRACKER_ALGORITHM_STD advanced tracker. The application only needs to set dwFeature2DTrackerConfig::detectorType with the correct value of its corresponding detector.

Working with SFM APIs

Below are the recommended steps for using imageprocessing APIs in SFM:

  1. Set dwFeature2DTrackerConfig:.algorithm = DW_FEATURE2D_TRACKER_ALGORITHM_SFM to allow sparse tracking results output.
  2. Perform tracking and reconstruction. DO NOT detect new features immediately after tracking.
  3. Call dwFeature2DTracker_compact() explicitly after SFM reconstruction is done. Fill the d_newToOldMap device array by dwFeature2DTracker_getNewToOldMap().
  4. Call dwReconstructor_compact* by d_newToOldMap received in step 3.
  5. Detect new features when compact is done.

Pseudo code for a typical workflow:

// Track features
// Reconstruction
// ...
// Compact data
dwFeature2DTracker_getNewToOldMap();
// Detect new features

For the full implementation refer to Structure from Motion (SFM) Sample.

Port dwScalingFeatureListHandle_t and dwScalingFeatureListPointers to dwTemplateArray

Similar to dwFeatureHistoryArray, template tracker provides a dwTemplateArray to replace the old dwScalingFeatureListHandle_t and dwScalingFeatureListPointers.

The new dwTemplateArray struct provides pointers such as: *bboxes, *statuses, *ids, *ages, *scaleFactors and *templateCount like the old dwScalingFeatureListPointers did. It is also created and destroyed like old handle dwScalingFeatureListHandle_t. dwTemplateArray can be created on page-able/pinned CPU or CUDA memory.

Old name New name
dwScalingFeatureListPointers::locations dwTemplateArray::bboxes
dwScalingFeatureListPointers::sizes dwTemplateArray::bboxes
dwScalingFeatureListPointers::statuses dwTemplateArray::statuses
dwScalingFeatureListPointers::ids dwTemplateArray::ids
dwScalingFeatureListPointers::ages dwTemplateArray::ages
dwScalingFeatureListPointers::scaleFactors dwTemplateArray::scaleFactors
dwScalingFeatureListPointers::bNewTemplate
dwScalingFeatureListPointers::featureCount dwTemplateArray::templateCount
dwScalingFeatureList_initialize() dwTemplateArray_create()
dwScalingFeatureList_release() dwTemplateArray_destroy()
dwScalingFeatureList_reset() dwTemplateArray_reset()
dwScalingFeatureList_getDataBasePointer() dwTemplateArray::data, dwTemplateArray::bytes

For new dw_imageprocessing APIs, there's no need to call addEmptyFeature, selectValid(), compact() and applySizeFilter() explicitly. They are automatically called within the DriveWorks module.

Example: In the old dw_feature APIs, the application needed to create a dwScalingFeatureListHandle_t object and call dwScalingFeatureList_getDataPointers() to get the interpretable dwScalingFeatureListPointers.

// Old code
dwScalingFeatureList_initialize(&templateList, 0, maxTemplateCount, pxlType, context);
void* d_database8;
size_t databytes;
dwScalingFeatureListPointers templateDataGPU;
dwScalingFeatureList_getDataBasePointer(&d_database8, &databytes, templateList);
dwScalingFeatureList_getDataPointers(&templateDataGPU, d_database8, templateList));
// access the pointers of dwFeatureListPointers on GPU
templateDataGPU.locations[...]
templateDataGPU.sizes[...]

If the application needed a CPU copy of the templateList handle, it first had to allocate CPU memory and call dwScalingFeatureList_getDataPointers() to interpret it into dwScalingFeatureListPointers.

// Old code
uint8_t* h_database8 = new uint8_t[databytes];
dwScalingFeatureListPointers templateDataCPU;
dwScalingFeatureList_getDataPointers(&templateDataCPU, h_database8, templateList));
// access the pointers of dwScalingFeatureListPointers on CPU
templateDataCPU.locations[...]
templateDataCPU.sizes[...]

It also needed the raw data pointer to copy data between the CPU and GPU.

// GPU -> CPU copy
cudaMemcpy(h_database8, d_database8, databytes, cudaMemcpyDeviceToHost);
// CPU -> GPU copy
cudaMemcpy(d_database8, h_database8, databytes, cudaMemcpyHostToDevice);

In the new dw_imageprocessing APIs, the application only needs to create the dwTemplateArray object on the CPU and GPU. It provides both raw pointers and interpretable pointers. It also provides the dwTemplateArray_copyAsync() API to easily copy data between two template arrays.

// New code
dwTemplateArray_create(&templateCPU, maxFeatureCount, DW_MEMORY_TYPE_CPU, context);
dwTemplateArray_create(&templateGPU, maxFeatureCount, DW_MEMORY_TYPE_CUDA, context);
// access the pointers on GPU
templateGPU.statuses[...]
templateGPU.bboxes[...]
// access the pointers on CPU
templateCPU.statuses[...]
templateCPU.bboxes[...]
// GPU -> CPU copy
dwTemplateArray_copyAsync(&templateCPU, &templateGPU, 0);
// CPU -> GPU copy
dwTemplateArray_copyAsync(&templateGPU, &templateCPU, 0);

The application can also create dwTemplateArray on pinned memory, it only needs to change DW_MEMORY_TYPE_CPU to DW_MEMORY_TYPE_PINNED during creation.

Like dwScalingFeatureListHandle_t, there are also reset() and destroy() APIs. dwTemplateArray must be destroyed before the application exits.

// New code

Port dwScalingFeatureTrackerHandle_t to dwTemplateTrackerHandle_t

Old name New name
dwScalingFeatureTracker_initialize() dwTemplateTracker_initialize()
dwScalingFeatureTracker_initDefaultParams() dwTemplateTracker_initDefaultParams()
dwScalingFeatureTracker_reset() dwTemplateTracker_reset()
dwScalingFeatureTracker_release() dwTemplateTracker_release()
dwScalingFeatureTracker_trackAsync() dwTemplateTracker_trackImage()
dwTemplateTracker_trackPyramid()
dwFeature2DTracker_getNewToOldMap()

For new dw_imageprocessing APIs, there's no need to call dwScalingFeatureTracker_updateTemplateAsync() explicitly. It is automatically called within the DriveWorks module.

The new dwTemplateTrackerParameters parameter covers all settings from the old dwScalingFeatureTrackerConfig and dwScalingFeatureListParameters parameters.

Old dwScalingFeatureXXX settings New dwTemplateTrackerParameters
dwTemplateTrackerParameters::type
dwScalingFeatureListParameters::maxFeatureCount dwTemplateTrackerParameters::maxTemplateCount
dwScalingFeatureListParameters::imageWidth dwTemplateTrackerParameters::imageWidth
dwScalingFeatureListParameters::imageHeight dwTemplateTrackerParameters::imageHeight
dwScalingFeatureListParameters::pyramidLevelCount dwTemplateTrackerParameters::maxPyramidLevel
dwScalingFeatureTrackerConfig::numIterCoarse dwTemplateTrackerParameters::numIterationsCoarse
dwScalingFeatureTrackerConfig::numIterFine dwTemplateTrackerParameters::numIterations
dwScalingFeatureTrackerConfig::thresholdUpdate dwTemplateTrackerParameters::thresholdUpdate
dwScalingFeatureTrackerConfig::thresholdKill dwTemplateTrackerParameters::thresholdKill
dwScalingFeatureTrackerConfig::thresholdStop dwTemplateTrackerParameters::thresholdStop
dwScalingFeatureTrackerConfig::maxScaleChange dwTemplateTrackerParameters::maxScaleChange
dwTemplateTrackerParameters::validWidth
dwTemplateTrackerParameters::validHeight

Here's a complete example showing the template tracker pipeline with the old dw_feature APIs and new dw_imageprocessing APIs.

old dw_feature
// Old dw_features code
// Create template list
dwScalingFeatureListHandle_t templateList;
dwScalingFeatureList_initialize(&templateList, cudaStream, maxTemplateCount, DW_TYPE_UINT8, context);
// create pyramid
dwPyramidHandle_t pyramidCurrent, pyramidPrevious;
dwPyramidConfig config{};
config.width = width;
config.height = height;
config.levelCount = numLevel;
config.dataType = DW_TYPE_UINT8;
dwPyramid_initialize(&pyramidCurrent, &config, cudaStream, context);
dwPyramid_initialize(&pyramidPrevious, &config, cudaStream, context);
// create tracker and detector
dwScalingFeatureTrackerConfig trackerConfig{};
dwScalingFeatureTracker_initDefaultParams(&trackerConfig);
dwScalingFeatureTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
// Old dw_feature APIs requires additional CUDA buffers for compact.
uint32_t *d_validCount, *d_validIndexes, *d_invalidCount, *d_invalidIndexes;
cudaMalloc(&d_validCount, sizeof(uint32_t));
cudaMalloc(&d_validIndexes, sizeof(uint32_t) * maxFeatureCount);
cudaMalloc(&d_invalidCount, sizeof(uint32_t));
cudaMalloc(&d_invalidIndexes, sizeof(uint32_t) * maxFeatureCount);
// add bboxes to track, assuming there're nBoxToTrack, bounding box info is stored in hBBoxes array
uint8_t* d_database8;
size_t databytes;
dwScalingFeatureList_getDataBasePointer(&d_database8, &databytes, templateList);
uint8_t* h_database8 = new uint8_t[databytes];
dwScalingFeatureListPointers templateDataCPU;
dwScalingFeatureList_getDataPointers(&templateDataCPU, h_database8, templateList));
// assuming there're nBoxToTrack stored in hBBoxes array
for (uint32_t i = 0; i < nBoxToTrack; i++)
{
templateDataCPU.locations[i] = {hBBoxes[i].x, hBBoxes[i].y};
templateDataCPU.sizes[i] = {hBBoxes[i].width, hBBoxes[i].height};
templateDataCPU.statuses[i] = DW_FEATURE_STATUS_DETECTED;
templateDataCPU.ages[i] = 1;
templateDataCPU.bNewTemplate[i] = true;
}
*templateDataCPU.featureCount = nBoxToTrack;
cudaMemcpy(d_database8, h_database8, databytes, cudaMemcpyHostToDevice);
dwScalingFeatureList_addEmptyFeature(nBoxToTrack, dwScalingFeatureListHandle_t obj);
// Assuming sensor provides single channel image as input
while (image = camera->getFrame())
{
std::swap(pyramidCurrent, pyramidPrevious);
dwPyramid_build(&image, pyramidCurrent);
dwImageCUDA *imageCurrent, *imagePrevious;
dwPyramid_getLevelImageCUDA(&imageCurrent, 0, pyramidCurrent);
dwPyramid_getLevelImageCUDA(&imagePrevious, 0, pyramidPrevious);
// Update template
dwScalingFeatureTracker_updateTemplateAsync(templateList, imagePrevious, tracker);
// Track
dwScalingFeatureTracker_trackAsync(templateList, imageCurrent, tracker);
// Apply proximity filters to make features uniformly distributed
dwScalingFeatureList_applySizeFilter(maxWidth, maxHeight, templateList);
// Determine which features to throw away
dwScalingFeatureList_selectValid(d_validCount, d_validIndexes,
d_invalidCount, d_invalidIndexes, templateList);
// Compact list
dwScalingFeatureList_compact(templateList, d_validCount, d_validIndexes,
d_invalidCount, d_invalidIndexes);
}
// release resources
delete[] h_database8;
dwScalingFeatureList_release(featureList);
dwScalingFeatureTracker_release(tracker);
dwPyramid_release(pyramidCurrent);
dwPyramid_release(pyramidPrevious);
cudaFree(d_validCount);
cudaFree(d_validIndexes);
cudaFree(d_invalidCount);
cudaFree(d_invalidIndexes);
new dw_imageprocessing
// New dw_imageprocessing code
// create template array
dwTemplateArray templateGPU, templateCPU;
dwTemplateArray_create(&templateGPU, maxTemplateCount, DW_MEMORY_TYPE_CUDA, context);
dwTemplateArray_create(&templateCPU, maxTemplateCount, DW_MEMORY_TYPE_CPU, context);
// Create pyramid
dwPyramidImage pyramidCurrent, pyramidPrevious;
dwPyramid_create(&pyramidCurrent, numLevel, width, height, DW_TYPE_UINT8, context);
dwPyramid_create(&pyramidPrevious, numLevel, width, height, DW_TYPE_UINT8, context);
// create template tracker
dwTemplateTrackerParameters trackerConfig = {};
trackerConfig.maxTemplateCount = maxTemplateCount;
trackerConfig.imageWidth = width;
trackerConfig.imageHeight = height;
trackerConfig.validWidth = maxWidth;
trackerConfig.validHeight = maxHeight;
dwTemplateTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
// assuming there're nBoxToTrack stored in hBBoxes array
for (uint32_t i = 0; i < nBoxToTrack; i++)
{
templateCPU.bboxes[i] = hBBoxes[i];
templateCPU.ages[i] = 1;
}
*templateCPU.templateCount = nBoxToTrack;
dwTemplateArray_copyAsync(&templateGPU, &templateCPU, 0);
// Assuming sensor provides single channel image as input, all handles and buffers are initialized.
while (image = camera->getFrame())
{
std::swap(pyramidCurrent, pyramidPrevious);
dwImageFilter_computePyramid(&pyramidCurrent, &imageY, cudaStream, context);
dwImageCUDA *imageCurrent, *imagePrevious;
dwImage_getCUDA(&imageCurrent, pyramidCurrent.levelImages[0]);
dwImage_getCUDA(&imagePrevous, pyramidPrevious.levelImages[0]);
// Track
dwTemplateTracker_trackImage(&templateGPU, imageCurrent, imagePrevious, tracker);
}
// release resources
dwPyramid_destroy(pyramidCurrent);
dwPyramid_destroy(pyramidPrevious);

Lane Graph

The APIs previously contained in dw/world/LaneGraphProducer.h are no longer available, while the data structures are now part of LaneGraph.h.


Lane Perception

The following data structures have been moved from LaneDetector.h to LandmarkDetectorTypes.h:

  • ::dwLanePositionType.
  • ::dwLaneMarking.
  • ::dwLaneDetection.

Lidar Accumulator

The following API has been removed:

dwStatus dwLidarAccumulator_hasFullSweepAvailable(bool* available, dwLidarAccumulatorHandle_t obj);

dwLidarAccumulator_addPacket() will return DW_SUCCESS if the full sweep is ready.

For more information see Lidar Accumulator (replaced with `dwPointCloudAccumulator`).


Maps

The following APIs have been renamed:

Old name New name
dwMaps_transformLaneDividersToLocalLinesPerLocalLayout() dwMaps_transformLaneDividersToLocalLines()
dwMaps_transformRoadFeaturesToLocalLinesPerLocalLayout() dwMaps_transformRoadFeaturesToLocalLines()
dwMaps_computeBearing() dwMaps_computeBearingFromGeoPoints()
dwMaps_computeBearingFromRotMatrix() dwMaps_computeBearingFromRotation()
dwMaps_computeLocalToENUMatrix() dwMaps_computeRotationFromBearing()
dwMapTracker_updateCurrentPose() dwMapTracker_updateWithGlobalPose()

The following API has been removed:

dwStatus dwMaps_transformLaneDividersToLocalLines(dwMapsLaneDividerLineBuffer* lineSegments,
const dwMapsLaneDividerBuffer* laneDividers,
const dwMapsGeoPoint* localOrigin,
const dwMatrix3d* localToENURotation33,
const dwMapsLocalBounds* bounds,
const dwVector3f* directionFilterVector,
float32_t directionFilterAngleRadian,
bool ignoreLaneDirection);

dwMaps_transformLaneDividersToLocalLines() should be used instead.

For more information see maps_mainsection.


Object Detection

The dwObjectDetector API has been simplified. The functions

  • dwObjectDetector_inferDeviceAsync()
  • dwObjectDetector_interpretHost()
  • dwObjectDetector_getClusteredObjects()
  • dwObjectDetector_getRawDetections()
  • dwObjectDetector_bindInput()
  • dwObjectDetector_bindOutput()
  • dwObjectDetector_processDeviceAsync()
  • dwObjectDetector_processHost()

have been removed in favor of dwObjectDetector_detectObjects().

For more information see object_mainsection.


Objects

The following data structures have been renamed:

Old name New name
dwObjectData dwObjectGeneric
dwObjectDataCamera dwObjectCamera
dwObjectDataRadar dwObjectRadar
dwObjectDataLidar dwObjectLidar
dwObjectDataFused dwObjectFused

dwObjectHandle_t has been removed and replaced with plain structs. In place of dwObjectHandleList, use dwObjectArray. Becase object handles are no longer part of the code base, the following APIs have been removed:

Removed APIs
dwStatus dwObject_createCamera(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataCamera* otherData)
dwStatus dwObject_createRadar(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataRadar* otherData)
dwStatus dwObject_createLidar(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataLidar* otherData)
dwStatus dwObject_createFused(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataFused* otherData)
dwStatus dwObject_destroy(dwObjectHandle_t handle)
dwStatus dwObject_reset(dwObjectHandle_t handle)
dwStatus dwObject_getData(dwObjectData* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataCamera(dwObjectDataCamera* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataRadar(dwObjectDataRadar* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataLidar(dwObjectDataLidar* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataFused(dwObjectDataFused* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_setData(const dwObjectData* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataCamera(const dwObjectDataCamera* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataRadar(const dwObjectDataRadar* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataLidar(const dwObjectDataLidar* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataFused(const dwObjectDataFused* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_addDataCamera(const dwObjectData* info, const dwObjectDataCamera* otherInfo, dwObjectHandle_t handle)
dwStatus dwObject_addDataRadar(const dwObjectData* info, const dwObjectDataRadar* otherInfo, dwObjectHandle_t handle)
dwStatus dwObject_addDataLidar(const dwObjectData* info, const dwObjectDataLidar* otherInfo, dwObjectHandle_t handle)
dwStatus dwObject_addDataFused(const dwObjectData* info, const dwObjectDataFused* otherInfo, dwObjectHandle_t handle)

Because dwObjectHandle_t is no longer part of the code base, calling create to create an object is no longer necessary. For example, the old way of creating an object required calling:

dwObjectHandle_t objHandle = DW_NULL_HANDLE;
dwObjectData data{};
dwObjectDataCamera cameraData{};
dwObject_createCamera(&objHandle, &data, &cameraData);

The new way to do this is written as:

dwObjectCamera obj{};

Object history is no longer stored as part of the object, but is now a separate history array: dwObjectHistoryArray. The old way to get a historical object from an object handle was:

uint32_t historicalIndex = 10; // Arbitrary historical index, 0 being the most recent
dwObject_getData(&data, historicalIndex, objHandle);

The new way to do this is written as:

uint32_t historicalIndex = 10;
uint32_t objectIndex = 0; // This would be the index of the specific object in the history array
dwObjectArray objArray{};
...
dwObjectCamera* objects = static_cast<dwObjectCamera*>(objArray.objects);
objects[historicalIndex * objArray.maxObjectCount + objectIndex];

In order to add data to the history, you no longer call an API but use the indices into the objects array to insert into the history table.

Object arrays have also been introduced as dwObjectArray. These work very similarly to dwObjectHistoryArray except that it is a 1D contiguous array of objects.


Point Cloud Processing

dwPointCloudMemoryType has been replaced by the following data structure:

dwPointCloudRangeImage structure has been removed. Range image creator now accepts dwImageHandle_t instead.

For more information see Point Cloud Processing.


Rig Configuration

The module dwRigConfiguration has been renamed dwRig.

  • Header file has been renamed from RigConfiguration.h to Rig.h.
  • The object handles have been renamed to dwRigHandle_t and dwConstRigHandle_t, respectively.
  • The prefixes for all functions declared in Rig.h have been updated from dwRigConfiguration to dwRig.
  • dwRig_initializeCalibratedCamera is now dwCameraModel_initialize in calibration/cameramodel/CameraModel.h.

SFM

  • Feature status has been changed from dwFeatureStatus to dwFeature2DStatus.
  • Prototype for dwReconstructor_compactWorldPoints and dwReconstructor_compactFeatureHistory has changed. The input has changed from const uint32_t* d_validIndexCount, const uint32_t* d_validIndexes, const uint32_t* d_invalidIndexCount, const uint32_t* d_invalidIndexes to const uint32_t* d_validIndexCount, const uint32_t* d_newToOldMap, while d_newToOldMap is obtained with dwFeature2DTracker_getNewToOldMap.
  • Please refer to the "Working with SFM APIs" section in Image Processing for how to work with new dw_imageprocessing APIs.

For more information see Structure from Motion (SFM) and Features.


VehicleIO

This release adds support for multiple simultanous VehicleIO backends, and the following changes are required:

For more information see VehicleIO.


Wait Conditions

The previous API header file waitcondition/Classifier.h has been split into waitcondition/camera/TrafficLightSignClassifier.h and waitcondition/camera/TrafficLightSignClassifier_processPipeline.h.

  • dwObjectClassifierHandle_t has been renamed to dwTrafficLightSignClassifierHandle_t.

The following APIs have been renamed:

Old name New name
dwObjectClassifier_initializeFromLightNet(dwObjectClassifierHandle_t* obj, dwLightNetHandle_t dnn, dwContextHandle_t ctx) dwTrafficLightSignClassifier_initializeFromLightNet(dwTrafficLightSignClassifierHandle_t* obj, dwLightNetHandle_t dnn, dwContextHandle_t ctx)
dwObjectClassifier_initializeFromSignNet(dwObjectClassifierHandle_t* obj, dwSignNetHandle_t dnn, dwContextHandle_t ctx) dwTrafficLightSignClassifier_initializeFromSignNet(dwTrafficLightSignClassifierHandle_t* obj, dwSignNetHandle_t dnn, dwContextHandle_t ctx)
dwObjectClassifier_reset(dwObjectClassifierHandle_t obj) dwTrafficLightSignClassifier_reset(dwTrafficLightSignClassifierHandle_t obj)
dwObjectClassifier_release(dwObjectClassifierHandle_t obj) dwTrafficLightSignClassifier_release(dwTrafficLightSignClassifierHandle_t obj)
dwObjectClassifier_setCUDAStream(cudaStream_t stream, dwObjectClassifierHandle_t obj) dwTrafficLightSignClassifier_setCUDAStream(cudaStream_t stream, dwTrafficLightSignClassifierHandle_t obj)
dwObjectClassifier_getCUDAStream(cudaStream_t* stream, dwObjectClassifierHandle_t obj) dwTrafficLightSignClassifier_getCUDAStream(cudaStream_t* stream, dwTrafficLightSignClassifierHandle_t obj)
dwObjectClassifier_interpretHost(uint32_t numObjects, dwObjectClass type, dwObjectClassifierHandle_t obj)dwTrafficLightSignClassifier_interpretHost(uint32_t numObjects, dwObjectClass type, dwTrafficLightSignClassifierHandle_t obj)
dwObjectClassifier_inferDeviceAsync(const dwImageCUDA* const* imageArray, dwObject* objectList, uint32_t numObjects, dwObjectClass type, dwObjectClassifierHandle_t obj) dwTrafficLightSignClassifier_inferDeviceAsync(const dwImageCUDA* const* imageArray, dwObjectCamera* objectList, uint32_t numObjects, dwObjectClass type, dwTrafficLightSignClassifierHandle_t obj)
dwObjectClassifier_getClassifiedObjects(dwObject* objectList, uint32_t numObjects, dwObjectClassifierHandle_t obj) dwTrafficLightSignClassifier_getClassifiedObjects(dwObjectCamera* objectList, uint32_t numObjects, dwTrafficLightSignClassifierHandle_t obj)

For more information see waitcondition_mainsection.