dnn/dataconditioner/docs/usecase1.md

Go to the documentation of this file.

# Copyright (c) 2019-2020 NVIDIA CORPORATION.  All rights reserved.

@page dataconditioner_usecase1 Data Conditioner Workflow

@note SW Release Applicability: This tutorial is applicable to modules in both **NVIDIA DriveWorks** and **NVIDIA DRIVE Software** releases.

This code snipped shows how the Data Conditioner module is typically used. Note that error handling is left out for clarity.

### Initialization

Initialize DataConditioner parameters with default values.

In order to initialize the Data Conditioner module, it is first required to initialize the Data Conditioner parameters

```{.cpp}
    dwDataConditionerParams params{};
    dwStatus dwDataConditioner_initParams(dwDataConditionerParams &params);
```

`dwDataConditionerParams()` permits to set the following parameters:

- float32_t meanValue[::DW_MAX_IMAGE_PLANES]: mean value to be subtracted from each input image pixel. Default is the 0-vector. This shall be used if the network has been trained with mean-centered data. 
- dwImageCUDA *meanImage: mean image to be subtracted from each input image. meanImage is expected to be float16 or float32. The pixel format is required to be R or RGBA with interleaved channels. The dimensions of the mean image must meet the dimensions of the network input. Default is the null pointer. This is an alternative to  meanValue, if a specific mean image is to be subtracted. Note: if both  meanValue and meanImage are provided, both values are subtracted.
- dwBool splitPlanes: flag to indicate whether the image is in interleaved (false) or planar (true) format.  Default is false.
- float32_t scaleCoefficient: Scale pixel intensities. Default is 1.0. It shall be used if the network has been trained with images whose pixel values has been scaled to a specified range, e.g., in [0, 1]. If scaleCoefficient is 1.0, the output pixel intensities are always ranged between [0,255], regardless of the input pixel intensity range.
- dwBool ignoreAspectRatio: Indicates whether aspect ratio must be ignored during scaling operation. Default is false. 
- dwBool doPerPlaneMeanNormalization: Indicates whether per-plane mean normalization must be performed. If true, mean value is computed for each plane on the image and this value is subtracted from pixel intensities of the corresponding plane. 

Modify parameters as required by the network or by the application.

```{.cpp}
    // Ignore aspect ratio of input image is to be scaled.
    params.ignoreAspectRatio = true;
    // Set mean value to {127.5, 127.5, 127.5}.
    params.meanValue[0] = 127.5;
    params.meanValue[1] = 127.5;
    params.meanValue[2] = 127.5;
```

Once the Data Conditioner parameters have been defined, the Data Conditioner object can be initialized. There are two ways to accomplish this:

```{.cpp}
    dwStatus dwDataConditioner_initialize(dwDataConditionerHandle_t *dataConditioner,
        const dwBlobSize *networkInputBlobSize,
        const dwDataConditionerParams *dataConditionerParams,
        cudaStream_t stream, dwContextHandle_t ctx);
```

In this case, the user is required to provide the network input blob dimensions, which can be acquired from `dwDNN` module via `dwDNN_getInputSize()` once the network is loaded. The batch size in networkInputBlobSize can be modified to allow for a batch of images to be prepared in parallel.


```{.cpp}
    dwStatus dwDataConditioner_initializeFromTensorProperties(dwDataConditionerHandle_t* obj,
        const dwDNNTensorProperties* outputProperties,
        uint32_t maxNumImages,
        const dwDataConditionerParams* dataConditionerParams,
        cudaStream_t stream, dwContextHandle_t ctx);
```

This initialization function requires `dwDNNTensorProperties`, which can also be acquired from `dwDNN` module via `dwDNN_getInputTensorProperties()`. `maxNumImages` determines how many images can be prepared in parallel. 

`ctx` is assumed to be a previously initialized `::dwContextHandle_t`.

### Data Preparation

Allocate CUDA memory to store the output of DataConditioner module. 

```{.cpp}
    // Compute the number elements to store in output
    size_t dataConditionerOutputTotalSize = dnnInputBlobSize.batch * dnnInputBlobSize.channels * dnnInputBlobSize.height * dnnInputBlobSize.width;
    cudaMalloc(&dataConditionerOutput, sizeof(float32_t) * dataConditionerOutputTotalSize);
```

Perform operations on a given image and store the result in `dataConditionerOutput`. All operations are performed asynchronously with the host code.
There are two possible ways to execute data preparation: with raw pointers and with `dwDNNTensorHandle_t`.

With raw pointers:

```{.cpp}
    // Set a region of interest in the given image. In this case, region of interest is the whole image.
    dwRect regionOfInterest{};
    regionOfInterest.width = inputImage.prop.width;
    regionOfInterest.height = inputImage.prop.height;

    // Prepare image using DataConditioner
    dwDataConditioner_prepareDataRaw(dataConditionerOutput, &inputImage, 1, &regionOfInterest, cudaAddressModeClamp, dataConditioner);
```

With `dwDNNTensorHandle_t`:

```{.cpp}
    // Set a region of interest in the given image. In this case, region of interest is the whole image.
    dwRect regionOfInterest{};
    regionOfInterest.width = inputImage.prop.width;
    regionOfInterest.height = inputImage.prop.height;

    dwDNNTensorHandle_t tensorOutput;

    // Allocate tensor output ...

    // Prepare image using DataConditioner
    dwDataConditioner_prepareData(tensorOutput, &inputImage, 1, &regionOfInterest, cudaAddressModeClamp, dataConditioner);
```

inputImages contains pointers to the images of the batch that shall be prepared. The number of images in inputImages is given through numImages and shall not exceed the (possibly modified) batch size of the network. The `regionOfInterest` parameter defines a specific region in all images to which the desired transformations as well as network inference shall be applied. The region of interest is identified by the coordinates of the top left corner, and the width and the height of the rectangle. If full images are of interest, then the top left corner must be set as (0,0), width and height according to the images at hand. The internal resizing of roi to match the network input size is defined in such a way that no image content is lost, but undefined border values might be created. addressMode shall be used to set the fill-up strategy for the undefined border values. Two modes are allowed cudaAddressModeBorder and cudaAddressModeClamp.

In a nutshell, `dwDataConditioner_prepareData()` or `dwDataConditioner_prepareDataRaw()` crops and resizes the defined ROI from each input image in order to match the network input size, applies the desired transformations, and returns the transformed image batch in `dataConditionerOutput`, which can be used as input to dwDNN_infer() (see @ref dnn_mainsection module Inference).

## Run Inference and Transform Output to Input Image Space

```
    // Prepared data is usually given to dwDNN module for inference.
    runInference();
```

Assuming that the network used in `runInference` returns coordinates relative to the output of the network, the interpreted output of the 
network is transformed back from the network input coordinate frame to the input image space.

```{.cpp}
    dwStatus dwDataConditioner_outputPositionToInput(float32_t *outputX,
        float32_t *outputY,
        float32_t inputX,
        float32_t inputY,
        const dwRect *regionOfInterest,
        dwDataConditionerHandle_t obj)
```
where the point to be transformed back is passed through inputX and inputY and is returned in outputX and outputY. The same `regionOfInterest` passed to `dwDataConditioner_prepareData()` shall be used.

## Free the Memory

Finally, free previously allocated memory.

```{.cpp}
    // Free resources
    cudaFree(dataConditionerOutput);
    dwDataConditioner_release(&dataConditioner)
```

For more information see:
- @ref dwx_object_detector_tracker_sample