Offline Data Augmentation

Training a deep neural network can be a daunting task, and the most important component of training a model is the data. Acquiring curated and annotated datasets is often a manual process involving thousands of person-hours of painstaking labelling. Even if the data is carfully collected and planned, it is very difficult to estimate all the corner cases that a network can have, and repeating the process of collecting missing data and annotating it is very expensive and has long turnover times.

Online augmentation with the training data loader is a good way to increase variation in the dataset. However, the augmented data is generated randomly based on the distribution the data loader follows when sampling the data. To achieve a high level of accuracy, the model may need to be trained for a long time.

To circumvent these limitations, generate a dataset with the required augmentations, and give control to the user, the TAO Data Services provides an Offline Data Augmentation service. Offline augmentation can dramatically increase the size of the dataset when collecting and labeling data is expensive or not possible. The augmentation service provides several custom GPU-accelerated augmentation routines:

• Spatial augmentation

• Color-space augmentation

• Image blur

Spatial Augmentation

Spatial augmentation comprises routines where data is augmented in space. The following spatial augmentation operations are supported:

• Rotate (optionally, with AI-assisted bounding-box refinement)

• Resize

• Translate

• Shear

• Flip

Color-Space Augmentation

Color-space augmentation comprises routines where the image data is augmented in the color space. The following operators are supported:

• Hue Rotation

• Brightness offset

• Contrast shift

Image Blur

Along with the above augmentation operations, the augmentation service also supports image blurring or sharpening, which is described further in FilterKernel config.

The spatial augmentation routines are applied to the images as well as the groundtruth labels, while the color augmentation routines and channel-wise blur operator are applied only to images. The sample workflow is as follows:

Offline Data Augmentation expects a directory of images and a directory containing KITTI text labels or a COCO JSON file.

Refer to the Data Annotation Format KITTI and COCO sections for more information about the data formats.

The configuration YAML file for augmentation experiements contains four major components, as well as a few global parameters for launching multi-GPU job and saving the job log.

• Spatial-augmentation config

• Color-augmentation config

• Blur config

• Data config (output image dimension, dataset type, etc.)

Spatial-Augmentation Config

The spatial-augmentation config contains parameters to configure the spatial augmentation routines.

            

            
{
angle: 0.5
units: degrees
}
        

            

            
{
flip_vertical: true
flip_horizontal: true
}
        

            

            
{
translate_x: 8
translate_y: 8
}
        

            

            
{
shear_ratio_x: 0.2
shear_ratio_y: 0.2
}
        

Rotation Config

The rotation operation rotates the image at an angle. The transformation matrix for shear operation is computed as follows:

            

            
[x_new, y_new, 1] = [x, y, 1] * [[cos(angle) sin(angle)              zero]
                                 [-sin(angle)           cos(angle)           zero]
                                 [x_t                         y_t                    one]]
Where x_t, y_t are defined as
x_t = height * sin(angle) / 2.0 - width * cos(angle) / 2.0 + width / 2.0
y_t = -1 * height * cos(angle) / 2.0 + height / 2.0 - width * sin_(angle) / 2.0
Here height = height of the output image, width = width of the output image.
        

Shear Config

The shear operation introduces a slant to the object along the x or y dimension. The transformation matrix for shear operation is computed as follows:

            

            
[x_new, y_new, 1] = [x, y, 1] * [[1.0             shear_ratio_y,    0],
                                          [shear_ratio_x,       1.0,         0],
                                    [x_t,                 y_t,       1.0]]
X_t = -height * shear_ratio_x / 2.
Y_t = -width * shear_ratio_y / 2.
Here height = height of the output image, width = width of the output image.
        

Flip Config

The operator flips an image and the bounding box coordinates along the horizontal and vertical axis.

Translation Config

The operator translates the image and polygon coordinates along the x and/or y axis.

Color Augmentation Config

The color augmentation config contains parameters to configure the color space augmentation routines, including the following:

Hue Config

This augmentation operator applies color space manipulation by converting the RGB image to HSV, performing hue rotation, and then returning with the corresponding RGB image.

Saturation Config

This augmentation operator applies color space manipulation by converting the RGB image to HSV, performing saturation shift, and then returning with the corresponding RGB image.

Brightness Config

This augmentation operator applies a channel-wise brightness shift.

Contrast Config

This augmentation operator applies contrast scaling across a center point to an image.

KernelFilter Config

The KernelFilter config applies the Gaussian blur operator to an image. A Gaussian kernel is formulated based on the following parameters, then a per-channel 2D convolution is performed between this image and kernel.

Data Config

The data config parameters configure the input and output data dimension, type, and location.

The offline augmentation service has a simple command line interface, which is defined as follows:

            

            
tao dataset augmentation generate [-h] -e <experiment spec file>
                                       --gpus <num_gpus>
        

• -e, --experiment_spec_file: The path to the YAML spec file

Optional Arguments

• --gpus: The number of GPUs to use. The default value is 1.

• -h, --help: Show this help message and exit.

Once the dataset is generated, you can use the code:dataset-convert tool to convert it to TFRecords so that it can be ingested by the train sub-task. Details about converting the data to TFRecords are described in the Data Input for Object Detection section, and training a model with this dataset is described in the Data Annotation Format section.

Below are some sample rendered augmented images:

Input image rotated by 5 degrees

Image rotated by 5 degrees, with hue rotation by 25 degrees and a saturation shift of 0.0