Offline Data Augmentation

Training a deep neural network can be a daunting task and the most important component of training a model is the data. Acquiring curated and annotated dataset can be a very tiring and manual process involving thousands of man hours of painstaking labelling. In spite of planning and collecting data, it is very difficult to estimate all the corner cases that a network may go through. Repeating the process of collecting the missing data and annotating is very expensive and has long turnover times.

Online augmentation in the training data loader is a good way to increase the variation in the dataset. However, the augmented data is generated randomly based on the distribution the data loader follows when sampling the data. In order to achieve good accuracy, the model may need to be trained for a long time. In order to circumvent this and generate a dataset with the required augmentations and give control to the user, TAO Toolkit provides an offline augmentation tool called augment. Offline augmentation can dramatically increase the size of the dataset when collecting and labeling data is expensive or not possible. The augment tools provides several custom GPU accelerated augmentation routines categorized into:

• Spatial augmentation

• Color space augmentation

• Image blur

Spatial augmentation comprises routines where data is augmented in space. The following spatial augmentation operations are supported in TAO Toolkit:

• Rotate

• Resize

• Translate

• Shear

• Flip

Color space augmentation comprises routines where the image data is augmented in the color space. The following color augmentations operators are supported:

• Hue Rotation

• Brightness offset

• Contrast shift

Along with the above mentioned augmentation operations, augment also enables use to blur images, using a Gaussian blur operator. More information about the operation is described in Blur config.

All augmentation routines currently provided with augment are supported only for an object detection dataset. The spatial augmentation routines are applied to the images as well as the labelled data coordinates, while the color augmentation routines and channel-wise blue operator are applied only to images as the object labels are not affected. The sample workflow of using augment is as follows:

Configuring the Augmentor

The augmentor has several components which the user can configure by using a simple protobuf based configuration file. The configuration file is divided into 4 major components.

1. Spatial augmentation config

2. Color augmentation config

3. Blur config

4. Data dimensions - output image width, output image height, output image channel, image extension.

This configuration file contains several nested protobuf elements and global parameters which are defined below.

Spatial Augmentation Config

Spatial augmentation config contains parameters to configure the spatial augmentation routines. This is a nested protobuf element called spatial_config containing protobuf elements for all the spatial augmentation operations.

            

            
{
angle: 0.5
units: degrees
}
        

            

            
{
flip_vertical: true
flip_horizontal: true
}
        

            

            
{
translate_x: 8
translate_y: 8
}
        

            

            
{
shear_ratio_x: 0.2
shear_ratio_y: 0.2
}
        

The augmentation operators may be enabled by simply defining the corresponding proto associated with it. When defining multiple proto elements, it implies that all the augmentation operations are cascaded.

If you don’t wish to introduce any of the supported augmentation operations, omit the field you wish to drop. The configurable parameters for the individual spatial augmentation operators are mentioned in the table below.

Rotation Config

The rotation operation rotates the image at an angle. The transformation matrix for shear operation is computed as:

            

            
[x_new, y_new, 1] = [x, y, 1] * [[cos(angle) sin(angle)              zero]
                                 [-sin(angle)           cos(angle)           zero]
                                 [x_t                         y_t                    one]]
Where x_t, y_t are defined as
x_t = height * sin(angle) / 2.0 - width * cos(angle) / 2.0 + width / 2.0
y_t = -1 * height * cos(angle) / 2.0 + height / 2.0 - width * sin_(angle) / 2.0
Here height = height of the output image, width = width of the output image.
        

Shear Config

The shear operation introduces a slant to the object along the x or the y dimension. The transformation matrix for shear operation is computed as:

            

            
[x_new, y_new, 1] = [x, y, 1] * [[1.0             shear_ratio_y,    0],
                                          [shear_ratio_x,       1.0,         0],
                                    [x_t,                 y_t,       1.0]]
X_t = -height * shear_ratio_x / 2.
Y_t = -width * shear_ratio_y / 2.
Here height = height of the output image, width = width of the output image.
        

Flip Config

This element configures the flip operator of augment. The operator flips an image and the bounding box coordinates along the horizontal and vertical axis.

Translation Config

This protobuf message configures the translation operator for augment. The operator translates the image and polygon coordinates along the x and/or y axis.

Color Augmentation Config

Color augmentation config contains parameters to configure the color space augmentation routines. This is a nested protobuf element called color_config containing protobuf elements for all the color augmentation operations.

            

            
{
hue_rotation_angle: 30
saturation_shift: 1.0
}
        

            

            
{
contrast: 0.0
center: 127.5
}
        

            

            
{
offset: 100
}
        

The augmentation operators may be enabled by simply defining the corresponding proto associated with it. When defining multiple proto elements, it implies that all the augmentation operations are cascaded.

If you don’t want to introduce any of the supported augmentation operations, simply omit the field you wish to drop. The configurable parameters for the individual color augmentation operators are mentioned in the table below.

Hue Saturation Config

This augmentation operator applies a color space manipulation by converting the RGB image to HSV applying hue rotation and saturation shift and then returning with the corresponding RGB image.

Brightness Config

This augmentation operator applies a channel-wise brightness shift.

Contrast Config

This augmentation operator applies contrast scaling across a center point to an image.

Dataloader

            

            
dataset_config {
  data_sources: {
    tfrecords_path: "/path/to/tfrecords/root/*"
    image_directory_path: "/path/to/dataset/root"
  }
  image_extension: "png"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_fold: 0
}
        

See Dataloader for more information.

Blur Config

This protobuf element configures the Gaussian blur operator to an image. A Gaussian kernel is formulated based on the parameters mentioned below and then a 2D convolution is performed between this image and kernel per channel.

For example, the following configuration file augments the image by:

1. Rotating an image by 5 deg.

2. Shearing along x axis by a ratio of 0.3.

3. Translating along x axis by 8 pixels.

            

            
# Spec file for augment.
spatial_config{
  rotation_config{
    angle: 5.0
    units: "degrees"
  }
  shear_config{
    shear_ratio_x: 0.3
  }
  translation_config{
    translate_x: 8
  }
}
color_config{
  hue_saturation_config{
    hue_rotation_angle: 25.0
    saturation_shift: 1.0
  }
}
# Setting up dataset config.
dataset_config{
  image_path: "image_2"
  label_path: "label_2"
}
output_image_width: 1248
output_image_height: 384
output_image_channel: 3
image_extension: ".png"
        

Running the Augmentor Tool

The augment tool has a simple command line interface, which is defined as follows:

            

            
tao augment [-h] -d /path/to/the/dataset/root
                 -a /path/to/augmentation/spec/file
                 -o /path/to/the/augmented/output
                 [-v]
        

Here are the command line parameters:

• -h, --help: Show this help message and exit

• -d, --dataset-folder: Path to the detection dataset

• -a, --augmentation-proto: Path to augmentation spec file

• -o, --output-dataset: Path to the augmented output dataset

• -v, --verbose: Flag to get detailed logs during the augmentation process

The augmented images and labels are generated in the path mentioned in the output-dataset parameter under the following directories mentioned in the dataset_config. For the sample config file mentioned above, the images and labels would be created in

• Augmented images: /path/to/augmented/output/image_2

• Augmented labels: /path/to/augmented/output/label_2

The log from a successful run of augment is mentioned below:

The dataset thus generated may then be used with the dataset-convert tool to be converted to TFRecords so that it may be ingested by the train sub task. The details about converting the data to TFRecords are described in the Data Input for Object Detection section, and training a model with this dataset is described in the Data Annotation Format section.

Sample rendered augmented images are shown below.

Input image rotated by 5 degrees

Image rotated by 5 degrees, hue rotation by 25 degrees and saturation shift of 0.0