DetectNet_v2

DetectNet_v2 is an NVIDIA-developed object-detection model that is included in the TAO Toolkit. DetectNet_v2 supports the following tasks:

• dataset_convert

• train

• evaluate

• inference

• prune

• calibration_tensorfile

• export

These tasks can be invoked from the TAO Toolkit Launcher using the following convention on the command-line:

            

            
tao detectnet_v2 <sub_task> <args_per_subtask>
        

where, args_per_subtask are the command-line arguments required for a given subtask. Each subtask is explained in detail in the following sections.

NVIDIA recommends following the workflow in the diagram below to generate a trained and optimized DetectNet_v2 model.

The object detection apps in TAO Toolkit expect data in KITTI format for training and evaluation.

See the Data Annotation Format page for more information about the KITTI data format.

The DetectNet_v2 app requires the KITTI formatted data to be converted to TFRecords for optimized iteration across the data batches. This can be done using the dataset_convert subtask under DetectNet_v2.

The dataset_convert tool requires a configuration file as input. Details of the configuration file and examples are included in the following sections.

Configuration File for Dataset Converter

The dataset_convert tool provides several configurable parameters. The parameters are encapsulated in a spec file to convert data from the KITTI format to the TFRecords format which the DetectNet_v2 trainer can ingest. This is a prototxt format file with two global parameters:

• kitti_config: A nested prototxt configuration with multiple input parameters

• image_directory_path: The path to the dataset root. The image_dir_name is appended to this path to get the input images and must be the same path specified in the experiment spec file.

Here are descriptions of the configurable parameters for the kitti_config field:

The sample configuration file shown below converts the Pascal VOC dataset with 80% training data and 20% validation data. This assumes that the data has been converted to KITTI format and is available for ingestion in the root directory path.

            

            
kitti_config {
  root_directory_path: "/workspace/tao-experiments/data/VOCtrainval_11-May-2012/VOCdevkit/VOC2012"
  image_dir_name: "JPEGImages_kitti/test"
  label_dir_name: "Annotations_kitti/test"
  image_extension: ".jpg"
  partition_mode: "random"
  num_partitions: 2
  val_split: 20
  num_shards: 10
}
image_directory_path: "/workspace/tao-experiments/data/VOCtrainval_11-May-2012/VOCdevkit/VOC2012"
        

Sample Usage of the Dataset Converter Tool

While KITTI is the accepted dataset format for object detection, the DetectNet_v2 trainer requires this data to be converted to TFRecord files for ingestion. The dataset_convert tool is described below:

            

            
tao detectnet_v2 dataset-convert [-h] -d DATASET_EXPORT_SPEC -o OUTPUT_FILENAME
                 [-f VALIDATION_FOLD]
        

You can use the follwoing optional arguments:

• -h, --help: Show this help message and exit

• -d, --dataset-export-spec: The path to the detection dataset spec containing the config for exporting .tfrecord files

• -o output_filename: The output filename

• -f, –validation-fold: The validation fold in 0-based indexing. This is required when modifying the training set, but otherwise optional.

The following example shows how to use the command with the dataset:

            

            
tao detectnet_v2 dataset_convert  [-h] -d <path_to_tfrecords_conversion_spec>
                                       -o <path_to_output_tfrecords>
        

The following is the output log from executing tao detectnet_v2 dataset_convert:

            

            
Using TensorFlow backend.
2019-07-16 01:30:59,073 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2019-07-16 01:30:59,243 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 10786    Val: 2696
2019-07-16 01:30:59,243 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validation set during training choose validation_fold 0.
2019-07-16 01:30:59,251 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:265: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2019-07-16 01:31:01,226 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
. .
sheep: 242
bottle: 205
..
boat: 171
car: 418
2019-07-16 01:31:20,772 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
..
2019-07-16 01:32:40,338 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
2019-07-16 01:32:49,063 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
sheep: 695
..
car: 1770

2019-07-16 01:32:49,064 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2019-07-16 01:32:49,064 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
sheep: 937
..
car: 2188
2019-07-16 01:32:49,064 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
sheep: sheep
..

boat: boat
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.



2019-07-16 01:32:49,064 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.
        

To perform training, evaluation, and inference for DetectNet_v2, you need to configure several components, each with their own parameters. The train and evaluate tasks for a DetectNet_v2 experiment share the same configuration file. The inference task uses a separate configuration file.

The specification file for DetectNet_v2 training configures these components of the training pipe:

• Model

• BBox ground truth generation

• Post processing module

• Cost function configuration

• Trainer

• Augmentation model

• Evaluator

• Dataloader

Model Config

The core object-detection model can be configured using the model_config option in the spec file.

The following is a sample model config to instantiate a ResNet-18 model with pretrained weights and freeze blocks 0 and 1 with all shortcuts set to projection layers.

            

            
# Sample model config for to instantiate a resnet18 model with pretrained weights and freeze blocks 0, 1
# with all shortcuts having projection layers.
model_config {
  arch: "resnet"
  pretrained_model_file: <path_to_model_file>
  freeze_blocks: 0
  freeze_blocks: 1
  all_projections: True
  num_layers: 18
  use_pooling: False
  use_batch_norm: True
  dropout_rate: 0.0
  objective_set: {
    cov {}
    bbox {
      scale: 35.0
      offset: 0.5
    }
  }
}
        

The following table describes the model_config parameters:

BBox Ground Truth Generator

DetectNet_v2 generates 2 tensors, cov and bbox. The image is divided into 16x16 grid cells. The cov tensor (short for “coverage” tensor) defines the number of grid cells that are covered by an object. The bbox tensor defines the normalized image coordinates of the object top left (x1, y1) and bottom right (x2, y2) with respect to the grid cell. For best results, you can assume the coverage area to be an ellipse within the bbox label with the maximum confidence assigned to the cells in the center and reducing coverage outwards. Each class has its own coverage and bbox tensor, thus the shape of the tensors are as follows:

• cov: Batch_size, Num_classes, image_height/16, image_width/16

• bbox: Batch_size, Num_classes * 4, image_height/16, image_width/16 (where 4 is the number of coordinates per cell)

Here is a sample rasterizer config for a 3 class detector:

            

            
# Sample rasterizer configs to instantiate a 3 class bbox rasterizer
bbox_rasterizer_config {
  target_class_config {
    key: "car"
    value: {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.4
      cov_radius_y: 0.4
      bbox_min_radius: 1.0
    }
  }
  target_class_config {
    key: "cyclist"
    value: {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.4
      cov_radius_y: 0.4
      bbox_min_radius: 1.0
    }
  }
  target_class_config {
    key: "pedestrian"
    value: {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.4
      cov_radius_y: 0.4
      bbox_min_radius: 1.0
    }
  }
  deadzone_radius: 0.67
}
        

The bbox_rasterizer has the following parameters that are configurable:

Post-Processor

The post-processor module generates renderable bounding boxes from the raw detection output. The process includes the following:

• Filtering out valid detections by thresholding objects using the confidence value in the coverage tensor.

• Clustering the raw filtered predictions using DBSCAN to produce the final rendered bounding boxes.

• Filtering out weaker clusters based on the final confidence threshold derived from the candidate boxes that get grouped into a cluster.

Here is an example of the definition of the post-processor for a 3-class network learning for car, cyclist, and pedestrian:

            

            
postprocessing_config {
  target_class_config {
    key: "car"
    value: {
      clustering_config {
        coverage_threshold: 0.005
        dbscan_eps: 0.15
        dbscan_min_samples: 0.05
        minimum_bounding_box_height: 20
      }
    }
  }
  target_class_config {
    key: "cyclist"
    value: {
      clustering_config {
        coverage_threshold: 0.005
        dbscan_eps: 0.15
        dbscan_min_samples: 0.05
        minimum_bounding_box_height: 20
      }
    }
  }
  target_class_config {
    key: "pedestrian"
    value: {
      clustering_config {
        coverage_threshold: 0.005
        dbscan_eps: 0.15
        dbscan_min_samples: 0.05
        minimum_bounding_box_height: 20
      }
    }
 }
}
        

This section defines parameters that configure the post-processor. For each class that you can train for, the postprocessing_config has a target_class_config element that defines the clustering parameters for this class. The parameters for each target class include the following:

The clustering_config element configures the clustering block for this class. Here are the parameters for this element:

In TAO Toolkit 3.21.08, DetectNet_v2 supports three methods for clustering raw detections for the network in final rendered bounding boxes.

• DBSCAN: Density Based Spatial Clustering of Application

• NMS: Non-Maximal suppression

• HYDRID: DBSCAN + NMS

Under HYBRID clustering, DetectNet_v2 post-processing first passes the raw network outputs to the DBSCAN clustering and uses the candidate boxes per cluster from DBSCAN as input to NMS. The NMS clustering generates the final rendered boxes.

Cost Function

This section describes how to configure the cost function to include the classes that you are training for. For each class you want to train, add a new entry for the target classes to the spec file. We recommend not changing the parameters within the spec file for best performance with these classes. The other parameters here should remain unchanged.

            

            
cost_function_config {
  target_classes {
    name: "car"
    class_weight: 1.0
    coverage_foreground_weight: 0.05
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 10.0
    }
  }
  target_classes {
    name: "cyclist"
    class_weight: 1.0
    coverage_foreground_weight: 0.05
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 1.0
    }
  }
  target_classes {
    name: "pedestrian"
    class_weight: 1.0
    coverage_foreground_weight: 0.05
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 10.0
    }
  }
  enable_autoweighting: True
  max_objective_weight: 0.9999
  min_objective_weight: 0.0001
}
        

Trainer

The following is a sample training_config block to configure a DetectNet_v2 trainer:

            

            
training_config {
  batch_size_per_gpu: 16
  num_epochs: 80
  learning_rate {
    soft_start_annealing_schedule {
      min_learning_rate: 5e-6
      max_learning_rate: 5e-4
      soft_start: 0.1
      annealing: 0.7
    }
  }
  regularizer {
    type: L1
    weight: 3e-9
  }
  optimizer {
    adam {
      epsilon: 1e-08
      beta1: 0.9
      beta2: 0.999
    }
  }
  cost_scaling {
    enabled: False
    initial_exponent: 20.0
    increment: 0.005
    decrement: 1.0
  }
}
        

The following table describes the parameters used to configure the trainer:

DetectNet_v2 currently supports the soft_start annealing learning rate schedule. The learning rate when plotted as a function of the training progress (0.0, 1.0) results in the following curve:

In this experiment, the soft_start was set as 0.3 and annealing as 0.7, with the minimum learning rate as 5e-6 and maximum learning rate, or base_lr, as 5e-4.

Augmentation Module

The augmentation module provides some basic pre-processing and augmentation when training. Here is a sample augmentation_config element:

            

            
# Sample augementation config for
augmentation_config {
  preprocessing {
    output_image_width: 960
    output_image_height: 544
    output_image_channel: 3
    min_bbox_width: 1.0
    min_bbox_height: 1.0
  }
  spatial_augmentation {

    hflip_probability: 0.5
    vflip_probability: 0.0
    zoom_min: 1.0
    zoom_max: 1.0
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    color_shift_stddev: 0.0
    hue_rotation_max: 25.0
    saturation_shift_max: 0.2
    contrast_scale_max: 0.1
    contrast_center: 0.5
  }
}
        

The augmentation_config contains three elements:

preprocessing: This nested field configures the input image and ground truth label pre-processing module. It sets the shape of the input tensor to the network. The ground truth labels are pre-processed to meet the dimensions of the input image tensors.

spatial_augmentation: This module supports basic spatial augmentation such as flip, zoom, and translate, which may be configured.

color_augmentation: This module configures the color space transformations, such as color shift, hue_rotation, saturation shift, and contrast adjustment.

The dataloader online augmentation pipeline applies spatial and color-space augmentation transformations in the following order:

1. The dataloader first performs the pre-processing operations on the input data (image and labels) read from the tfrecords files. Here the images and labels are cropped and scaled based on the parameters mentioned in the preprocessing config. The boundaries for generating the cropped image and labels from the original image is defined by the crop_left, crop_right, crop_top and crop_bottom parameters. This cropped data is then scaled by the scale factors defined by scale_height and scale_width. The transformation matrices for these operations are computed globally and do not change per image.

2. The net tensors generated from the pre-processing blocks are then passed through a pipeline of random augmentations in spatial and color domains. The spatial augmentations are applied to both images and label coordinates, while the color augmentations are applied only to images. To apply color augmentations, the output_image_channel parameter must be set to 3. For monochrome tensors, color augmentations are not applied. The spatial and color transformation matrices are computed per image, based on a uniform distribution along the maximum and minimum ranges defined by the spatial_augmentation and color_augmentation config parameters.

3. Once the spatial and color augmented net input tensors are generated, the output is then padded with zeros or clipped along the right and bottom edge of the image to fit the output dimensions defined in the preprocessing config.

Configuring the Evaluator

The evaluator in the detection training pipeline can be configured using the evaluation_config parameters. The following is an example evaluation_config element:

            

            
# Sample evaluation config to run evaluation in integrate mode for the given 3 class model,
# at every 10th epoch starting from the epoch 1.
evaluation_config {
  average_precision_mode: INTEGRATE
  validation_period_during_training: 10
  first_validation_epoch: 1
  minimum_detection_ground_truth_overlap {
    key: "car"
    value: 0.7
  }
  minimum_detection_ground_truth_overlap {
    key: "person"
    value: 0.5
  }
  minimum_detection_ground_truth_overlap {
    key: "bicycle"
    value: 0.5
  }
  evaluation_box_config {
    key: "car"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
  evaluation_box_config {
    key: "person"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
  evaluation_box_config {
    key: "bicycle"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
}
        

The following tables describe the parameters used to configure evaluation:

The evaluation_box_config field has these configurable inputs.

Dataloader

The dataloader defines the path to the data you want to train on and the class mapping for classes in the dataset that the network is to be trained for.

The following is an example dataset_config element:

            

            
dataset_config {
  data_sources: {
    tfrecords_path: "<path to the training tfrecords root/tfrecords train pattern>"
    image_directory_path: "<path to the training data source>"
  }
  image_extension: "jpg"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "automobile"
      value: "car"
  }
  target_class_mapping {
      key: "heavy_truck"
      value: "car"
  }
  target_class_mapping {
      key: "person"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "rider"
      value: "cyclist"
  }
  validation_fold: 0
}
        

In this example the tfrecords is assumed to be multi-fold, and the fold number to validate on is defined. However, evaluation doesn’t necessarily have to be run on a split of the training set. Many ML engineers choose to evaluate the model on a well chosen evaluation dataset that is exclusive of the training dataset. If you prefer to run evaluation on a different validation dataset as opposed to a split from the training dataset, then convert this dataset into tfrecords by using the dataset-convert tool as mentioned here and use the validation_data_source field in the dataset_config to define this dataset to the evaluator . In this case, do not forget to remove the validation_fold field from the spec. When generating the TFRecords for evaluation by using the validation_data_source field, please review the notes here.

            

            
validation_data_source: {
    tfrecords_path: " <path to tfrecords to validate on>/tfrecords validation pattern>"
    image_directory_path: " <path to validation data source>"
}
        

The parameters in dataset_config are defined as follows:

• data_sources: Captures the path to tfrecords to train on. This field contains 2 parameters:

  • tfrecords_path: Path to the individual tfrecords files. This path follows the UNIX style pathname pattern extension, so a common pathname pattern that captures all the tfrecords files in that directory can be used.

  • image_directory_path: Path to the training data root from which the tfrecords was generated.

• image_extension: Extension of the images to be used.

• target_class_mapping: This parameter maps the class names in the tfrecords to the target class to be trained in the network. An element is defined for every source class to target class mapping. This field was included with the intention of grouping similar class objects under one umbrella. For example: car, van, heavy_truck etc may be grouped under automobile. The “key” field is the value of the class name in the tfrecords file and the “value” field corresponds to the value that the network is expected to learn.

• validation_fold: In case of an n fold tfrecords, you define the index of the fold to use for validation. For sequencewise validation choose the validation fold in the range [0, N-1]. For random split partitioning, force the validation fold index to 0 as the tfrecord is just 2-fold.

Specification File for Inference

This spec file configures the infer tool of detectnet to generate valid bbox predictions. The inference tool consists of 2 blocks, namely the inferencer and the bbox handler. The inferencer instantiates the model object and preprocessing pipe. The bbox handler handles the post processing, rendering of bounding boxes and the serialization to KITTI format output labels.

Inferencer

The inferencer instantiates a model object that generates the raw predictions from the trained model. The model may be defined to run inference in the TAO Toolkit backend or the TensorRT backend.

A sample inferencer_config element for the inferencer spec is defined here:

            

            
inferencer_config{
  # defining target class names for the experiment.
  # Note: This must be mentioned in order of the networks classes.
  target_classes: "car"
  target_classes: "cyclist"
  target_classes: "pedestrian"
  # Inference dimensions.
  image_width: 1248
  image_height: 384
  # Must match what the model was trained for.
  image_channels: 3
  batch_size: 16
  gpu_index: 0
  # model handler config
  tensorrt_config{
    parser:  ETLT
    etlt_model: "/path/to/model.etlt"
    backend_data_type: INT8
    save_engine: true
    trt_engine: "/path/to/trt/engine/file"
    calibrator_config{
        calibration_cache: "/path/to/calibration/cache"
        n_batches: 10
        batch_size: 16
    }
  }
}
        

The inferencer_config parameters are explained in the table below.

As mentioned earlier, the infer tool is capable of running inference using the native TAO Toolkit backend and the TensorRT backend. They can be configured by using the tensorrt_config proto element or the tlt_config proto element respectively. You may use only one of the two in a single spec file. The definitions of the two model objects are:

TLT_Config

Bbox Handler

The bbox handler takes care of the post processing the raw outputs from the inferencer. It performs the following steps:

1. Thresholding the raw outputs to defines grid cells where the detections may be present per class.

2. Reconstructing the image space coordinates from the raw coordinates of the inferencer.

3. Clustering the raw thresholded predictions.

4. Filtering the clustered predictions per class.

5. Rendering the final bounding boxes on the image in its input dimensions and serializing them to KITTI format metadata.

A sample bbox_handler_config element is defined below.

            

            
bbox_handler_config{
  kitti_dump: true
  disable_overlay: false
  overlay_linewidth: 2
  classwise_bbox_handler_config{
    key:"car"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "car"
      bbox_color{
        R: 0
        G: 255
        B: 0
      }
      clustering_config{
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
  classwise_bbox_handler_config{
    key:"default"
    value: {
      confidence_model: "aggregate_cov"
      bbox_color{
        R: 255
        G: 0
        B: 0
      }
      clustering_config{
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
}
        

The parameters to configure the bbox handler are defined below.

The classwise_bbox_handler_config is a Proto object containing several parameters to configure the clustering algorithm as well as the bbox renderer.

After creating TFRecords ingestible by the TAO training (as outlined in Preprocessing the Dataset) and setting up a spec file, you are now ready to start training an object detection network.

The following outlines the DetectNet_v2 training command:

            

            
tao detectnet_v2 train [-h] -k <key>
                            -r <result directory>
                            -e <spec_file>
                            [-n <name_string_for_the_model>]
                            [--gpus <num GPUs>]
                            [--gpu_index <comma separate gpu indices>]
                            [--use_amp]
                            [--log_file <log_file>]
        

Required Arguments

• -r, --results_dir: The path to a folder where experiment outputs should be written.

• -k, –key: A user-specific encoding key to save or load a .tlt model.

• -e, --experiment_spec_file: The path to the spec file. The path may be absolute or relative to the working directory. By default, the spec from spec_loader.py is used.

Optional Arguments

• -n, --model_name: The name of the final step model saved. If not provided, defaults to the model.

• --gpus: The number of GPUs to use and processes to launch for training. The default value is 1.

• --gpu_index: The indices of the GPUs to use for training. The GPUs are referenced as per the indices mentioned in the ./deviceQuery CUDA samples.

• --use_amp: When defined, this flag enables Automatic Mixed Precision mode.

• --log_file: The path to the log file. Defaults to stdout.

• -h, --help: Show this help message and exit.

Input Requirement

• Input size: C * W * H (where C = 1 or 3, W > =480, H >=272 and W, H are multiples of 16)

• Image format: JPG, JPEG, PNG

• Label format: KITTI detection

Sample Usage

Here is an example of a command for training with two GPUs:

            

            
tao detectnet_v2 train -e <path_to_spec_file>
                       -r <path_to_experiment_output>
                       -k <key_to_load_the_model>
                       -n <name_string_for_the_model>
                       --gpus 2
        

Execute evaluate on a DetectNet_v2 model.

            

            
tao detectnet_v2 evaluate [-h] -e <experiment_spec
                               -m <model_file>
                               -k <key>
                               [--use_training_set]
                               [--gpu_index]
        

Required Arguments

• -e, --experiment_spec_file: The experiment spec file to set up the evaluation experiment. This should be the same as training spec file.

• -m, --model: The path to the model file to use for evaluation. This could be a .tlt model file or a tensorrt engine generated using the export tool.

• -k, -–key: The encryption key to decrypt the model. This argument is only required with a .tlt model file.

Optional Arguments

• -h, --help: Show this help message and exit.

• -f, --framework: The framework to use when running evaluation (choices: “tlt”, “tensorrt”). By default the framework is set to TensorRT.

• --use_training_set: Set this flag to run evaluation on the training dataset.

• --gpu_index: The index of the GPU to run evaluation on.

If you have followed the example in Training a Detection Model, you may now evaluate the model using the following command:

            

            
tao detectnet_v2 evaluate -e <path to training spec file>
                          -m <path to the model>
                          -k <key to load the model>
        

Use these steps to evaluate on a test set with ground truth labeled:

1. Create tfrecords for this training set by following the steps listed in the data input section.

2. Update the dataloader configuration part of the training spec file to include the newly generated tfrecords. For more information on the dataset config, refer to Creating an Experiment Spec File. You may create the tfrecords with any partition mode (sequence/random). The evaluate tool iterates through all the folds in the tfrecords patterns mentioned in the validation_data_source.

            

            
dataset_config {
  data_sources: {
    tfrecords_path: "<path to training tfrecords root>/<tfrecords_name*>"
    image_directory_path: "<path to training data root>"
  }
  image_extension: "jpg"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "automobile"
      value: "car"
  }
  ..
  ..
  ..
  target_class_mapping {
      key: "person"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "rider"
      value: "cyclist"
  }
  validation_data_source: {
    tfrecords_path: "<path to testing tfrecords root>/<tfrecords_name*>"
    image_directory_path: "<path to testing data root>"
  }
}
        

The rest of the experiment spec file should remain the same as the training spec file.

The infer task for detectnet_v2 may be used to visualize bboxes and/or generate frame-by-frame KITTI format labels on a single image or directory of images. An example of the command for this task is shown below:

            

            
tao detectnet_v2 inference [-h] -e </path/to/inference/spec/file>
                                -i </path/to/inference/input>
                                -o </path/to/inference/output>
                                -k <model key>
        

Required Parameters

• -e, --inference_spec: The path to an inference spec file.

• -i, --inference_input: The directory of input images or a single image for inference.

• -o, --inference_output: The directory to the output images and labels. The annotated images are in inference_output/images_annotated and labels are in inference_output/labels.

• -k, --enc_key: The key to load the model.

The tool automatically generates bbox rendered images in output_path/images_annotated. To get the bbox labels in KITTI format, configure the bbox_handler_config spec file using the kitti_dump parameter as mentioned here. This will generate the output in output_path/labels.

Pruning removes parameters from the model to reduce the model size without compromising the integrity of the model itself using the prune command.

The prune task includes these parameters:

            

            
tao detectnet_v2 prune [-h] -pm <pretrained_model>
                            -o <output_file>
                            -k <key>
                            [-n <normalizer>]
                            [-eq <equalization_criterion>]
                            [-pg <pruning_granularity>]
                            [-pth <pruning threshold>]
                            [-nf <min_num_filters>]
                            [-el [<excluded_list>]
        

Required Arguments

• -pm, --pretrained_model: The path to the pretrained model.

• -o, --output_file: The path to the output checkpoints.

• -k, --key: The key to load a .tlt model.

Optional Arguments

• -h, --help: Show this help message and exit.

• -n, –normalizer: Specify max to normalize by dividing each norm by the maximum norm within a layer; specify L2 to normalize by dividing by the L2 norm of the vector comprising all kernel norms. The default value is max.

• -eq, --equalization_criterion: Criteria to equalize the stats of inputs to an element-wise op layer or depth-wise convolutional layer. This parameter is useful for resnets and mobilenets. The options are arithmetic_mean, geometric_mean, union, and intersection (default: union).

• -pg, -pruning_granularity: The number of filters to remove at a time (default:8)

• -pth: The threshold to compare the normalized norm against (default:0.1)

• -nf, --min_num_filters: The minimum number of filters to keep per layer (default:16)

• -el, --excluded_layers: A list of excluded_layers (e.g. -i item1 item2) (default: [])

After pruning, the model needs to be retrained. See Re-training the Pruned Model for more details.

Using the Prune Command

Here’s an example of using the prune task:

            

            
tao detectnet_v2 prune -m /workspace/output/weights/resnet_003.tlt
                       -o /workspace/output/weights/resnet_003_pruned.tlt
                       -eq union
                       -pth 0.7 -k $KEY
        

Once the model has been pruned, there might be a slight decrease in accuracy because some previously useful weights may have been removed. To regain the accuracy, we recommend that you retrain this pruned model over the same dataset using the train task, as documented in the Training the model section, with an updated spec file that points to the newly pruned model as the pretrained model file.

You should turn off the regularizer in the training_config for detectnet to recover the accuracy when retraining a pruned model. You may do this by setting the regularizer type to NO_REG as mentioned here. All other parameters may be retained in the spec file from the previous training.

To load the pretrained model, set the load_graph flag under model_config to true.

The DetectNet_V2 model application in the TAO Toolkit includes an export sub-task to export and prepare a trained DetectNet_v2 model for Deploying to DeepStream. The export sub-task optionally generates the calibration cache for TensorRT INT8 engine calibration.

Exporting the model decouples the training process from deployment and allows conversion to TensorRT engines outside the TAO environment. TensorRT engines are specific to each hardware configuration and should be generated for each unique inference environment. This may be interchangeably referred to as a .trt or .engine file. The same exported TAO model may be used universally across training and deployment hardware. This is referred to as the .etlt file, or encrypted TAO file. During model export, the TAO model is encrypted with a private key, which is required when you deploy this model for inference.

INT8 Mode Overview

TensorRT engines can be generated in INT8 mode to run with lower precision, and thus improve performance. This process requires a cache file that contains scale factors for the tensors to help combat quantization errors, which may arise due to low-precision arithmetic. The calibration cache is generated using a calibration tensorfile when export is run with the --data_type flag set to int8. Pre-generating the calibration information and caching it removes the need for calibrating the model on the inference machine. Moving the calibration cache is usually much more convenient than moving the calibration tensorfile since it is a much smaller file and can be moved with the exported model. Using the calibration cache also speeds up engine creation as building the cache can take several minutes to generate depending on the size of the Tensorfile and the model itself.

The export tool can generate an INT8 calibration cache by ingesting training data using one of these options:

• Option 1: Providing a calibration tensorfile generated using the calibration_tensorfile task defined in DetectNet_v2. This command uses the data generators in the training pipeline to produce a drop of preprocessed batches of input images from the training dataset. Using this gives users the opportunity to maintain a record of the exact batches of the training data used to generate the calibration scale factors in the calibration cache file. However, this is a two-step process for generating an int8 cache file.

• Option 2: Pointing the tool to a directory of images that you want to use to calibrate the model. For this option, you will need to create a sub-sampled directory of random images that best represent your training dataset.

• Option 3: Using the training data loader directly to load the training images for INT8 calibration. This option is now the recommended approach as it helps to generate multiple random samples. This also ensures two important aspects of the data during calibration:

  • Data pre-processing in the INT8 calibration step is the same as in the training process.

  • The data batches are sampled randomly across the entire training dataset, thereby improving the accuracy of the int8 model.

  • Calibration occurs as a one-step process with the data batches being generated on the fly.

NVIDIA plans to eventually deprecate Option 1 and only support Options 2 and 3.

FP16/FP32 Model

The calibration.bin is only required if you need to run inference at INT8 precision. For FP16/FP32 based inference, the export step is much simpler. All that is required is to provide a model from the train step to export to convert it into an encrypted TAO model.

Generating an INT8 tensorfile Using the calibration_tensorfile Command

The INT8 tensorfile is a binary file that contains the preprocessed training samples, which may be used to calibrate the model. In this release, TAO Toolkit only supports calibration tensorfile generation for SSD, DSSD, DetectNet_v2, and classification models.

The sample usage for the calibration_tensorfile command to generate a calibration tensorfile is defined below:

            

            
tao detectnet_v2 calibration_tensorfile [-h] -e <path to training experiment spec file>
                                             -o <path to output tensorfile>
                                             -m <maximum number of batches to serialize>
                                            [--use_validation_set]
        

Required Arguments

• -e, --experiment_spec_file: The path to the experiment spec file (only required for SSD and FasterRCNN).

• -o, --output_path: The path to the output tensorfile that will be created.

• -m, --max_batches: The number of batches of input data to be serialized.

Optional Argument

• --use_validation_set: A flag specifying whether to use the validation dataset instead of the training set.

The following is a sample command to invoke the calibration_tensorfile command for a classification model:

            

            
tao detectnet_v2 calibration_tensorfile
                  -e $SPECS_DIR/classification_retrain_spec.cfg
                  -m 10
                  -o $USER_EXPERIMENT_DIR/export/calibration.tensor
        

Exporting the DetectNet_v2 Model

The following are command line arguments of the export command:

            

            
tao detectnet_v2 export [-h] -m <path to the .tlt model file generated by tao train>
                             -k <key>
                             [-o <path to output file>]
                             [--cal_data_file <path to tensor file>]
                             [--cal_image_dir <path to the directory images to calibrate the model]
                             [--cal_cache_file <path to output calibration file>]
                             [--data_type <Data type for the TensorRT backend during export>]
                             [--batches <Number of batches to calibrate over>]
                             [--max_batch_size <maximum trt batch size>]
                             [--max_workspace_size <maximum workspace size]
                             [--batch_size <batch size to TensorRT engine>]
                             [--experiment_spec <path to experiment spec file>]
                             [--engine_file <path to the TensorRT engine file>]
                             [--verbose Verbosity of the logger]
                             [--force_ptq Flag to force PTQ]
                             [--gen_ds_config Generate DeepStream config]
        

Required Arguments

• -m, --model: The path to the .tlt model file to be exported using export.

• -k, --key: The key used to save the .tlt model file.

• -e, --experiment_spec: The path to the spec file. This argument is required for faster_rcnn, ssd, dssd, yolo, and retinanet.

Optional Arguments

• -o, --output_file: The path to save the exported model to. The default path is ./<input_file>.etlt.

• --data_type: The desired engine data type. The options are fp32, fp16, and int8. A calibration cache will be generated in int8 mode. The default value is fp32. If using int8 mode, the following INT8 arguments are required.

• -s, --strict_type_constraints: A Boolean flag to indicate whether or not to apply the TensorRT strict_type_constraints when building the TensorRT engine. Note this is only for applying the strict type of int8 mode.

• --gen_ds_config: A Boolean flag indicating whether to generate the template DeepStream related configuration (“nvinfer_config.txt”) as well as a label file (“labels.txt”) in the same directory as the output_file. Note that the config file is NOT a complete configuration file and requires the user to update the sample config files in DeepStream with the parameters generated.

INT8 Export Mode Required Arguments

• --cal_data_file: The tensorfile generated from calibration_tensorfile for calibrating the engine. This can also be an output file if used with --cal_image_dir.

• --cal_image_dir: The directory of images to use for calibration.

INT8 Export Optional Arguments

• --cal_cache_file: The path to save the calibration cache file to. The default value is ./cal.bin.

• --batches: The number of batches to use for calibration and inference testing. The default value is 10.

• --batch_size: The batch size to use for calibration. The default value is 8.

• --max_batch_size: The maximum batch size of the TensorRT engine. The default value is 16.

• --max_workspace_size: The maximum workspace size of the TensorRT engine. The default value is 1073741824 = 1<<30.

• --experiment_spec: The experiment_spec for training/inference/evaluation. This is used to generate the graphsurgeon config script for FasterRCNN from the experiment_spec (which is only useful for FasterRCNN). Use this argument when DetectNet_v2 and FasterRCNN also set up the dataloader-based calibrator to leverage the training dataloader to calibrate the model.

• --engine_file: The path to the serialized TensorRT engine file. Note that this file is hardware specific and cannot be generalized across GPUs. Use this argument to quickly test your model accuracy using TensorRT on the host. As the TensorRT engine file is hardware specific, you cannot use this engine file for deployment unless the deployment GPU is identical to the training GPU.

• --force_ptq: A Boolean flag to force post-training quantization on the exported .etlt model.

Sample usage for the export sub-task

The following is a sample command to export a DetectNet_v2 model in INT8 mode. This command shows option 1: using the --cal_data_file option with the calibration.tensor generated using the calibration_tensorfile sub-task.

            

            
tao detectnet_v2 export
     -e $USER_EXPERIMENT_DIR/experiment_dir_retrain/experiment_spec.txt
     -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt
     -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt
     -k $KEY
     --cal_data_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor
     --data_type int8
     --batches 10
     --cal_cache_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.bin
     --engine_file $USER_EXPERIMENT_DIR/experiment_dir_final/resnet_18.engine
        

The following is an example log of a successful export:

            

            
Using TensorFlow backend.
2018-11-02 18:59:43,347 [INFO] iva.common.tlt-export: Loading model from resnet10_kitti_multiclass_v1.tlt
2018-11-02 18:59:47,572 [INFO] tensorflow: Restoring parameters from /tmp/tmp8crUBp.ckpt
INFO:tensorflow:Froze 82 variables.
2018-11-02 18:59:47,701 [INFO] tensorflow: Froze 82 variables.
Converted 82 variables to const ops.
2018-11-02 18:59:48,123 [INFO] iva.common.tlt-export: Converted model was saved into resnet10_kitti_multiclass_v1.etlt
2018-11-02 18:59:48,123 [INFO] iva.common.tlt-export: Input node: input_1
2018-11-02 18:59:48,124 [INFO] iva.common.tlt-export: Output node(s): ['output_bbox/BiasAdd', 'output_cov/Sigmoid']
        

The following is a sample command using the --cal_image_dir option for a DetectNet_v2 model using option 2.

            

            
tao detectnet_v2 export
    -m $USER_EXPERIMENT_DIR/detectnet_v2/model.tlt
    -o $USER_EXPERIMENT_DIR/detectnet_v2/model.int8.etlt
    -e $SPECS_DIR/detectnet_v2_kitti_retrain_spec.txt
    --key $KEY
    --cal_image_dir  $USER_EXPERIMENT_DIR/data/KITTI/val/image_2
    --data_type int8
    --batch_size 8
    --batches 10
    --cal_data_file $USER_EXPERIMENT_DIR/data/detectnet_v2/cal.tensorfile
    --cal_cache_file $USER_EXPERIMENT_DIR/data/detectnet_v2/cal.bin
    --engine_file $USER_EXPERIMENT_DIR/data/detectnet_v2/detection.trt
        

Generating a Template DeepStream Config File

TAO Toolkit supports serializing a template config file for the nvinfer element of deepstream to consume this model. This config file contains the network specific pre-processing parameters and network graph parameters for parsing the etlt model file. It also generates a label file that contains the names of the classes that the model was trained for in the order in which the outputs are generated. To generate the deepstream config, simply run the export command using the --gen_ds_config option.

The following example shows how to generate the DeepStream config:

            

            
tao detectnet_v2 export
      -m $USER_EXPERIMENT_DIR/detectnet_v2/model.tlt
      -o $USER_EXPERIMENT_DIR/detectnet_v2/model.int8.etlt
      -e $SPECS_DIR/detectnet_v2_kitti_retrain_spec.txt
      --key $KEY
      --cal_image_dir  $USER_EXPERIMENT_DIR/data/KITTI/val/image_2
      --data_type int8
      --batch_size 8
      --batches 10
      --cal_data_file $USER_EXPERIMENT_DIR/data/detectnet_v2/cal.tensorfile
      --cal_cache_file $USER_EXPERIMENT_DIR/data/detectnet_v2/cal.bin
      --engine_file $USER_EXPERIMENT_DIR/data/detectnet_v2/detection.trt
      --gen_ds_config
        

The template DeepStream config is generated in the same directory as the output model file as nvinfer_config.txt, while the labels are serialized in labels.txt file. Sample output of the nvinfer_config.txt and labels.txt are as follows:

• Sample nvinfer_config.txt

  Copy

  Copied!

              
  
              
  net-scale-factor=0.00392156862745098
  offsets=0;0;0
  infer-dims=3;544;960
  tlt-model-key=tlt_encode
  network-type=0
  num-detected-classes=3
  uff-input-order=0
  output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
  uff-input-blob-name=input_1
  model-color-format=0
          

• Sample labels.txt

  Copy

  Copied!

              
  
              
  person
  bag
  face
          

This section elaborates how to deploy a trained DetectNet_v2 model to the DeepStream SDK for inference, and the two options to consume an exported .etlt model file. For information about the DeepStream SDK itself, please refer to this section.

Generating an Engine Using tao-converter

The tao-converter tool is provided with the TAO Toolkit to facilitate the deployment of TAO trained models on TensorRT and/or Deepstream. This section elaborates on how to generate a TensorRT engine using tao-converter.

For deployment platforms with an x86-based CPU and discrete GPUs, the tao-converter is distributed within the TAO docker. Therefore, we suggest using the docker to generate the engine. However, this requires that the user adhere to the same minor version of TensorRT as distributed with the docker. The TAO docker includes TensorRT version 8.0.

Instructions for x86

For an x86 platform with discrete GPUs, the default TAO package includes the tao-converter built for TensorRT 8.0 with CUDA 11.3 and CUDNN 8.2. However, for any other version of CUDA and TensorRT, please refer to the overview section for download. Once the tao-converter is downloaded, follow the instructions below to generate a TensorRT engine.

1. Unzip the zip file on the target machine.

2. Install the OpenSSL package using the command:

   Copy

   Copied!

               
   
               
   sudo apt-get install libssl-dev
           

3. Export the following environment variables:

            

            
$ export TRT_LIB_PATH=”/usr/lib/x86_64-linux-gnu”
$ export TRT_INC_PATH=”/usr/include/x86_64-linux-gnu”
        

4. Run the tao-converter using the sample command below and generate the engine.

Instructions for Jetson

For the Jetson platform, the tao-converter is available to download in the NVIDIA developer zone. You may choose the version you wish to download as listed in the overview section. Once the tao-converter is downloaded, please follow the instructions below to generate a TensorRT engine.

1. Unzip the zip file on the target machine.

2. Install the OpenSSL package using the command:

   Copy

   Copied!

               
   
               
   sudo apt-get install libssl-dev
           

3. Export the following environment variables:

            

            
$ export TRT_LIB_PATH=”/usr/lib/aarch64-linux-gnu”
$ export TRT_INC_PATH=”/usr/include/aarch64-linux-gnu”
        

1. For Jetson devices, TensorRT comes pre-installed with Jetpack. If you are using older JetPack, upgrade to JetPack 4.5 or 4.6.

2. Run the tao-converter using the sample command below and generate the engine.

Using the tao-converter

            

            
tao-converter [-h] -k <encryption_key>
                   -d <input_dimensions>
                   -o <comma separated output nodes>
                   [-c <path to calibration cache file>]
                   [-e <path to output engine>]
                   [-b <calibration batch size>]
                   [-m <maximum batch size of the TRT engine>]
                   [-t <engine datatype>]
                   [-w <maximum workspace size of the TRT Engine>]
                   [-i <input dimension ordering>]
                   [-p <optimization_profiles>]
                   [-s]
                   [-u <DLA_core>]
                   input_file
        

Required Arguments

• input_file: The path to the .etlt model exported using export.

• -k: The key used to encode the .tlt model when doing the training.

• -d: A comma-separated list of input dimensions that should match the dimensions used for export. Unlike export, this cannot be inferred from calibration data.

• -o: A comma-separated list of output blob names that should match the output configuration used for export. For DetectNet_v2, set this argument to output_cov/Sigmoid,output_bbox/BiasAdd.

Optional Arguments

• -e: The path to save the engine to. The default path is default: ./saved.engine.

• -t: The desired engine data type. This option generates a calibration cache if in INT8 mode. The default value is fp32. The options are fp32, fp16, int8.

• -w: The maximum workspace size for the TensorRT engine. The default value is 1073741824(1<<30).

• -i: The input dimension ordering. The default value is nchw. The options are nchw, nhwc, nc. For detectnet_v2, we can omit this argument.

• -p: The optimization profiles for .etlt models with dynamic shape. This argument takes a comma-separated list of optimization profile shapes in the format <input_name>,<min_shape>,<opt_shape>,<max_shape>, where each shape has the format <n>x<c>x<h>x<w>. This can be specified multiple times if there are multiple input tensors for the model. This argument is only useful for new models introduced since TLT 3.0.

• -s: A Boolean value specifying whether to apply TensorRT strict type constraints when building the TensorRT engine.

• -u: Specifies the DLA core index when building the TensorRT engine on Jetson devices.

INT8 Mode Arguments

• -c: The path to the calibration cache file for INT8 mode. The default path is ./cal.bin.

• -b: The batch size used during the export step for INT8 calibration cache generation (default: 8).

• -m: The maximum batch size for the TensorRT engine. The default value is 16. If you encounter out-of-memory issues, decrease the batch size accordingly.

Sample Output Log

The following is a sample log for exporting a DetectNet_v2 model:

            

            
tao-converter -d 3,544,960
              -k nvidia_tlt
              -o output_cov/Sigmoid,output_bbox/BiasAdd
              /workspace/tao-experiments/detectnet_v2/resnet18_pruned.etlt
..
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[INFO] Detected 1 inputs and 2 output network tensors.
        

Once the model and/or TensorRT engine file has been generated, two extra files are required:

1. Label file

2. DS configuration file

Label File

The label file is a text file containing the names of the classes that the DetectNet_v2 model is trained to detect. The order in which the classes are listed here must match the order in which the model predicts the output. The export subtask in DetectNet_v2 generates this file when run with the --gen_ds_config flag enabled.

DeepStream Configuration File

The detection model is typically used as a primary inference engine. It can also be used as a secondary inference engine. To run this model in the sample deepstream-app, you must modify the existing config_infer_primary.txt file to point to this model.

Option 1: Integrate the model (.etlt) directly in the DeepStream app.

For this option, you will need to add the following parameters in the configuration file. The int8-calib-file is only required for INT8 precision.

            

            
tlt-encoded-model=<TLT exported .etlt>
tlt-model-key=<Model export key>
int8-calib-file=<Calibration cache file>
        

The tlt-encoded-model parameter points to the exported model (.etlt) from TLT. The tlt-model-key is the encryption key used during model export.

Option 2: Integrate the TensorRT engine file with DeepStream app.

Step 1: Generate TensorRT engine using tao-converter. Detailed instructions are provided in the Generating an engine using tao-converter section above.

Step 2: Once the engine file is generated successfully, modify the following parameters to use this engine with DeepStream.

            

            
model-engine-file=<PATH to generated TensorRT engine>
        

All other parameters are common between the two approaches. Update the label-file-path parameter in the config file with the path to the labels.txt that was generated at export.

            

            
labelfile-path=<Classification labels>
        

For all options, see the configuration file below. To learn more about all the parameters, refer to the DeepStream Development Guide under the GsT-nvinfer section.

            

            
[property]
gpu-id=0
# preprocessing parameters.
net-scale-factor=0.0039215697906911373
model-color-format=0

# model paths.
int8-calib-file=<Path to optional INT8 calibration cache>
labelfile-path=<Path to detectNet_v2_labels.txt>
tlt-encoded-model=<Path to DetectNet_v2 TLT model>
tlt-model-key=<Key to decrypt the model>
infer-dims=c;h;w # where c = number of channels, h = height of the model input, w = width of model input
uff-input-order=0 # 0 implies that the input blob is in chw order
uff-input-blob-name=input_1
batch-size=4
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=3
interval=0
gie-unique-id=1
is-classifier=0
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
#enable_dbscan=0

[class-attrs-all]
threshold=0.2
group-threshold=1
## Set eps=0.7 and minBoxes for enable-dbscan=1
eps=0.2
#minBoxes=3
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0