# FasterRCNN

FasterRCNN is a public object detection model that is supported by NVIDIA TAO Toolkit. FasterRCNN in TAO Toolkit supports below tasks:

• dataset_convert

• train

• evaluate

• inference

• prune

• export

These tasks can be invoked from the TAO Toolkit Launcher using the following convention on the command-line:

Copy
Copied!

tao faster_rcnn <sub_task> <args_per_subtask>


where, args_per_subtask are the command-line arguments required for a given subtask. Each subtask is explained in detail in the following sections.

## Preparing the Input Data Structure

The dataset structure of FasterRCNN is identical to that of DetectNet_v2. The only difference is the command line used to generate the TFRecords from KITTI text labels. To generate TFRecords for FasterRCNN training, use this command:

Copy
Copied!

tao faster_rcnn dataset_convert [-h] -d <dataset_spec>
-o <output_tfrecords_file>
[--gpu_index <gpu_index>]


### Required Arguments

• -d, --dataset_spec: path to the dataset spec file.

• -o, --output_filename: path to the output TFRecords file.

### Optional Arguments

• --gpu_index: The GPU index to run this command on. We can specify the GPU index used to run this command if the machine has multiple GPUs installed. Note that this command can only run on a single GPU.

## Creating a Configuration File

The experiments specification (spec file for short) defines all the necessary parameters required to in the entire workflow of a FasterRCNN model, from training to export. Below is a sample of the FasterRCNN spec file. The format of the spec file is a protobuf text (prototxt) message and each of its fields can be either a basic data type or a nested proto message. The top level structure of the spec file is summarized in the table below. From the table, we can see the spec file has 9 components: random_seed, verbose, enc_key, dataset_config, augmentation_config, model_config, training_config, inference_config and evaluation_config.

Here’s a sample of the FasterRCNN spec file:

Copy
Copied!

random_seed: 42
enc_key: 'nvidia_tlt'
verbose: True
model_config {
input_image_config {
image_type: RGB
image_channel_order: 'bgr'
size_height_width {
height: 384
width: 1248
}
image_channel_mean {
key: 'b'
value: 103.939
}
image_channel_mean {
key: 'g'
value: 116.779
}
image_channel_mean {
key: 'r'
value: 123.68
}
image_scaling_factor: 1.0
max_objects_num_per_image: 100
}
arch: "resnet:18"
anchor_box_config {
scale: 64.0
scale: 128.0
scale: 256.0
ratio: 1.0
ratio: 0.5
ratio: 2.0
}
freeze_bn: True
freeze_blocks: 0
freeze_blocks: 1
roi_mini_batch: 256
rpn_stride: 16
use_bias: False
roi_pooling_config {
pool_size: 7
pool_size_2x: False
}
all_projections: True
use_pooling:False
}
dataset_config {
data_sources: {
tfrecords_path: "/workspace/tao-experiments/tfrecords/kitti_trainval/kitti_trainval*"
image_directory_path: "/workspace/tao-experiments/data/training"
}
image_extension: 'png'
target_class_mapping {
key: 'car'
value: 'car'
}
target_class_mapping {
key: 'van'
value: 'car'
}
target_class_mapping {
key: 'pedestrian'
value: 'person'
}
target_class_mapping {
key: 'person_sitting'
value: 'person'
}
target_class_mapping {
key: 'cyclist'
value: 'cyclist'
}
validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 1248
output_image_height: 384
output_image_channel: 3
min_bbox_width: 1.0
min_bbox_height: 1.0
}
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 0
translate_max_y: 0
}
color_augmentation {
hue_rotation_max: 0.0
saturation_shift_max: 0.0
contrast_scale_max: 0.0
contrast_center: 0.5
}
}
training_config {
enable_augmentation: True
enable_qat: False
batch_size_per_gpu: 8
num_epochs: 12
retrain_pruned_model: "/workspace/tao-experiments/data/faster_rcnn/model_1_pruned.tlt"
output_model: "/workspace/tao-experiments/data/faster_rcnn/frcnn_kitti_resnet18_retrain.tlt"
rpn_min_overlap: 0.3
rpn_max_overlap: 0.7
classifier_min_overlap: 0.0
classifier_max_overlap: 0.5
gt_as_roi: False
std_scaling: 1.0
classifier_regr_std {
key: 'x'
value: 10.0
}
classifier_regr_std {
key: 'y'
value: 10.0
}
classifier_regr_std {
key: 'w'
value: 5.0
}
classifier_regr_std {
key: 'h'
value: 5.0
}

rpn_mini_batch: 256
rpn_pre_nms_top_N: 12000
rpn_nms_max_boxes: 2000
rpn_nms_overlap_threshold: 0.7

regularizer {
type: L2
weight: 1e-4
}

optimizer {
sgd {
lr: 0.02
momentum: 0.9
decay: 0.0
nesterov: False
}
}

learning_rate {
soft_start {
base_lr: 0.02
start_lr: 0.002
soft_start: 0.1
annealing_points: 0.8
annealing_points: 0.9
annealing_divider: 10.0
}
}

lambda_rpn_regr: 1.0
lambda_rpn_class: 1.0
lambda_cls_regr: 1.0
lambda_cls_class: 1.0
}
inference_config {
images_dir: '/workspace/tao-experiments/data/testing/image_2'
model: '/workspace/tao-experiments/data/faster_rcnn/frcnn_kitti_resnet18_retrain.epoch12.tlt'
batch_size: 1
detection_image_output_dir: '/workspace/tao-experiments/data/faster_rcnn/inference_results_imgs_retrain'
labels_dump_dir: '/workspace/tao-experiments/data/faster_rcnn/inference_dump_labels_retrain'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
object_confidence_thres: 0.0001
bbox_visualize_threshold: 0.6
classifier_nms_max_boxes: 100
classifier_nms_overlap_threshold: 0.3
}
evaluation_config {
model: '/workspace/tao-experiments/data/faster_rcnn/frcnn_kitti_resnet18_retrain.epoch12.tlt'
batch_size: 1
validation_period_during_training: 1
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
classifier_nms_max_boxes: 100
classifier_nms_overlap_threshold: 0.3
object_confidence_thres: 0.0001
use_voc07_11point_metric:False
gt_matching_iou_threshold: 0.5
}


 Parameter Description Data Type and Constraints Default/Suggested Value random_seed The random seed for the experiment. Unsigned int 42 enc_key The encoding and decoding key for the TAO models, can be overridden by the command line arguments of tao faster_rcnn train, tao faster_rcnn evaluate, and tao faster_rcnn inference. Str, should not be empty – verbose Controls the logging level during the experiments. Will print more logs if True. Boolean(True or False) False dataset_config The configurations of the dataset, this is the same as dataset_config in DetectNet_v2. proto message – augmentation_config The configuration of the data augmentation, same as DetectNet_v2. proto message – model_config The configuration of the model architecture. proto message – training_config The configurations for doing training with the model. proto message – inference_config The configuration for doing inference with the model. proto message – evaluation_config The configuration for doing evaluation with the model. proto message –

### Dataset

The dataset_config defines the dataset of a FasterRCNN experiments (including training dataset and validation dataset). The definition of FasterRCNN dataset is identical to that of DetectNet_v2. Check the DetectNet_v2 dataset_config documentation for the details of this parameter.

### Data augmentation

The augmentation_config defines the data augmentation during the training of a FasterRCNN model. The definition of FasterRCNN data augmentation is identical to that of DetectNet_v2. Check the DetectNet_v2 augmentation_config documentation for the details of this parameter.

### Model architecture

The model_config defines the FasterRCNN model architecture. In this parameter, we can choose the backbone of the FasterRCNN model, enabling BatchNormalization layers or not, whether or not to freeze the BatchNormalization layers during training, and whether or not to freeze some blocks in the model during training. With this parameter, we can define a specialized FasterRCNN model architecture from the general FasterRCNN application, according to the use cases. Detailed description of this parameter is summarized in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value input_image_config Defines the input image format, including the image channel number, channel order, width and height, and the preprocessings (subtract per-channel mean and divided by a scaling factor) for it before feeding input the model. See below for details. proto message – arch The feature extractor (backbone) for the FasterRCNN model. FasterRCNN supports 14 backbones. str type. The architecture can be ResNet, VGG , GoogLeNet, MobileNet or DarkNet. For each specific architecture, it can have different layers or versions. Details listed below. ResNet series: resnet:10, resnet:18, resnet:34, resnet:50, resnet:101 VGG series: vgg:16, vgg:19 GoogLeNet: googlenet MobileNet series: mobilenet_v1, mobilenet_v2 DarkNet: darknet:19, darknet:53 EfficientNet: efficientnet:b0, efficientnet:b1 Here a notational convention can be used, i.e., for models that can have different numbers of layers, use a colon followed by the layer number as the suffix of the model name. E.g., resnet: – anchor_box_config Configurations of the anchor boxes. proto message. – roi_mini_batch The batch size of ROIs for training the RCNN. int. 256 rpn_stride Cummulative stride from model input to RPN. This value is fixed (16) in current implementation. int. 16 freeze_bn A flag to freeze all the BatchNormalization layers in the model. Freezing a BatchNormalization layer means freezing its moving mean and moving variance while its gamma and beta parameters are still trainable. This is usually used in FasterRCNN training with a small batch size so the moving means and moving variances are initialized from the pretrained model and fixed during training. Boolean. False dropout_rate The dropout rate is applicable to the Dropout layers in the model(if there are any). Currently only VGG 16/19 and EfficientNet has Dropout layers. float. In the interval (0, 1). 0.0 drop_connect_rate The drop_connect rate for EfficientNet. float. In the interval (0, 1). 0.0 freeze_blocks The list of block IDs to freeze during training. Some times we want to freeze some blocks in the model after loading the pretrained models for some reason (save GPU memory, make training process more stable, etc.). list of ints. For ResNet, the valid block IDs for freezing is any subset of {0, 1, 2, 3}(inclusive). For VGG, the valid block IDs for freezing is any subset of {1, 2, 3, 4, 5}(inclusive). For GoogLeNet, the valid block IDs for freezing is any subset of {0, 1, 2, 3, 4, 5, 6, 7} (inclusive). For MobileNet V1, the valid block IDs is any subset of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}(inclusive). For MobileNet V2, the valid block IDs is any subset of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13}(inclusive). For DarkNet, the valid blocks IDs is any subset of {0, 1, 2, 3, 4, 5}(inclusive). For EfficientNet, the valid block IDs is any subset of { 0, 1, 2, 3, 4, 5, 6, 7}(inclusive). leave it unspecified. use_bias A flag to use bias for convlutional layers in the model. If the model has BatchNormalization layers, we usually set it to False. Boolean. False roi_pooling_config The configuration for the ROIPooling (CropAndResize) layer in the model. proto message. – all_projections A flag to replace all the shortcut layers with projection layers in the model. Only valid for ResNet and MobileNet V2. Boolean. False use_pooling A flag to use pooling layers in the model or not. This parameter is valid only for VGG and ResNet. If set to True, pooling layers will be used in the model(produces the same model structures as in papers). Otherwise, strided convlutional layers will be used and pooling layers will be omitted. Boolean. False activation Defines the activation function used in the model. Only valid for EfficientNet. For INT8 deployment, EfficientNet with relu activation will produces much better accuracy (mAP) than the original swish activation. proto message. –

Each of the above proto message parameters will be described in detail below.

#### Input image configurations

The input_image_config defines the supported format of images by FasterRCNN model. We can customize the input image size, the per-channel mean values and scaling factor for image preprocessing. We can also specify the image type (RGB or grayscale) for our training/validation dataset, and the order of the channel if we are going to use RGB images during training. This is described in the table below in detail.

 Parameter Description Data Type and Constraints Default/Suggested Value image_type The type of the images in the dataset. enum type, either RGB or GRAY_SCALE RGB size_min Specify the input image’s smaller side size, exclusive with size_height_width. proto message with only one min parameter to specify the smaller side size in pixel. – size_height_width Specify the input image’s height and width, exclusive with size_min. proto message with two parameters: height and width to specify a fixed image size. – image_channel_order The image channel order. str type. Can be rgb or bgr for RGB images. l for grayscale images. – image_channel_mean Per-channel mean values for the input images. proto dict that maps each channel to its mean values. – image_scaling_factor The image scaling factor to scale the images. Each pixel value will be divided by this number. float. 1.0 max_objects_num_per_image The maximum number of objects of an image in the dataset. int. 100
Note

The maximum number of objects in an image depends on the dataset. It is important to set the parameter max_objects_num_per_image to be no less than this number. Otherwise, training will fail.

#### Anchor boxes

The parameter anchor_box_config defines the anchor box sizes and aspect ratios in the FasterRCNN model. There are two sub-parameters for it: scale and ratio. Each of them is a list of floats as below.

 Parameter Description Data Type and Constraints Default/Suggested Value scale Anchor box scales (sizes) in pixel. list of floats. – ratio Aspect ratios of the anchor boxes. list of floats. –

#### ROIPooling(CropAndResize)

The roi_pooling_config parameter defines the parameters required in ROIPooling(CropAndResize) layer in the model. Described in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value pool_size The output spatial size (height and width) of the pooled ROIs. Only square ROIs are supported, so this parameter is for both height and width. int. 7 pool_size_2x A flag to double the pooled ROIs’ size. If this is set to True. CropAndResize will produces ROIs of size 2*pool_size and in RCNN it will be downsampled 2x to get back to pool_size. Boolean. –

#### Activation function

The parameter activation defines the type and parameter for the activation function in a FasterRCNN model. This parameter is only valid for EfficientNet.

 Parameter Description Data Type and Constraints Default/Suggested Value activation_type Type of the activation function. Only relu and swish are supported. str. –

### Training configurations

The proto message training_config defines all the necessary parameters required for a FasterRCNN training experiment. Each parameter is described in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value enable_augmentation A flag to enable data augmentation in training. Boolean. True pretrained_weights The path to the pretrained weights for initializing the FasterRCNN model. str. – retrain_pruned_model The path to the pruned model that we are going to retrain. str. – resume_from_model The path to the model for which that we are going to resume an interrupted training. str. – rpn_min_overlap The lower IoU threshold used to match anchor boxes to groundtruth boxes. If the IoU of an anchor box and any groundtruth box is below this threshold, then this anchor box will be regarded as an negative anchor box. float. In the interval (0, 1). 0.3 rpn_max_overlap The higher IoU threshold used to match anchor boxes to groundtruth boxes. If the IoU of an anchor box and some groundtruth box is higher this threshold, then this anchor box will be regarded as an positive anchor box. float. In the interval (0, 1). 0.7 classifier_min_overlap The lower IoU threshold used to generate the proposal target. If the IoU of a ROI and a groundtruth box is above this number and below classifier_max_overlap, then this ROI is regarded as a negative ROI (background) during training. float. In the interval (0, 1). 0.0 classifier_max_overlap The higher IoU threshold used to generate the proposal target. If the IoU of a ROI and a groundtruth box is above this number, then this ROI is regarded as a positive ROI during training. float. In the interval (0, 1). 0.0 gt_as_roi A flag to include groundtruth boxes in the positive ROIs for training the RCNN. Boolean. False std_scaling A scaling factor (multiplier) for RPN regression loss. float. 1.0 classifier_regr_std Scaling factors (denominators) for the RCNN regression loss. A map from ‘x’, ‘y’, ‘w’, ‘h’ to its corresponding scaling factor, respectively. proto dict. {'x': 10, 'y': 10, 'w': 5, 'h': 5} batch_size_per_gpu Training batch size per GPU. int. – num_epochs Number of epochs for the training. int. 20 output_model The path to the checkpoint TAO models during training. str. – checkpoint_interval The period in epochs that we will save the checkpoint. Setting this number to be greater than num_epochs will essentially disable checkpointing. int. 1 rpn_pre_nms_top_N The number of boxes (ROIs) to be retained before the NMS in Proposal layer. int. – rpn_nms_max_boxes The maximum number of boxes (ROIs) to be retained after the NMS in Proposal layer. int. – rpn_nms_overlap_threshold The IoU threshold for NMS in Proposal layer. float. In the interval (0, 1). 0.7 regularizer The configuration for regularizer. proto message. – optimizer The configuration for optimizer. proto message. – learning_rate The configuration for learning rate scheduler. proto message. – lambda_rpn_regr Weighting factor for RPN regression loss. float. 1.0 lambda_rpn_class Weighting factor for RPN classification loss. float. 1.0 lambda_cls_regr Weighting factor for RCNN regression loss. float. 1.0 lambda_cls_class Weighting factor for RCNN classification loss. float. 1.0 enable_qat A flag to enable QAT (quantization-aware training). FasterRCNN does not support loading a non-QAT pruned model and retraining with QAT enabled. Boolean. False model_parallelism List of fraction for model parallelism. Each number is a fraction that represents the percentage of model layers to be placed on a GPU. For example two repeated model_parallelism: 0.5 indicates the training will use 2 GPUs and each GPU will have a half of model layers on it. repeated float – visualizer Visualization configuration during training. proto message – early_stopping The parameters for early stopping. proto message –

The description for proto messages are summarized further below.

#### Regularizer

 Parameter Description Data Type and Constraints Default/Suggested Value type The type of the regularizer. enum type. L1, L2 or NO_REG – weight The penality of the regularizer. float. –

#### Optimizer

Three types of optimizers are supported by FasterRCNN: Adam, SGD and RMSProp. Only one of them should be specified in spec file. No matter which one is chosen, it will be wrapped in a optimizer proto. For example:

Copy
Copied!

optimizer {
lr: 0.00001
beta_1: 0.9
beta_2: 0.999
decay: 0.0
}
}


The Adam optimizer parameters are summarized in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value lr learning rate. This is actually overriden by the learning rate scheduler and hence not useful. float. 0.00001 beta_1 Momentum for the means of the model parameters. float. 0.9 beta_2 Momentum for the variances of the model parameters. float. 0.999 decay decay factor for the learning rate. Not useful float. 0.0

The SGD optimizer parameters are summarized in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value lr learning rate. Not useful as the learning rate is overriden by the learning rate scheduler. float. 0.00001 momentum Momentum of SGD. float. 0.0 decay decay factor of the learning rate. Not useful as overriden by learning rate scheduler. float. 0.0 nesterov A flag to enable Nesterov momentum for SGD. Boolean. False

The RMSProp optimizer parameters are summarized in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value lr learning rate. Not useful as learning rate is overriden by learning rate scheduler. float. 0.00001

#### Learning rate scheduler

The parameter learning_rate defines the learning rate scheduler in a FasterRCNN training. Two types of learning rate schedulers are supported in FasterRCNN: soft_start and step. NO matter which one is chosen, it will be wrapped in a learning_rate proto message. For example:

Copy
Copied!

learning_rate {
step {
base_lr: 0.00001
gamma: 1.0
step_size: 30
}
}


The parameters of soft_start scheduler is described in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value base_lr Maximum learning rate during the training. float. – start_lr The initial learning rate at the start of the training. float. Smaller than base_lr. – soft_start The duration (in percentage of total epochs) of the soft start phase of the learning rate curve. float. In the interval (0, 1). – annealing_points List of time points at which to decrease the learning rate. Also in percentage. list of floats. – annealing_divider divider to decrease the learning rate at each of annealing_points. float. –

The parameters of step scheduler is described in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value base_lr base learning rate at the start of training. float. – gamma multiplier to decrease learning rate. float. – step_size the step size (in percentage of total epochs) at which the learning rate is multiplied by gamma. float. –
Note

The learning rate is automatically scaled with the number of GPUs used during training, or the effective learning rate is learning_rate * n_gpu.

#### Visualization during training

Visualization during training is configured by the visualizer parameter. The parameters of it are described in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value enabled Boolean flag to enable or disable this feature bool. – num_images The maximum number of images to be visualized in TensorBoard. int. 3

Visualization during training supports 3 types of visualizations, namely: scalar, image and histogram. These types of visualization all leverage the TensorBoard tool. Each type will have a tab in TensorBoard GUI interface. With the scalar tab, it can visualize scalars like loss, learning rate and validation mAP over time(training step). With the image tab, it can visualize augmented images during training, with bounding boxes drawn on the them. With the histogram tab, it can visualize histograms of each layer’s weights and bias of the model being trained.

If the parameter enabled is set to True, then all above visualizations will be enabled. Otherwise, all visualization will be disabled.

The parameter num_images is used to limit the maximum number of images to be visualized on the image tab in TensorBoard.

During the training, visualization can be done anywhere that can access the TensorBoard log directory. Usually the TAO Toolkit containers will map volumes to host machine, so TensorBoard can be called on host machine. The command tensorboard --logdir=/path/to/logs can be used to open the TensorBoard visualization GUI in web browser. Make sure tensorboard is installed before running this command. One can run pip3 install tensorboard to install it if it is not installed in the environment. The /path/to/logs argument is the path to the directory used to save the .tlt model, with the suffix /logs appended.

#### Early Stopping

The parameters for early stopping are described in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value monitor The metric to monitor in order to enable early stopping. string loss patience The number of checks of monitor value before stopping the training. int Positive integers min_delta The delta of the minimum value of monitor value below which we regard it as not decreasing. float Non-negative floats

### Inference configurations

The parameter inference_config defines all the parameters required for running inference against a FasterRCNN model.

 Parameter Description Data Type and Constraints Default/Suggested Value images_dir The path to the directory of images to run inference on. str. – model Path to the .tlt model or TensorRT engine to run inference. str. – batch_size Batch size for running inference. int. 1 rpn_pre_nms_top_N The number of boxes (ROIs) to be retained before the NMS in Proposal layer in inference. int. – rpn_nms_max_boxes The maximum number of boxes (ROIs) to be retained after the NMS in Proposal layer in inference. int. – rpn_nms_overlap_threshold The IoU threshold for NMS in Proposal layer. float. In the interval (0, 1). 0.7 object_confidence_thres Object confidence score threshold in NMS. All the objects whose confidence is lower than this number will filtered out in NMS. float. In the interval (0, 1). 0.0001 classifier_nms_max_boxes The maximum number of boxes to retain in RCNN NMS. int. 100 classifier_nms_overlap_threshold RCNN NMS IoU threshold. float. In the interval (0, 1). 0.3 detection_image_output_dir Output directory for detection images. str. – bbox_caption_on A flag to display the class name and confidence for each detected object in an image. Boolean. False labels_dump_dir Output directory to save the labels of the detected objects. str. – trt_inference The configurations for TensorRT based inference. If this parameter is set, inference will use TensorRT engine instead of .tlt model. proto message. – nms_score_bits The number of bits to represent the score values in NMS plugin in TensorRT OSS. The valid range is integers in [1, 10]. Setting it to any other values will make it fall back to ordinary NMS. Currently this optimized NMS plugin is only avaible in FP16 but it should also be selected by INT8 data type as there is no INT8 NMS in TensorRT OSS and hence this fastest implementation in FP16 will be selected. If falling back to ordinary NMS, the actual data type when building the engine will decide the exact precision(FP16 or FP32) to run at. int. In the interval [1, 10]. 0

#### TensorRT based inference

The parameter trt_inference defines all the parameters for TensorRT based inference. When specified, Inference will use TensorRT engine instead of the .tlt model. The TensorRT engine is assumed to be generated by the tao-converter tool. All the parameters are summarized in the table below.

 Parameter Description Data Type and Constraints Default/Suggested Value trt_engine Path to the TensorRT engine file to load. str. –

Note

The parameter trt_inference is deprecated. The parameter model can now support either a .tlt model or a TensorRT engine(any path that does not end with .tlt extension).

### Evaluation configurations

The parameter evaluation_config defines all the required parameters for running evaluation against a FasterRCNN model. This parameter is very similar to inference_config.

 Parameter Description Data Type and Constraints Default/Suggested Value model Path to the model to run evaluation. Can be either a .tlt model or a TensorRT engine. str. – batch_size Batch size for running inference. int. 1 rpn_pre_nms_top_N The number of boxes(ROIs) to be retained before the NMS in Proposal layer in evaluation. int. – rpn_nms_max_boxes The maximum number of boxes(ROIs) to be retained after the NMS in Proposal layer in evaluation. int. – rpn_nms_overlap_threshold The IoU threshold for NMS in Proposal layer. float. In the interval (0, 1). 0.7 object_confidence_thres Object confidence score threshold in NMS. All the objects whose confidence is lower than this number will filtered out in NMS. float. In the interval (0, 1). 0.0001 classifier_nms_max_boxes The maximum number of boxes to retain in RCNN NMS. int. 100 classifier_nms_overlap_threshold RCNN NMS IoU threshold. float. In the interval (0, 1). 0.3 use_voc07_11point_metric A flag to use PASCAL VOC 2007 11-point AP metric. Boolean. – validation_period_during_training The period(in epochs) for doing validation during training. int. 1 trt_evaluation The configurations for TensorRT based evaluation. If this parameter is set, evaluation will use TensorRT engine instead of .tlt model. proto message. – gt_matching_iou_threshold IoU threshold to match detected boxes with groundtruth boxes. Exclusive with gt_matching_iou_threshold_range below. float. 0.5 gt_matching_iou_threshold_range Range of IoU thresholds for computing AP at multiple IoU thresholds and computing COCO mAP. Exclusive with gt_matching_iou_threshold above. proto message. – visualize_pr_curve Boolean flag to enable or disable visualization of Precision-Recall curve. bool. –
Note

The parameter visualize_pr_curve, if set to True, will produce an image of precision-recall curve during the evaluate command, the exact path of the image can be seen in the screen log. By checking the image, we can see each class’s performance regarding the tradeoff between precision and recall.

#### TensorRT based evaluation

In the above table, the definition of trt_evaluation is the same as trt_inference parameter described before.

Note

The parameter trt_evaluation is deprecated. The parameter model can now support either a .tlt model or a TensorRT engine(any path that does not end with .tlt extension).

#### Evaluation IoU Range

The gt_matching_iou_threshold_range parameter is described in table below.

 Parameter Description Data Type and Constraints Default/Suggested Value start start point of the IoU list(inclusive). float. In the interval (0, 1). 0.5 step step size of the IoU list. float. In the interval (0, 1). 0.05 end end point of the IoU list(exclusive). float. In the interval (0, 1]. 1.0

## Training the model

To run training of a FasterRCNN model, use this command:

Copy
Copied!

tao faster_rcnn train [-h] -e <experiment_spec>
[-k <enc_key>]
[--gpus <num_gpus>]
[--num_processes <number_of_processes>]
[--gpu_index <gpu_index>]
[--use_amp]
[--log_file <log_file_path>]


### Required Arguments

• -e, --experiment_spec_file: Experiment specification file to set up the evaluation experiment. This should be the same as training specification file.

### Optional Arguments

• -h, --help: Show this help message and exit.

• -k, --enc_key: TAO encoding key, can override the one in the spec file.

• --gpus: The number of GPUs to be used in the training in a multi-GPU scenario (default: 1).

• --num_processes, -np: Number of processes to be spawned for training. It defaults to be -1(equal to --gpus, for the use case of data parallelism). In the case of model parallelism, this argument should be explicitly set to 1 or more, depending on the actual scenario. Setting --gpus to be larger than 1 and --num_processes to 1 correspoinding to the model parallelism use case; while setting both --gpus and num_processes to be larger than 1 corresponding to the case of enabling both model parallelism and data parallelism. For example, --gpus=4 and --num_processes=2 means 2 horovod processes will be spawned and each of them will occupy 2 GPUs for model parallelism.

• --gpu_index: The GPU indices used to run the training. We can specify the GPU indices used to run training when the machine has multiple GPUs installed.

• --use_amp: A flag to enable AMP training.

• --log_file: Path to the log file. Defaults to stdout.

### Input Requirement

• Input size: C * W * H (where C = 1 or 3, W >= 128, H >= 128)

• Image format: JPG, JPEG, PNG

• Label format: KITTI detection

### Sample Usage

Here’s an example of using the FasterRCNN training command:

Copy
Copied!

tao faster_rcnn train --gpu_index 0 -e <experiment_spec>


### Using a Pretrained Model

Usually, using a pretrained model (weights) file for the initial training of FasterRCNN helps get better accuracy. NVIDIA recommends using the pretrained weights provided in NVIDIA GPU Cloud (NGC). FasterRCNN loads the pretrained weights by name. That is, layer by layer, if TAO Toolkit finds a layer whose name and weights (bias) shape in the pretrained weights file matches a layer in the TAO model, it will load that layer’s weights (and bias, if any) into the model. If some layer in the TAO Toolkit cannot find a matching layer in the pretrained weights, then TAO Toolkit will skip that layer and will use random initialization for that layer instead. An exception is that if TAO Toolkit finds a matching layer in the pretrained weights (and bias, if any) but the shape of the pretrained weights (or bias, if any) in that layer does not match the shape of weights (bias) for the corresponding layer in a TAO model, it will also skip that layer.

For some layers that have no weights (bias), nothing will be done for it(hence will be skipped). So, in total, there are three possible statuses to indicate how a layer’s pretrained weights loading is going on:

• "Yes" means a layer has weights (bias) and is loaded from the pretrained weights file successfully for initialization.

• "No" means a layer has weights (bias) but due to mismatched weights (bias) shape(or probably something else), the weights (bias) cannot be loaded successfully and will use random initialization instead.

• "None" means a layer has no weights (bias) at all and will not load any weights. In the FasterRCNN training log, there is a table that shows the pretrained weights loading status for each layer in the model.

To use a pretrained model in FasterRCNN training, set the pretrained_weights path to point to a pretrained .tlt model (generated with the same encryption key as the FasterRCNN training), a Keras .hdf5 model or a Keras .h5 weights.

Note

At the start of the training, FasterRCNN will print the pretrained model loading status (per-layer). If facing with bad mAP with the model, we can double check this log to see if the pretrained model is loaded properly or not.

Note

FasterRCNN does not support loading a non-QAT pruned model and retraining it with QAT enabled. To make the retrained model a QAT model, it is required to do the initial training with QAT enabled too.

### Re-training a pruned model

A FasterRCNN model can be retrained one or more times. The typical use case is retraining for a pruned model. To retrain an existing FasterRCNN model, set the retrain_pruned_model path to point to an existing FasterRCNN model.

### Resuming an interrupted training

Sometimes a training job can be interrupted due to some reason (e.g., system crash). In these cases, there is no need to redo the training from the start. We can resume the interrupted training from the last checkpoint(saved .tlt model during training). In this case, set the resume_from_model path in spec file to point to the last checkpoint and re-run the training to resume the job.

### Input shape: static and dynamic

FasterRCNN training can support both static input shape and dynamic input shape. Static input shape means the input’s width and height are constant numbers like 960 x 544. Static shape is the most commonly used case in practice. To enable static input shape, we should specify it in input_image_config and augmentation_config. We should use size_height_width in input_image_config to specify the input height and width. Again, we should specify the same two numbers in augmentation_config. That is, we specify the output_image_height and output_image_width in augmentation_config.

With static input shape, we can offline resize the images to the target resolution or we can enable automatic resize during training. By setting enable_auto_resize in augmentation_config to True we will enable automatic resize during training. Automatic resize will reduce the effort to manually resize the images each time we want to train the model on a different resolution. But since resize happens during training, it will potentially increase the training time. Users should make this tradeoff between offline resize and automatic(online) resize.

Dynamic input shape means the input’s height and width are not a constant number but rather can change during training for different images. This kind of input shape is originally proposed in the literature(such as in FasterRCNN paper) where we resize the image and keep aspect ratio such that the resultant image’s smaller side is a given number. Besides the limit on smaller side, we also have a limit on the larger side. If we resize and keep aspect ratio but the resultant image’s larger side’s size exceed this limit on larger side, then we will resize and keep aspect ratio such that the larger side’s size is a given number. In that case, the smaller side will be also no more than its limit. FasterRCNN can support this kind of dynamic input shape. To enable this feature, we have to specify size_min in input_image_config and specify output_image_min and output_image_max in augmentation_confg. size_min and output_image_min indicates the limit of the smaller side’s size, while output_image_max indicates the limit on the larger side’s size.

Note that there are some limitations regarding the dynamic shape of FasterRCNN.

• TAO FasterRCNN training/evaluation/inference can only work with batch size 1.

• TAO FasterRCNN export & DeepStream(TensorRT) inference/evaluation does not support dynamic shape for now.

### Model parallelism

FasterRCNN supports model parallelism. Model parallelism is a technique that we split the entire model on multiple GPUs and each GPU will hold a part of the model. A model is splitted by layers. For example, if a model has 100 layers, then we can place the layer 0-49 on GPU 0 and layer 50-99 on GPU 1. Model parallelism will be useful when the model is huge and cannot fit into a single GPU even with batch size 1. Model parallelism is also useful if we want to increase the batch size that is seen by BatchNormalization layers and hence potentially improve the accuracy. This feature can be enabled by setting model_parallelism in training_config. For example,

Copy
Copied!

model_parallelism: 0.3
model_parallelism: 0.7


will enable a 2-GPU model parallelism where the first GPU will hold 30% of the model layers and the second GPU will hold 70% of the model layers. The percentage of model layers can be adjusted with some trial-and-error so all GPUs consumes almost the same GPU memory size and in that case we can use the largest batch size for this model-parallelised training.

Model parallelism can be enabled jointly with data parallelism. For example, in above case we enabled a 2-GPU model parallelism, at the same time we can also enable 4 horovod processes for it. In this case, we have 4 horovod processes for data parallelism and each process will have the model splitted on 2 GPUs.

## Evaluating the model

To run evaluation for a faster_rcnn model, use this command:

Copy
Copied!

tao faster_rcnn evaluate [-h] -e <experiment_spec>
[-k <enc_key>]
[--gpu_index <gpu_index>]
[--log_file <log_file_path>]
[-m <model_path>]


### Required Arguments

• -e, --experiment_spec_file: Experiment spec file to set up the evaluation experiment. This should be the same as a training spec file.

### Optional Arguments

• -h, --help: show this help message and exit.

• -k, --enc_key：The encoding key, can override the one in the spec file.

• --gpu_index: The GPU index used to run the evaluation. We can specify the GPU index used to run evaluation when the machine has multiple GPUs installed. Note that evaluation can only run on a single GPU.

• --log_file: Path to the log file. Defaults to stdout.

• -m, --model: Path to the model to run evaluation. The model can be either a .tlt model or a TensorRT engine(any path that does not end with .tlt extension). The model path(if provided in the command line here) will override the evaluation_config.model in the spec file.

### Evaluation Metrics

#### The PASCAL VOC 2007 vs 2012 metrics

For FasterRCNN, the evaluation will produce 4 metrics for the evaluated model: AP (average precision), precision, recall and RPN_recall for each class in the evaluation dataset. inally, it will also print the mAP (mean average precision) as a single metric number. Two modes are supported for computing the AP, i.e., the PASCAL VOC 2007 and 2012 metrics. This can be configured in the spec file’s evaluation_config.use_voc_11_point_metric parameter. If this parameter is set to True, then AP calculation will use VOC 2007 method, otherwise it will use the VOC 2012 method.

#### Setting IoU value/range for computing AP/mAP

For matching the detected objects to groundtruth objects, we can define different IoU thresholds. An IoU of 0.5 is used in PASCAL VOC metrics, while in MS COCO a list of IoUs are used to compute the AP. For example, in MS COCO, the mAP@[0.5:0.05:0.95] is the averaged AP at 10 different IoUs, starting from 0.5 and ends with 0.95, with a step size of 0.05. TAO FasterRCNN supports evaluating AP at a list of IoUs and computing the mAP across the range of IoUs. Specifically, setting gt_matching_iou_threshold in evaluation_config will produce the AP/mAP at a single IoU; setting gt_matching_iou_threshold_range for a list (range) of IoUs will produce AP at these IoU values and the mAP. In order to compute PASCAL VOC mAP, we can set the former to 0.5. While in order to compute COCO mAP, we can set the latter to be start: 0.5, step: 0.05 and end: 1.0.

The RPN_recall metric indicates the recall capability of the RPN of the FasterRCNN model. The higher the RPN_recall metric, it means RPN can better detect an object as foreground (but it doesn’t say anything on which class this object belongs to since that is delegated to RCNN). The RPN_recall metric is mainly used for debugging on the accuracy issue of a FasterRCNN model.

## Running inference on the model

The inference tool for FasterRCNN networks can be used to visualize bboxes or generate frame by frame KITTI format labels on a directory of images. You can execute this tool from the command line as shown here:

Copy
Copied!

tao faster_rcnn inference [-h] -e <experiment_spec>
[-k <enc_key>]
[--gpu_index <gpu_index>]
[--log_file <log_file_path>]
[-m <model_path>]


### Required Arguments

• -e, --experiment_spec_file: Path to the experiment specification file for FasterRCNN training.

### Optional Arguments

• -h, --help: Print help log and exit.

• -k, --enc_key: The encoding key, can override the one in the spec file.

• --gpu_index: The GPU index to run inference on. We can specify the GPU index used to run inference if the machine has multiple GPUs installed. Note that inference can only run on a single GPU.

• --log_file: Path to the log file. Defaults to stdout.

• -m, --model: The path to the model to be used for inference. The model can be either a .tlt model or a TensorRT engine(any path that does not end with the .tlt extension). The model path(if provided in the command line here) will override the inference_config.model in the spec file.

## Pruning the model

Pruning removes parameters from the model to reduce the model size without compromising the integrity of the model itself using the tao faster_rcnn prune command.

The tao faster_rcnn prune command includes these parameters:

Copy
Copied!

tao faster_rcnn prune [-h] -m <model>
-o <output_file>
-k <key>
[-n <normalizer>]
[-eq <equalization_criterion>]
[-pg <pruning_granularity>]
[-pth <pruning threshold>]
[-nf <min_num_filters>]
[-el [<excluded_list>]
[--gpu_index <gpu_index>]
[--log_file <log_file_path>]


### Required Arguments

• -m, --model: Path to a pretrained .tlt model to be pruned.

• -o, --output_file: Path to save the pruned .tlt model.

• -k, --ke: Key to load a :code.tlt model.

### Optional Arguments

• -h, --help: Show this help message and exit.

• -n, –normalizer: max to normalize by dividing each norm by the maximum norm within a layer; L2 to normalize by dividing by the L2 norm of the vector comprising all kernel norms. (default: max)

• -eq, --equalization_criterion: Criteria to equalize the stats of inputs to an element wise op layer or depth-wise convolutional layer. This parameter is useful for resnets and mobilenets. Options are arithmetic_mean, geometric_mean, union, and intersection. (default: union)

• -pg, -pruning_granularity: Number of filters to remove at a time (default:8)

• -pth: Threshold to compare normalized norm against (default:0.1)

Note

NVIDIA recommends changing the threshold to keep the number of parameters in the model to within 10-20% of the original unpruned model.

• -nf, --min_num_filters: Minimum number of filters to keep per layer (default:16)

• -el, --excluded_layers: List of excluded_layers. Examples: -i item1 item2 (default: [])

• --gpu_index: The GPU index to run pruning on. We can specify the GPU index used to run pruning if the machine has multiple GPUs installed. Note that pruning can only run on a single GPU.

• --log_file: Path to the log file. Defaults to stdout.

After pruning, the model needs to be retrained. See Re-training the Pruned Model for more details.

### Using the Prune Command

Here’s an example of using the tao faster_rcnn prune command:

Copy
Copied!

tao faster_rcnn prune -m /workspace/output/weights/resnet_003.tlt
-o /workspace/output/weights/resnet_003_pruned.tlt
-eq union
-pth 0.7
-k nvidia_tlt


## Retraining the pruned model

Once the model has been pruned, there might be a slight decrease in accuracy. This happens because some previously useful weights may have been removed. In order to regain the accuracy, NVIDIA recommends that you retrain this pruned model over the same dataset. To do this, use the tao faster_rcnn train command as documented in Training the model with an updated spec file that points to the newly pruned model as the pretrained model file.

Users are advised to turn off the regularizer(set regularizer type to NO_REG) or use a smaller weight decay in the spec file to recover the accuracy when retraining a pruned model. All the other parameters may be retained in the spec file from the previous training.

For FasterRCNN, it is important to set the retrain_pruned_model path to point to the pruned model.

## Exporting the model

Exporting the model decouples the training process from inference and allows conversion to TensorRT engines outside the TAO environment. TensorRT engines are specific to each hardware configuration and should be generated for each unique inference environment. The exported model may be used universally across training and deployment hardware. The exported model format is referred to as .etlt. Like .tlt, the .etlt model format is also a encrypted model format with the same key of the .tlt model that it is exported from. This key is required when deploying this model.

FasterRCNN export can optionally generate a (partial) DeepStream configuration file and label file. See below.

### INT8 Mode Overview

TensorRT engines can be generated in INT8 mode to improve performance, but require a calibration cache at engine creation-time. The calibration cache is generated using a calibration tensor file, if export is run with the --data_type flag set to int8. Pre-generating the calibration information and caching it removes the need for calibrating the model on the inference device. Using the calibration cache also speeds up engine creation as building the cache can take several minutes to generate depending on the size of the calibration data and the model itself.

The export tool can generate INT8 calibration cache by ingesting training data using either of these options:

• Option 1: Using the training data loader to load the training images for INT8 calibration. This option is now the recommended approach to support multiple image directories by leveraging the training dataset loader. This also ensures two important aspects of data during calibration:

• Data pre-processing in the INT8 calibration step is the same as in the training process.

• The data batches are sampled randomly across the entire training dataset, thereby improving the accuracy of the INT8 model.

• Option 2: Pointing the tool to a directory of images that you want to use to calibrate the model. For this option, make sure to create a sub-sampled directory of random images that best represent your training dataset.

### FP16/FP32 Model

The calibration.bin is only required if you need to run inference at INT8 precision. For FP16/FP32 based inference, the export step is much simpler. All that is required is to provide a .tlt model from the training/retraining step to be converted into an .etlt.

### Exporting the Model

Here’s an example of the tao faster_rcnn export command:

Copy
Copied!

tao faster_rcnn export [-h] -m <path to the .tlt model file generated by training>
-k <key>
--experiment_spec <path to experiment spec file>
[-o <path to output file>]
[--cal_data_file <path to tensor file>]
[--cal_image_dir <path to the directory images to calibrate the model]
[--cal_cache_file <path to output calibration file>]
[--data_type <data type for the TensorRT backend during export>]
[--batches <number of batches to calibrate over>]
[--max_batch_size <maximum trt batch size>]
[--max_workspace_size <maximum workspace size]
[--batch_size <batch size to TensorRT engine>]
[--engine_file <path to the TensorRT engine file>]
[--gen_ds_config]
[--verbose]
[--strict_type_constraints]
[--force_ptq]
[--gpu_index <gpu_index>]
[--log_file <log_file_path>]


#### Required Arguments

• -m, --model: Path to the .tlt model file to be exported.

• -k, --key: Key used to save the .tlt model file.

• -e, --experiment_spec: Path to the spec file.

#### Optional Arguments

• -o, --output_file: Path to save the exported model to. The default is ./<input_file>.etlt.

• --data_type: Desired engine data type, generates calibration cache if in INT8 mode. The options are: {fp32, fp16, int8} The default value is fp32. If using INT8, the following INT8 arguments are required.

• --gen_ds_config: A Boolean flag indicating whether to generate the partial DeepStream related configuration (“nvinfer_config.txt”) as well as a label file (“labels.txt”) in the same directory as the output_file. Note that the config file is NOT a complete configuration file and requires the user to update the sample config files in DeepStream with the parameters generated.

• -s, --strict_type_constraints: A Boolean flag to indicate whether or not to apply the TensorRT strict type constraints when building the TensorRT engine.

• --gpu_index: The index of (discrete) GPUs used for exporting the model. We can specify the GPU index to run export if the machine has multiple GPUs installed. Note that export can only run on a single GPU.

• --log_file: Path to the log file. Defaults to stdout.

### INT8 Export Mode Required Arguments

• --cal_data_file: The output tensorfile if used with --cal_image_dir.

• --cal_image_dir: Directory of images to use for calibration.

Note

--cal_image_dir parameter for images and applies the necessary preprocessing to generate a tensorfile at the path mentioned in the --cal_data_file parameter which is in turn used for calibration. The number of batches in the tensorfile generated is obtained from the value set to the --batches parameter, and the batch_size is obtained from the value set to the --batch_size parameter. Be sure that the directory mentioned in --cal_image_dir has at least batch_size * batches number of images in it. The valid image extensions are .jpg, .jpeg, and .png. In this case, the input_dimensions of the calibration tensors are derived from the input layer of the .tlt model.

### INT8 Export Optional Arguments

• --cal_cache_file: Path to save the calibration cache file. The default value is ./cal.bin.

• --batches: Number of batches to use for calibration and inference testing.The default value is 10.

• --batch_size: Batch size to use for calibration. The default value is 8.

• --max_batch_size: Maximum batch size of TensorRT engine. The default value is 16.

• --max_workspace_size: Maximum workspace size of TensorRT engine. The default value is: 1073741824(1<<30).

• --engine_file: Path to the serialized TensorRT engine file. Note that this file is hardware specific, and cannot be generalized across GPUs. Useful to quickly test your model accuracy using TensorRT on the host. As TensorRT engine file is hardware specific, you cannot use this engine file for deployment unless the deployment GPU is identical to training GPU.

• --force_ptq: A boolean flag to force post training quantization on the exported etlt model.

Note

When exporting a model trained with QAT enabled, the tensor scale factors to calibrate the activations are peeled out of the model and serialized to a TensorRT readable cache file defined by the cal_cache_file argument. However, do note that the current version of QAT doesn’t natively support DLA INT8 deployment in the Jetson device. In order to deploy this model on a Jetson with DLA INT8, use the --force_ptq flag to use TensorRT post training quantization to generate the calibration cache file.

### Exporting a Model

Here’s a sample command using the data loader for loading calibration data to calibrate a FasterRCNN model using option 1.

Copy
Copied!

tao faster_rcnn export --gpu_index 0
-m $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain.epoch12.tlt -o$USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain_int8.etlt
-e $SPECS_DIR/default_spec_resnet18_retrain_spec.txt -k nvidia_tlt --data_type int8 --batch_size 8 --batches 10 --cal_cache_file$USER_EXPERIMENT_DIR/data/faster_rcnn/cal.bin


## Deploying to DeepStream

The deep learning and computer vision models that you’ve trained can be deployed on edge devices, such as a Jetson Xavier or Jetson Nano, a discrete GPU, or in the cloud with NVIDIA GPUs. TAO Toolkit has been designed to integrate with DeepStream SDK, so models trained with TAO Toolkit will work out of the box with DeepStream SDK.

DeepStream SDK is a streaming analytic toolkit to accelerate building AI-based video analytic applications. This section will describe how to deploy your trained model to DeepStream SDK.

To deploy a model trained by TAO Toolkit to DeepStream we have two options:

• Option 1: Integrate the .etlt model directly in the DeepStream app. The model file is generated by export.

• Option 2: Generate a device specific optimized TensorRT engine using tao-converter. The generated TensorRT engine file can also be ingested by DeepStream.

Machine-specific optimizations are done as part of the engine creation process, so a distinct engine should be generated for each environment and hardware configuration. If the TensorRT or CUDA libraries of the inference environment are updated (including minor version updates), or if a new model is generated, new engines need to be generated. Running an engine that was generated with a different version of TensorRT and CUDA is not supported and will cause unknown behavior that affects inference speed, accuracy, and stability, or it may fail to run altogether.

Option 1 is very straightforward. The .etlt file and calibration cache are directly used by DeepStream. DeepStream will automatically generate the TensorRT engine file and then run inference. TensorRT engine generation can take some time depending on size of the model and type of hardware. Engine generation can be done ahead of time with Option 2. With option 2, the tao-converter is used to convert the .etlt file to TensorRT; this file is then provided directly to DeepStream.

See the Exporting the Model section for more details on how to export a TAO model.

### TensorRT Open Source Software (OSS)

TensorRT OSS build is required for FasterRCNN models. This is required because several TensorRT plugins that are required by these models are only available in TensorRT open source repo and not in the general TensorRT release. Specifically, for FasterRCNN, we need the cropAndResizePlugin and proposalPlugin.

If the deployment platform is x86 with NVIDIA GPU, follow instructions for x86. If your deployment is on NVIDIA Jetson platform, follow instructions for Jetson.

#### TensorRT OSS on x86

Building TensorRT OSS on x86:

1. Install Cmake (>=3.13).

Note

TensorRT OSS requires cmake >= v3.13, so install cmake 3.13 if your cmake version is lower than 3.13c

Copy
Copied!

sudo apt remove --purge --auto-remove cmake
tar xvf cmake-3.13.5.tar.gz
cd cmake-3.13.5/
./configure
make -j$(nproc) sudo make install sudo ln -s /usr/local/bin/cmake /usr/bin/cmake  2. Get GPU architecture. The GPU_ARCHS value can be retrieved by the deviceQuery CUDA sample: Copy Copied!  cd /usr/local/cuda/samples/1_Utilities/deviceQuery sudo make ./deviceQuery  If the /usr/local/cuda/samples doesn’t exist in your system, you could download deviceQuery.cpp from this GitHub repo. Compile and run deviceQuery. Copy Copied!  nvcc deviceQuery.cpp -o deviceQuery ./deviceQuery  This command will output something like this, which indicates the GPU_ARCHS is 75 based on CUDA Capability major/minor version. Copy Copied!  Detected 2 CUDA Capable device(s) Device 0: "Tesla T4" CUDA Driver Version / Runtime Version 10.2 / 10.2 CUDA Capability Major/Minor version number: 7.5  3. Build TensorRT OSS: Copy Copied!  git clone -b 21.08 https://github.com/nvidia/TensorRT cd TensorRT/ git submodule update --init --recursive export TRT_SOURCE=pwd cd$TRT_SOURCE
mkdir -p build && cd build


Note

Make sure your GPU_ARCHS from step 2 is in TensorRT OSS CMakeLists.txt. If GPU_ARCHS is not in TensorRT OSS CMakeLists.txt, add -DGPU_ARCHS=<VER> as below, where <VER> represents GPU_ARCHS from step 2.

Copy
Copied!

/usr/local/bin/cmake .. -DGPU_ARCHS=xy  -DTRT_LIB_DIR=/usr/lib/x86_64-linux-gnu/ -DCMAKE_C_COMPILER=/usr/bin/gcc -DTRT_BIN_DIR=pwd/out
make nvinfer_plugin -j$(nproc)  After building ends successfully, libnvinfer_plugin.so* will be generated under \pwd\/out/. 4. Replace the original libnvinfer_plugin.so*: Copy Copied!  sudo mv /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.8.x.y${HOME}/libnvinfer_plugin.so.8.x.y.bak   // backup original libnvinfer_plugin.so.x.y
sudo cp $TRT_SOURCE/pwd/out/libnvinfer_plugin.so.8.m.n /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.8.x.y sudo ldconfig  #### TensorRT OSS on Jetson (ARM64) 1. Install Cmake (>=3.13) Note TensorRT OSS requires cmake >= v3.13, while the default cmake on Jetson/Ubuntu 18.04 is cmake 3.10.2. Upgrade TensorRT OSS using: Copy Copied!  sudo apt remove --purge --auto-remove cmake wget https://github.com/Kitware/CMake/releases/download/v3.13.5/cmake-3.13.5.tar.gz tar xvf cmake-3.13.5.tar.gz cd cmake-3.13.5/ ./configure make -j$(nproc)
sudo make install
sudo ln -s /usr/local/bin/cmake /usr/bin/cmake


2. Get GPU architecture based on your platform. The GPU_ARCHS for different Jetson platform are given in the following table.

 Jetson Platform GPU_ARCHS Nano/Tx1 53 Tx2 62 AGX Xavier/Xavier NX 72
3. Build TensorRT OSS:

Copy
Copied!

git clone -b 21.03 https://github.com/nvidia/TensorRT
cd TensorRT/
git submodule update --init --recursive
export TRT_SOURCE=pwd
cd $TRT_SOURCE mkdir -p build && cd build  Note The -DGPU_ARCHS=72 below is for Xavier or NX, for other Jetson platform, change 72 referring to GPU_ARCHS from step 2. Copy Copied!  /usr/local/bin/cmake .. -DGPU_ARCHS=72 -DTRT_LIB_DIR=/usr/lib/aarch64-linux-gnu/ -DCMAKE_C_COMPILER=/usr/bin/gcc -DTRT_BIN_DIR=pwd/out make nvinfer_plugin -j$(nproc)


After building ends successfully, libnvinfer_plugin.so* will be generated under ‘pwd’/out/.

4. Replace "libnvinfer_plugin.so*" with the newly generated.

Copy
Copied!

sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8.x.y ${HOME}/libnvinfer_plugin.so.8.x.y.bak // backup original libnvinfer_plugin.so.x.y sudo cp pwd/out/libnvinfer_plugin.so.8.m.n /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8.x.y sudo ldconfig  ### Generating an Engine Using tao-converter The tao-converter tool is provided with the TAO Toolkit to facilitate the deployment of TAO trained models on TensorRT and/or Deepstream. This section elaborates on how to generate a TensorRT engine using tao-converter. For deployment platforms with an x86-based CPU and discrete GPUs, the tao-converter is distributed within the TAO docker. Therefore, we suggest using the docker to generate the engine. However, this requires that the user adhere to the same minor version of TensorRT as distributed with the docker. The TAO docker includes TensorRT version 8.0. #### Instructions for x86 1. Copy /opt/nvidia/tools/tao-converter to the target machine. 2. Install TensorRT for the respective target machine. 3. For FasterRCNN, we need to build TensorRT Open source software on the machine. Instructions to build TensorRT OSS on x86 can be found in TensorRT OSS on x86 section above or in this GitHub repo. 4. Run tao-converter using the sample command below and generate the engine. #### Instructions for Jetson For the Jetson platform, the tao-converter is available to download in the dev zone. Once the tao-converter is downloaded, follow the instructions below to generate a TensorRT engine. 1. Unzip tao-converter-trt7.1.zip on the target machine. 2. Install the open ssl package using the command: Copy Copied!  sudo apt-get install libssl-dev  3. Export the following environment variables: Copy Copied!  $ export TRT_LIB_PATH=”/usr/lib/aarch64-linux-gnu”
\$ export TRT_INC_PATH=”/usr/include/aarch64-linux-gnu”


1. For Jetson devices, TensorRT comes pre-installed with Jetpack. If you are using older JetPack, upgrade to the latest one that tao-converter can support.

2. For FasterRCNN, instructions to build TensorRT OSS on Jetson can be found in TensorRT OSS on Jetson (ARM64) section above or in this GitHub repo.

3. Run the tao-converter using the sample command below and generate the engine.

Note

Make sure to follow the output node names as mentioned in Exporting the Model.

#### Using the tao-converter

Copy
Copied!

tao-converter [-h] -k <encryption_key>
-d <input_dimensions>
-o <comma separated output nodes>
[-c <path to calibration cache file>]
[-e <path to output engine>]
[-b <calibration batch size>]
[-m <maximum batch size of the TRT engine>]
[-t <engine datatype>]
[-w <maximum workspace size of the TRT Engine>]
[-i <input dimension ordering>]
[-p <optimization_profiles>]
[-s]
[-u <DLA_core>]
input_file


##### Required Arguments
• input_file: Path to the .etlt model exported using export.

• -k: The key used to encode the .tlt model when doing the traning.

• -d: Comma-separated list of input dimensions that should match the dimensions used for export. Unlike export this cannot be inferred from calibration data. This parameter is not required for new models introduced in TAO Toolkit 3.0-21.08 (e.g., LPRNet, UNet, GazeNet, etc).

• -o: Comma-separated list of output blob names that should match the output configuration used for export. This parameter is not required for new models introduced in TAO Toolkit 3.0 (e.g., LPRNet, UNet, GazeNet, etc). For FasterRCNN, set this argument to NMS.

##### Optional Arguments
• -e: Path to save the engine to. (default: ./saved.engine)

• -t: Desired engine data type, generates calibration cache if in INT8 mode. The default value is fp32. The options are {fp32, fp16, int8}.

• -w: Maximum workspace size for the TensorRT engine. The default value is 1073741824(1<<30).

• -i: Input dimension ordering, all other TAO commands use NCHW. The default value is nchw. The options are {nchw, nhwc, nc}. For FasterRCNN, we can omit it (defaults to nchw).

• -p: Optimization profiles for .etlt models with dynamic shape. Comma separated list of optimization profile shapes in the format <input_name>,<min_shape>,<opt_shape>,<max_shape>, where each shape has the format: <n>x<c>x<h>x<w>. Can be specified multiple times if there are multiple input tensors for the model. This is only useful for new models introduced in TAO Toolkit 3.21.08. This parameter is not required for models that are already existed in version 2.0.

• -s: TensorRT strict type constraints. A Boolean to apply TensorRT strict type constraints when building the TensorRT engine.

• -u: Use DLA core. Specifying DLA core index when building the TensorRT engine on Jetson devices.

##### INT8 Mode Arguments
• -c: Path to calibration cache file, only used in INT8 mode. The default value is ./cal.bin.

• -b: Batch size used during the export step for INT8 calibration cache generation. (default: 8).

• -m: Maximum batch size for TensorRT engine.(default: 16). If meet with out-of-memory issue, please decrease the batch size accordingly. This parameter is not required for .etlt models generated with dynamic shape (This is only possible for new models introduced in TAO Toolkit 3.21.08).

##### Sample Output Log

Here is a sample log for exporting a FasterRCNN model.

Copy
Copied!

tao-converter -d 3,544,960 \
-k nvidia_tlt \
-o NMS \
/workspace/tao-experiments/faster_rcnn/resnet18_pruned.epoch45.etlt
..
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[INFO] Detected 1 inputs and 2 output network tensors.


### Integrating the model to DeepStream

There are 2 options to integrate models from TAO with DeepStream:

• Option 1: Integrate the model (.etlt) with the encrypted key directly in the DeepStream app. The model file is generated by export.

• Option 2: Generate a device specific optimized TensorRT engine using tao-converter. The TensorRT engine file can also be ingested by DeepStream.

For FasterRCNN, we will need to build TensorRT Open source plugins and custom bounding box parser. The instructions are provided below in the TensorRT OSS section above and the required code can be found in this GitHub repo.

In order to integrate the models with DeepStream, you need the following:

1. Download and install DeepStream SDK. The installation instructions for DeepStream are provided in the DeepStream Development Guide.

2. An exported .etlt model file and optional calibration cache for INT8 precision.

3. A labels.txt file containing the labels for classes in the order in which the networks produces outputs.

4. A sample config_infer_*.txt file to configure the nvinfer element in DeepStream. The nvinfer element handles everything related to TensorRT optimization and engine creation in DeepStream.

DeepStream SDK ships with an end-to-end reference application which is fully configurable. Users can configure input sources, inference model, and output sinks. The app requires a primary object detection model, followed by an optional secondary classification model. The reference application is installed as deepstream-app. The graphic below shows the architecture of the reference application.

There are typically 2 or more configuration files that are used with this app. In the install directory, the config files are located in samples/configs/deepstream-app or sample/configs/tlt_pretrained_models. The main config file configures all the high level parameters in the pipeline above. This would set input source and resolution, number of inferences, tracker, and output sinks. The other supporting config files are for each individual inference engine. The inference specific config files are used to specify models, inference resolution, batch size, number of classes and other customization. The main config file will call all the supporting config files. Here are some config files in samples/configs/deepstream-app for your reference.

• source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt: Main config file

• config_infer_primary.txt: Supporting config file for primary detector in the pipeline above

• config_infer_secondary_*.txt: Supporting config file for secondary classifier in the pipeline above

The deepstream-app will only work with the main config file. This file will most likely remain the same for all models and can be used directly from the DeepStream SDK will little to no change. User will only have to modify or create config_infer_primary.txt and config_infer_secondary_*.txt.

#### Integrating a FasterRCNN Model

To run a FasterRCNN model in DeepStream, you need a label file and a DeepStream configuration file. In addition, you need to compile the TensorRT Open source software and FasterRCNN bounding box parser for DeepStream.

A DeepStream sample with documentation on how to run inference using the trained FasterRCNN models from TAO Toolkit is provided on GitHub here.

##### Prerequisite for FasterRCNN Model
1. FasterRCNN requires the cropAndResizePlugin and the proposalPlugin. This plugin is available in the TensorRT open source repo. Detailed instructions to build TensorRT OSS can be found in TensorRT Open Source Software (OSS).

2. FasterRCNN requires custom bounding box parsers that are not built-in inside the DeepStream SDK. The source code to build custom bounding box parsers for FasterRCNN is available here. The following instructions can be used to build bounding box parser:

Step 1: Install git-lfs (git >= 1.8.2)

Copy
Copied!

curl -s https://packagecloud.io/install/repositories/github/git-lfs/
script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install


Copy
Copied!

git clone -b release/tlt3.0 https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps


Step 3: Build

Copy
Copied!

// or Path for DS installation
export CUDA_VER=10.2         // CUDA version, e.g. 10.2
make


This generates libnvds_infercustomparser_tlt.so in the directory post_processor.

##### Label File

The label file is a text file containing the names of the classes that the FasterRCNN model is trained to detect. The order in which the classes are listed here must match the order in which the model predicts the output. This order is derived from the order the objects are instantiated in the target_class_mapping field of the FasterRCNN experiment specification file. During the training, TAO FasterRCNN will make all the class names in lower case and sort them in alphabetical order. For example, if the target_class_mapping label file is:

Copy
Copied!

target_class_mapping {
key: "car"
value: "car"
}
target_class_mapping {
key: "person"
value: "person"
}
target_class_mapping {
key: "bicycle"
value: "bicycle"
}


The actual class name list is bicycle, car, person. The example of the corresponding label_file_frcnn.txt file is (we always append a background class at the end):

Copy
Copied!

bicycle
car
person
background


Note

If --gen_ds_config is provided during TAO export of a FasterRCNN model, then a label file named labels.txt will be generated automatically. Without knowing the above details, the labels.txt file can be used directly in DeepStream inference.

##### DeepStream Configuration File

The detection model is typically used as a primary inference engine. It can also be used as a secondary inference engine. To run this model in the sample deepstream-app, you must modify the existing config_infer_primary.txt file to point to this model as well as the custom parser.

Option 1: Integrate the model (.etlt) directly in the DeepStream app.

For this option, users will need to add the following parameters in the configuration file. The int8-calib-file is only required for INT8 precision.

Copy
Copied!

tlt-encoded-model=<TLT exported .etlt>
tlt-model-key=<Model export key>
int8-calib-file=<Calibration cache file>


The tlt-encoded-model parameter points to the exported model (.etlt) from TAO Toolkit. The tlt-model-key is the encryption key used during model export.

Option 2: Integrate TensorRT engine file with DeepStream app.

Step 1: Generate TensorRT engine using tao-converter. See the Generating an engine using tao-converter section above for detailed instructions.

Step 2: Once the engine file is generated successfully, modify the following parameters to use this engine with DeepStream.

Copy
Copied!

model-engine-file=<PATH to generated TensorRT engine>


All other parameters are common between the 2 approaches. To use the custom bounding box parser instead of the default parsers in DeepStream, modify the following parameters in [property] section of primary infer configuration file:

Copy
Copied!

parse-bbox-func-name=NvDsInferParseCustomNMSTLT
custom-lib-path=<PATH to libnvds_infercustomparser_tlt.so>


Add the label file generated above using:

Copy
Copied!

labelfile-path=<Classification labels>


For all the options, see the configuration file below. To learn about what all the parameters are used for, refer to DeepStream Development Guide.

Here’s a sample config file, config_infer_primary.txt:

Copy
Copied!

[property]
gpu-id=0
net-scale-factor=1.0
offsets=<image mean values as in the training spec file> # e.g.: 103.939;116.779;123.68
model-color-format=1
labelfile-path=<Path to frcnn_labels.txt>
tlt-encoded-model=<Path to FasterRCNN model>
tlt-model-key=<Key to decrypt the model>
infer-dims=<c;h;w> # e.g., 3;544;960 Where c = number of channels, h = height of the model input, w = width of model input
uff-input-order=0
uff-input-blob-name=<input_blob_name> # e.g.: input_image
batch-size=<batch size> e.g.: 1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=<number of classes to detect(including background)> #
e.g.: 5
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
output-blob-names=<output_blob_names> e.g.: NMS
parse-bbox-func-name=NvDsInferParseCustomNMSTLT
custom-lib-path=<PATH to libnvds_infercustomparser_tlt.so>

[class-attrs-all]
pre-cluster-threshold=0.6
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0


Note

If --gen_ds_config is provided during TAO export of a FasterRCNN model, then a config file named nvinfer_config.txt will be generated automatically. This file is an incomplete config file for DeepStream inference; you should copy and paste available fields in this partial config file to you own complete config file.