Use this procedure to run the Transfer Learning Toolkit.

• Run the toolkit: Run the toolkit using this command. The docker starts in the /workplace folder by default.

  docker run --runtime=nvidia -it nvcr.io/nvidia/tlt-streamanalytics:<version> /bin/bash

• Access local directories: To access local directories from inside the docker you need to mount them in the docker. Use this option, -v <source_dir>:<mount_dir>, to mount local directories in the docker. For example the command to run the toolkit mounting the /home/<username>/tlt-experiments directory in your disk to the /workspace/tlt-experiments in docker would be:

  docker run --runtime=nvidia -it -v /home/<username>/tlt-experiments:/workspace/tlt-experiments nvcr.io/nvidia/tlt-streamanalytics:<version> /bin/bash

  It is useful to mount separate volumes for the dataset and the experiment results so that they persist outside of the docker. In this way the data is preserved after the docker is closed. Any data that is generated to, or referred from a directory inside the docker, will be lost if it is not either copied out of the docker, or written to or read from volumes outside of the docker.
• Use the examples: Examples using ResNet18 backbone for detecting objects with either DetectNet_v2, SSD, or FasterRCNN architectures are available as Jupyter Notebooks. To run the examples that are available, enable the jupyter notebook included in the docker to run in your browser:

  docker run --runtime=nvidia -it -v /home/<username>/tlt-experiments:/workspace/tlt-experiments -p 8888:8888 tlt-streamanalytics:<version>

  Go to the examples folder: cd examples/

  Execute this command from inside the docker to start the jupyter notebook:

  jupyter notebook --ip 0.0.0.0 --allow-root

  Copy and paste the link produced from this command into your browser to access the notebook. The /workspace/examples folder will contain a demo notebook.

The Transfer Learning Toolkit docker gives you access to a repository of pretrained models that can serve as a starting point when training deep neural networks. These models are hosted on the Nvidia GPU Cloud (NGC). The TLT docker interfaces with NGC via the NGC Catalog CLI. More information about the NGC Catalog CLI is available here. https://docs.nvidia.com/ngc/ngc-catalog-cli-user-guide/index.html". Please follow the instructions given here to configure the NGC CLI and download the models.

Configure the NGC API key

Using the NGC API Key obtained in Transfer Learning Toolkit Requirements, configure the enclosed ngc cli by executing this command and following the prompts:

ngc config set

Getting a list of models

Use this command to get a list of models that are hosted in the NGC model registry:

ngc registry model list <model_glob_string>

Here is an example of using this command:

ngc registry model list nvidia/iva/tlt_*_classification

Downloading a model

Use this command to download the model you have chosen from the NGC model registry:

ngc registry model download-version <ORG/model_name:version> -d <path_to_download_dir>

For example, use this command to download the resnet 18 classification model to the $USER_EXPERIMENT_DIR directory.

ngc registry model download-version nvidia/iva/tlt_resnet18_classification:1 -d $USER_EXPERIMENT_DIR/pretrained_resnet18

Downloaded 82.41 MB in 9s, Download speed: 9.14 MB/s                
----------------------------------------------------
Transfer id: tlt_iva_classification_resnet18_v1 Download status: Completed.
Downloaded local path: /workspace/tlt-experiments/pretrained_resnet18/
tlt_resnet18_classification_v1
Total files downloaded: 2 
Total downloaded size: 82.41 MB
Started at: 2019-07-16 01:29:53.028400
Completed at: 2019-07-16 01:30:02.053016
Duration taken: 9s seconds

Classification expects a directory of images with the following structure, where each class has its own directory with the class name. The naming convention for train/val/test can be different, because the path of each set is individually specified in the spec file. See Specification file for classification for more information.

|--dataset_root:
    |--train
        |--audi:
            |--1.jpg
            |--2.jpg
        |--bmw:
            |--01.jpg
            |--02.jpg
    |--val
        |--audi:
            |--3.jpg
            |--4.jpg
        |--bmw:
            |--03.jpg
            |--04.jpg
    |--test
        |--audi:
            |--5.jpg
            |--6.jpg
        |--bmw:
            |--05.jpg
            |--06.jpg

The object detection apps in TLT expect data in KITTI file format. For DetectNet_v2 and SSD, this data is converted to TFRecords for training. TFRecords help iterate faster through the data. The steps to convert the data for TFRecords are covered in Conversion to TFRecords. For FasterRCNN, the KITTI format data may be ingested directly, and more on this is covered in Specification file for FasterRCNN.

The SSD and DetectNet_v2 apps, as mentioned in Data input for object detection, require KITTI format data to be converted to TFRecords. To do so, the Transfer Learning Toolkit includes the tlt-dataset-convert tool. This tool requires a configuration file as input. Configuration file details and sample usage examples are included in the following sections.

Here is an example of a specification file for model classification.

model_config {

  # Model architecture can be chosen from:
  # ['resnet', 'vgg', 'googlenet', 'alexnet', 'mobilenet_v1', 'mobilenet_v2', 'squeezenet']

  arch: "resnet"

  # for resnet --> n_layers can be [10, 18, 50]
  # for vgg --> n_layers can be [16, 19]

  n_layers: 18
  use_bias: True
  use_batch_norm: True
  all_projections: True
  use_pooling: False
  freeze_bn: False
  freeze_blocks: 0
  freeze_blocks: 1

  # image size should be "3, X, Y", where X,Y >= 16
  input_image_size: "3,224,224"
}

eval_config {
  eval_dataset_path: "/path/to/your/eval/data"
  model_path: "/path/to/your/model"
  top_k: 3
  conf_threshold: 0.5
  batch_size: 256
  n_workers: 8

}

train_config {
  train_dataset_path: "/path/to/your/train/data"
  val_dataset_path: "/path/to/your/val/data"
  pretrained_model_path: "/path/to/your/pretrained/model"
  # optimizer can be chosen from ['adam', 'sgd']

  optimizer: "sgd"
  batch_size_per_gpu: 256
  n_epochs: 80
  n_workers: 16

  # regularizer
  reg_config {
    type: "L2"
    scope: "Conv2D,Dense"
    weight_decay: 0.00005

  }

  # learning_rate

  lr_config {

    # "step" and "soft_anneal" are supported.

    scheduler: "soft_anneal"

    # "soft_anneal" stands for soft annealing learning rate scheduler.
    # the following 4 parameters should be specified if "soft_anneal" is used.
    learning_rate: 0.005
    soft_start: 0.056
    annealing_points: "0.3, 0.6, 0.8"
    annealing_divider: 10
    # "step" stands for step learning rate scheduler.
    # the following 3 parameters should be specified if "step" is used.
    # learning_rate: 0.006
    # step_size: 10
    # gamma: 0.1
  }
}

To do training, evaluation and inference for DetectNet_v2, several components need to be configured, each with their own parameters. The tlt-train and tlt-evaluate commands for a DetectNet_v2 experiment share the same configuration file. The tlt-infer command uses a separate configuration file.

Here's a sample of the FasterRCNN spec file:

random_seed: 42
enc_key: "<your_enc_key>"
verbose: True
network_config {
input_image_config {
image_type: RGB
image_channel_order: 'bgr'
    size_min {
min:600
}
    image_channel_mean {
        key: 'b'
        value: 103.939
}
    image_channel_mean {
        key: 'g'
        value: 116.779
}
    image_channel_mean {
        key: 'r'
        value: 123.68
}
    image_scaling_factor: 1.0
}
feature_extractor: "vgg"
anchor_box_config {
scale: 128.0
scale: 256.0
scale: 512.0
ratio: 1.0
ratio: 0.5
ratio: 2.0
}
freeze_bn: True
freeze_blocks: 1
freeze_blocks: 2
roi_mini_batch: 256
rpn_stride: 16
conv_bn_share_bias: True
roi_pooling_config {
pool_size: 7
pool_size_2x: True
}
}
training_config {
kitti_data_config {
images_dir: '/workspace/tlt-experiments/data/voc0712trainval/images'
labels_dir: '/workspace/tlt-experiments/data/voc0712trainval/labels_kitti'
}
training_data_parser: 'raw_kitti'
data_augmentation {
use_augmentation: True
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 0
translate_max_y: 0
}
color_augmentation {
color_shift_stddev: 0.0
hue_rotation_max: 0.0
saturation_shift_max: 0.0
contrast_scale_max: 0.0
contrast_center: 0.5
}
}
num_epochs: 12
class_mapping {
key: 'horse'
value: 0
}
class_mapping {
key: "pottedplant"
value: 1
}
class_mapping {
key: "train"
value: 2
}
class_mapping {
key: "person"
value: 3
}
class_mapping {
key: "bird"
value: 4
}
class_mapping {
key: "car"
value: 5
}
class_mapping {
key: "chair"
value: 6
}
class_mapping {
key: "tvmonitor"
value: 7
}
class_mapping {
key: "bus"
value: 8
}
class_mapping {
key: "sofa"
value: 9
}
class_mapping {
key: "dog"
value: 10
}
class_mapping {
key: "motorbike"
value: 11
}
class_mapping {
key: "bicycle"
value: 12
}
class_mapping {
key: "sheep"
value: 13
}
class_mapping {
key: "boat"
value: 14
}
class_mapping {
key: "cat"
value: 15
}
class_mapping {
key: "bottle"
value: 16
}
class_mapping {
key: "diningtable"
value: 17
}
class_mapping {
key: "cow"
value: 18
}
class_mapping {
key: "aeroplane"
value: 19
}
class_mapping {
key: "background"
value: 20
}

pretrained_model: ""
pretrained_weights: "/workspace/tlt-experiments/data/vgg16_weights_tf_dim_ordering_tf_kernels.h5"
output_weights: "/workspace/tlt-experiments/faster_rcnn_exp/faster_rcnn_pascal_voc.tltw"
output_model: "/workspace/tlt-experiments/faster_rcnn_exp/faster_rcnn_pascal_voc.tlt"
rpn_min_overlap: 0.3
rpn_max_overlap: 0.7
classifier_min_overlap: 0.0
classifier_max_overlap: 0.5
gt_as_roi: False
std_scaling: 1.0
classifier_regr_std {
key: 'x'
value: 10.0
}
classifier_regr_std {
key: 'y'
value: 10.0
}
classifier_regr_std {
key: 'w'
value: 5.0
}
classifier_regr_std {
key: 'h'
value: 5.0
}

rpn_mini_batch: 256
rpn_pre_nms_top_N: 12000
rpn_nms_max_boxes: 2000
rpn_nms_overlap_threshold: 0.7
reg_config {
reg_type: 'L2'
weight_decay: 1e-4
}

optimizer {
adam {
lr: 0.00001
beta_1: 0.9
beta_2: 0.999
decay: 0.0
}
}

lr_scheduler {
step {
base_lr: 0.00001
gamma: 1.0
step_size: 30
}
}

lambda_rpn_regr: 1.0
lambda_rpn_class: 1.0
lambda_cls_regr: 1.0
lambda_cls_class: 1.0

inference_config {
images_dir: '/workspace/tlt-experiments/data/voc07test/images'
model: '/workspace/tlt-experiments/faster_rcnn_exp/faster_rcnn_pascal_voc.epoch12.tlt'
detection_image_output_dir: '/workspace/tlt-experiments/faster_rcnn_exp/infer_results_imgs'
labels_dump_dir: '/workspace/tlt-experiments/faster_rcnn_exp/infer_dump_labels'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
bbox_visualize_threshold: 0.6
classifier_nms_max_boxes: 300
classifier_nms_overlap_threshold: 0.3
}
evaluation_config {
dataset {
images_dir : '/workspace/tlt-experiments/data/voc07test/images'
labels_dir: '/workspace/tlt-experiments/data/voc07test/labels_kitti'
}
data_parser: 'raw_kitti'
model: '/workspace/tlt-experiments/faster_rcnn_exp/faster_rcnn_pascal_voc.epoch12.tlt'
labels_dump_dir: '/workspace/tlt-experiments/faster_rcnn_exp/eval_dump_labels'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
classifier_nms_max_boxes: 300
classifier_nms_overlap_threshold: 0.3
object_confidence_thres: 0.0001
use_voc07_11point_metric:True
}
}

network_config {
input_image_config {
image_type: RGB
image_channel_order: 'bgr'
    size_min {
min:600
}
    image_channel_mean {
        key: 'b'
        value: 103.939
}
    image_channel_mean {
        key: 'g'
        value: 116.779
}
    image_channel_mean {
        key: 'r'
        value: 123.68
}
    image_scaling_factor: 1.0
}
feature_extractor: "vgg"
anchor_box_config {
scale: 128.0
scale: 256.0
scale: 512.0
ratio: 1.0
ratio: 0.5
ratio: 2.0
}
freeze_bn: True
freeze_blocks: 1
freeze_blocks: 2
roi_mini_batch: 256
rpn_stride: 16
conv_bn_share_bias: True
roi_pooling_config {
pool_size: 7
pool_size_2x: True
}
}

training_config {
kitti_data_config {
images_dir : '<path_to_the_training_images_directory>'
labels_dir: '<path_to_the_training_KITTI_labels_directory>'
}
training_data_parser: 'raw_kitti'
data_augmentation {
use_augmentation: True
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 0
translate_max_y: 0
}
color_augmentation {
color_shift_stddev: 0.0
hue_rotation_max: 0.0
saturation_shift_max: 0.0
contrast_scale_max: 0.0
contrast_center: 0.5
}
}
num_epochs: 12
class_mapping {
key: 'Car'
value: 0
}
class_mapping {
key: 'Van'
value: 0
}
class_mapping {
key: "Pedestrian"
value: 1
}
class_mapping {
key: "Person_sitting"
value: 1
}
class_mapping {
key: 'Cyclist'
value: 2
}
class_mapping {
key: "background"
value: 3
}
class_mapping {
key: "DontCare"
value: -1
}
class_mapping {
key: "Truck"
value: -1
}
class_mapping {
key: "Misc"
value: -1
}
class_mapping {
key: "Tram"
value: -1
}
pretrained_model: "<path_to_the_pretrained_model>"
pretrained_weights: "<path_to_the_pretrained_weights>"
output_weights: "<path_to_the_output_weights_during_training>"
output_model: "<path_to_the_output_model_during_training>"
rpn_min_overlap: 0.3
rpn_max_overlap: 0.7
classifier_min_overlap: 0.0
classifier_max_overlap: 0.5
gt_as_roi: False
std_scaling: 1.0
classifier_regr_std {
key: 'x'
value: 10.0
}
classifier_regr_std {
key: 'y'
value: 10.0
}
classifier_regr_std {
key: 'w'
value: 5.0
}
classifier_regr_std {
key: 'h'
value: 5.0
}


rpn_mini_batch: 256
rpn_pre_nms_top_N: 12000
rpn_nms_max_boxes: 2000
rpn_nms_overlap_threshold: 0.7


reg_config {
reg_type: 'L2'
weight_decay: 1e-4
}

 

optimizer {
adam {
lr: 0.00001
beta_1: 0.9
beta_2: 0.999
decay: 0.0

}
}
lr_scheduler {
step {
base_lr: 0.00001
gamma: 1.0
step_size: 30
}
}

lambda_rpn_regr: 1.0
lambda_rpn_class: 1.0
lambda_cls_regr: 1.0
lambda_cls_class: 1.0

inference_config {
images_dir: '<path_to_the_inference_images_directory>'
model: '<path_to_the_model_to_do_inference_on>'
detection_image_output_dir: '<path_to_the_dumped_images_directory>'
labels_dump_dir: '<path_to_the_dumped_labels_directory>'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
bbox_visualize_threshold: 0.6
classifier_nms_max_boxes: 300
classifier_nms_overlap_threshold: 0.3
}

evaluation_config {
dataset {
images_dir : '<path_to_the_evaluation_images_directory>'
labels_dir: '<path_to_the_evaluation_KITTI_labels_directory>'
}
data_parser: 'raw_kitti'
model: '<path_to_the_model_to_do_evaluation_on>'
labels_dump_dir: '<path_to_the_dumped_labels_directory>'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
classifier_nms_max_boxes: 300
classifier_nms_overlap_threshold: 0.3
object_confidence_thres: 0.0001
use_voc07_11point_metric:False
}

}

For SSD, both training and evaluation require a specification file.

SSD config

ssd_config {
  aspect_ratios_global: "[1.0, 2.0, 0.5, 3.0, 0.33]"
  scales: "[0.1, 0.24166667, 0.38333333, 0.525, 0.66666667, 0.80833333, 0.95]"
  two_boxes_for_ar1: true
  clip_boxes: false
  loss_loc_weight: 1.0
  focal_loss_alpha: 0.25
  focal_loss_gamma: 2.0
  variances: "[0.1, 0.1, 0.2, 0.2]"
  arch: "resnet18"
  freeze_bn: True
  freeze_blocks: 0
  freeze_blocks: 1
}

aspect_ratios_global or aspect_ratios

Note: Only one of aspect_ratios_global or aspect_ratios is required.

aspect_ratios_global should be a 1-d array inside quotation marks. Anchor boxes of aspect ratios defined in aspect_ratios_global will be generated for each feature layer used for prediction. Example: "[1.0, 2.0, 0.5, 3.0, 0.33]"

aspect_ratios should be a list of lists inside quotation marks. The length of the outer list must be equivalent to the number of feature layers used for anchor box generation. And the i-th layer will have anchor boxes with aspect ratios defined in aspect_ratios[i]. Here's an example: "[[1.0,2.0,0.5], [1.0,2.0,0.5], [1.0,2.0,0.5], [1.0,2.0,0.5], [1.0,2.0,0.5], [1.0, 2.0, 0.5, 3.0, 0.33]]"

two_boxes_for_ar1

This setting is only relevant for layers that have 1.0 as aspect ratio. If two_boxes_for_ar1 is true, two boxes with be generated for aspect ratio 1. One whose scale is the scale for this layer and the other one whose scale is the geometric mean of the scale for this layer and the scale for the next layer.

Scales or combination of min_scale and max_scale

Note: Only one of scales and the combination of min_scale and max_scale is required.

Scales should be a 1-d array inside quotation marks. It is a list of positive floats containing scaling factors per convolutional predictor layer. This list must be one element longer than the number of predictor layers, so if two_boxes_for_ar1 is true, the second aspect ratio 1.0 box for the last layer can have a proper scale. Except for the last element in this list, each positive float is the scaling factor for boxes in that layer. For example, if for one layer the scale is 0.1, then the generated anchor box with aspect ratio 1 for that layer (the first aspect ratio 1 box if two_boxes_for_ar1 is true) will have its height and width as 0.1*min(img_h, img_w).

min_scale and max_scale are two positive floats. If both of them appear in the config, the program can automatically generate the scales by evenly splitting the space between min_scale and max_scale.

clip_boxes

If true, all corner anchor boxes will be truncated so they are fully inside the feature images.

loss_loc_weight

This is a positive float controlling how much location regression loss should contribute to the final loss. The final loss is calculated as classification_loss + loss_loc_weight * loc_loss

focal_loss_alpha and focal_loss_gamma

Focal loss is calculated as:

focal_loss_alpha defines α and focal_loss_gamma defines γ in the formula. NVIDIA recommends α=0.25 and γ=2.0 if you don't know what values to use.

variances

Variances should be a list of 4 positive floats. The four floats, in order, represent variances for box center x, box center y, log box height, log box width. The box offset for box center (cx, cy) and log box size (height/width) w.r.t. anchor will be divided by their respective variance value. Therefore, larger variances result in less significant differences between two different boxes on encoded offsets. The formula for offset calculation is:

steps

An optional list inside quotation marks whose length is the number of feature layers for prediction. The elements should be floats or tuples/lists of two floats. Steps define how many pixels apart the anchor box center points should be. If element is a float, both vertical and horizontal margin is the same. Otherwise, the first value is step_vertical and the second value is step_horizontal. If steps are not provided, anchorboxes will be distributed uniformly inside the image.

offsets

An optional list of floats inside quotation marks whose length is the number of feature layers for prediction. The first anchor box will have offsets[i]*steps[i] pixels margin from the left and top borders. If offsets are not provided, 0.5 will be used as default value.

arch

A string indicating which feature extraction architecture you want to use. Currently, "resnet10" and "resnet18" are supported.

freeze_bn

Whether to freeze all batch normalization layers during training.

freeze_blocks

Optionally, you can have more than 1 freeze_blocks field. Weights of layers in those blocks will be freezed during training. See Model config for more information.

training_config {
  batch_size_per_gpu: 18
  num_epochs: 120
  learning_rate {
    soft_start_annealing_schedule {
      min_learning_rate: 5e-5
      max_learning_rate: 4e-2
      soft_start: 0.01
      annealing: 0.3
    }
  }
  regularizer {
    type: L1
    weight: 3.00000002618e-09
  }
}

eval_config {
  validation_period_during_training: 10
  averge_precision_mode: SAMPLE
  matching_iou_threshold: 0.5
}

nms_config {
 confidence_threshold: 0.05
 clustering_iou_threshold: 0.5
 top_k: 200
}

augmentation_config {
  preprocessing {
    output_image_width: 1024
    output_image_height: 256
    crop_right: 1024
    crop_bottom: 256
    min_bbox_width: 1.0
    min_bbox_height: 1.0
  }
  spatial_augmentation {
    hflip_probability: 0.5
    vflip_probability: 0.0
    zoom_min: 0.7
    zoom_max: 1.8
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298
    contrast_scale_max: 0.10000000149
    contrast_center: 0.5
  }
}

dataset_config {
  data_sources: {
    tfrecords_path: "/path/to/tfrecords/root/*"
    image_directory_path: "/path/to/dataset/root"
  }
  image_extension: "png"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_fold: 0
}

Use the tlt-train command to tune a pre-trained model:

tlt-train [-h] classification --gpus <num GPUs>
           -k <encoding key>
           -r <result directory>
           -e <spec file>
           

Required arguments:

• -r, --results_dir : Path to a folder where the experiment outputs should be written.
• -k, --key : User specific encoding key to save or load a .tlt model.
• -e, --experiment_spec_file: Path to the experiment spec file.

Optional arguments:

• --gpus : Number of GPUs to use and processes to launch for training. The default value is 1.

Here's an example of using the tlt-train command:

tlt-train classification -e /workspace/tlt_drive/spec/spec.cfg -r /workspace/output -k $YOUR_KEY

Output Log

Here's the output log from the successful use of this command:

Using TensorFlow backend.
..
_____________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
=============================================================================
input_1 (InputLayer)            (None, 3, 224, 224)  0                                           
..
..
..
________________________________________________________________________________
predictions (Dense)             (None, 20)           10260       flatten_1[0][0]                  
================================================================================
Total params: 11,558,548
Trainable params: 11,546,900
Non-trainable params: 11,648
________________________________________________________________________________

Epoch 1/80
124/311 [==========>...................] - ETA: 49s - loss: 4.1188 - acc: 0.06592018-10-11 22:09:13.292358: W tensorflow/core/framework/allocator.cc:101] Allocation of 38535168 exceeds 10% of system memory.

After following the steps, go here to create TFRecords ingestible by the TLT training, and setting up a spec file. You are now ready to start training an object detection network.

DetectNet_v2 training command

tlt-train [-h] detectnet_v2 
           -k <key>
           -r <result directory>
           -e <spec_file>
           [--gpus <num GPUs>]
           

Required arguments

• -r, --results_dir : Path to a folder where experiment outputs should be written.
• -k, –key : User specific encoding key to save or load a .tlt model.
• -e, --experiment_spec_file : Path to spec file. Absolute path or relative to working directory. (default: spec from spec_loader.py is used).

Optional arguments

• --gpus : Number of GPUs to use and processes to launch for training. The default value is 1.
• -h, --help : To print help message

Sample usage

Here is an example of command for a 2 GPU training:

tlt-train detectnet_v2 -e <path_to_spec_file> 
                                 -r <path_to_experiment_output> 
                                 -k <key_to_load_the_model> 
                                 -n <name_string_for_the_model> 
                                 --gpus 2

Output log

Here's an example of the output log:

Using TensorFlow backend.
2018-11-06 01:03:16.402006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
..
..
_______________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                    
===============================================================================
input_1 (InputLayer)            (None, 3, 544, 960)  0                                                          
..
===============================================================================
Total params: 11,555,983
Trainable params: 11,544,335
Non-trainable params: 11,648
..
..
2018-11-06 01:04:06,173 [INFO] tensorflow: Running local_init_op.
..
INFO:tensorflow:loss = 0.07203477, epoch = 0.0, step = 0
2018-11-06 01:05:14,270 [INFO] tensorflow: loss = 0.07203477, epoch = 0.0, step = 0
INFO:tensorflow:Saving checkpoints for step-1.
..
2018-11-06 01:05:44,920 [INFO] tensorflow: loss = 0.05362146, epoch = 0.0663716814159292, step = 15 (5.978 sec)
INFO:tensorflow:global_step/sec: 0.555544
..
Validation cost: 0.000268
Mean average_precision (in %): 73.9490

class name      average precision (in %)
------------  --------------------------
person                         83.5255
bag                            54.1475
face                           84.1741

Use this command to execute the FasterRCNN training command:

tlt-train [-h] faster_rcnn -e <experiment_spec>

Required arguments:

• -e, --experiment_spec_file : Experiment specification file to set up the evaluation experiment. This should be the same as training specification file.

Optional arguments:

• -h, --help : Show this help message and exit.

Sample usage

Here's an example of using the FasterRCNN training command:

tlt-train faster_rcnn -e <experiment_spec>

Here's a sample output log:

Using TensorFlow backend.
2019-07-04 08:43:12.677469: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-04 08:43:12.970675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:85:00.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-07-04 08:43:12.970727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-04 08:43:13.542863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-04 08:43:13.542924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-07-04 08:43:13.542933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-07-04 08:43:13.543743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14935 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0)
2019-07-04 08:43:13,555 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/train.py: valid_class_mapping: {u'Cyclist': 2, u'Car': 0, u'background': 3, u'Pedestrian': 1}
WARNING:tensorflow:From /app/iva/common/py_image.binary.runfiles/pip_deps2__tensorflow_gpu_1_13_1/extracted/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-07-04 08:43:13,563 [WARNING] tensorflow: From /app/iva/common/py_image.binary.runfiles/pip_deps2__tensorflow_gpu_1_13_1/extracted/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-07-04 08:43:14,284 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/train.py: Base featuremap: activation_13/Relu:0
________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
================================================================================
input_1 (InputLayer)            (None, 3, 384, 1280) 0                                            
________________________________________________________________________________

..
________________________________________________________________________________
add_7 (Add)                     (256, 512, 7, 7)     0           block_4a_bn_2[0][0]              
                                                                 block_4a_bn_shortcut[0][0]       
________________________________________________________________________________
activation_15 (Activation)      (256, 512, 7, 7)     0           add_7[0][0]                      
________________________________________________________________________________
block_4b_conv_1 (Conv2D)        (256, 512, 7, 7)     2359808     activation_15[0][0]              
________________________________________________________________________________
block_4b_bn_1 (BatchNormalizati (256, 512, 7, 7)     2048        block_4b_conv_1[0][0]            
________________________________________________________________________________
activation_16 (Activation)      (256, 512, 7, 7)     0           block_4b_bn_1[0][0]              
________________________________________________________________________________
block_4b_conv_2 (Conv2D)        (256, 512, 7, 7)     2359808     activation_16[0][0]              
________________________________________________________________________________
block_4b_conv_shortcut (Conv2D) (256, 512, 7, 7)     262656      activation_15[0][0]              
________________________________________________________________________________
block_4b_bn_2 (BatchNormalizati (256, 512, 7, 7)     2048        block_4b_conv_2[0][0]            
________________________________________________________________________________
block_4b_bn_shortcut (BatchNorm (256, 512, 7, 7)     2048        block_4b_conv_shortcut[0][0]     
________________________________________________________________________________
add_8 (Add)                     (256, 512, 7, 7)     0           block_4b_bn_2[0][0]              
                                                                 block_4b_bn_shortcut[0][0]       
________________________________________________________________________________
2019-07-04 08:43:14,937 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/train.py: training example num: 6481
2019-07-04 08:43:15,579 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/train.py: Starting training
2019-07-04 08:43:15,579 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/train.py: Epoch 1/12

tlt-train [-h] ssd -e <experiment_spec> 
                   -r <output_dir> 
                   -k <key> 
                   -m <pretrained_model>
                   --gpus <num_gpus>

Required arguments:

• -r, --results_dir: Path to the folder where the experiment output is written.
• -k, --key: Provide the encryption key to decrypt the model.
• -e, --experiment_spec_file: Experiment specification file to set up the evaluation experiment. This should be the same as training specification file.

Optional arguments:

• --gpus num_gpus: Number of GPUs to use and processes to launch for training. The default = 1.
• -m, --resume_model_weights: Path to a pre-trained model or model to continue training.
• --initial_epoch: Epoch number to resume from.
• -h, --help: Show this help message and exit.

tlt-train ssd --gpus 2 -e /path/to/spec.txt -r /path/to/result -k $KEY

Here's a sample output log:

Using TensorFlow backend.
2019-07-08 17:36:56.866657: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-08 17:36:56.866840: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-08 17:36:57.259900: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x65046d0 executing computations on platform CUDA. Devices:
2019-07-08 17:36:57.259958: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): TITAN Xp, Compute Capability 6.1
2019-07-08 17:36:57.259975: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (1): TITAN Xp, Compute Capability 6.1
2019-07-08 17:36:57.264088: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3298305000 Hz
2019-07-08 17:36:57.264882: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x656e670 executing computations on platform Host. Devices:
2019-07-08 17:36:57.264916: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-08 17:36:57.265106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 10.81GiB
2019-07-08 17:36:57.265131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-08 17:36:57.269875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-08 17:36:57.269894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-07-08 17:36:57.269903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-07-08 17:36:57.269991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10515 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:01:00.0, compute capability: 6.1)]            
_______________________________________________________________________________
activation_2 (Activation)       (18, 64, 64, 256)    0           block_1a_bn_1[0][0]              
_______________________________________________________________________________
block_1a_conv_2 (Conv2D)        (18, 64, 64, 256)    36928       activation_2[0][0]               
_______________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (18, 64, 64, 256)    4160        activation_1[0][0]               
_______________________________________________________________________________
block_1a_bn_2 (BatchNormalizati (18, 64, 64, 256)    256         block_1a_conv_2[0][0]            
_______________________________________________________________________________
block_1a_bn_shortcut (BatchNorm (18, 64, 64, 256)    256         block_1a_conv_shortcut[0][0]     
_______________________________________________________________________________
add_1 (Add)                     (18, 64, 64, 256)    0           block_1a_bn_2[0][0]              
                                                                 block_1a_bn_shortcut[0][0]       
...
...
_______________________________________________________________________________
conf_reshape_0 (Reshape)        (18, 24576, 1, 3)    0           permute_1[0][0]                  
_______________________________________________________________________________
conf_reshape_1 (Reshape)        (18, 6144, 1, 3)     0           permute_3[0][0]                  
_______________________________________________________________________________
conf_reshape_2 (Reshape)        (18, 1536, 1, 3)     0           permute_5[0][0]                  
_______________________________________________________________________________
conf_reshape_3 (Reshape)        (18, 384, 1, 3)      0           permute_7[0][0]                  
_______________________________________________________________________________
conf_reshape_4 (Reshape)        (18, 96, 1, 3)       0           permute_9[0][0]                  
_______________________________________________________________________________
conf_reshape_5 (Reshape)        (18, 24, 1, 3)       0           permute_11[0][0]                 
_______________________________________________________________________________
..

ssd_predictions (Reshape)       (18, 32760, 15)      0           concatenate_1[0][0]              
================================================================================
Total params: 18,866,812
Trainable params: 18,852,092
Non-trainable params: 14,720
________________________________________________________________________________
2019-07-08 17:37:30,754 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/train.pyc: Number of images in the training dataset:    6142
2019-07-08 17:37:30,754 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/train.pyc: Number of images in the validation dataset:      1339

Epoch 1/120
171/171 [======================================================] - 94s 547ms/step - loss: 2.3210
...
Number of images in the evaluation dataset: 1339
()
Producing predictions batch-wise: 100% 75/75 [00:36<00:00,  2.57it/s]
Matching predictions to ground truth, class 1/3.: 100% 131693/131693 [00:10<00:00, 12953.23it/s]
Matching predictions to ground truth, class 2/3.: 100% 15162/15162 [00:00<00:00, 26290.28it/s]
Matching predictions to ground truth, class 3/3.: 100% 36838/36838 [00:01<00:00, 19611.29it/s]
Computing precisions and recalls, class 1/3
Computing precisions and recalls, class 2/3
Computing precisions and recalls, class 3/3
Computing average precision, class 1/3
Computing average precision, class 2/3
Computing average precision, class 3/3
2019-07-08 17:55:12,060 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/train.pyc: car           AP    0.815
2019-07-08 17:55:12,060 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/train.pyc: cyclist       AP    0.136
2019-07-08 17:55:12,061 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/train.pyc: pedestrian    AP    0.433
2019-07-08 17:55:12,061 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/train.pyc:               mAP   0.462

Execute tlt-evaluate on a classification model.

tlt-evaluate classification [-h] -e <experiment_spec_file> -k <key>

Required arguments

• -e, --experiment_spec_file: Path to the experiment spec file..
• -k, –key : Provide the encryption key to decrypt the model .

Optional arguments

• -h, --help : show this help message and exit.

If you followed the example in Training a classification model, you can run the evaluation:

tlt-evaluate classification -e classification_spec.cfg -k $YOUR_KEY

The resulting log file will be similar to this:

Using TensorFlow backend.
..
..
______________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==============================================================================
input_1 (InputLayer)            (None, 3, 224, 224)  0                                           
______________________________________________________________________________
conv1 (Conv2D)                  (None, 64, 112, 112) 9472        input_1[0][0]                    
______________________________________________________________________________
..
..
..
predictions (Dense)             (None, 20)           10260       flatten[0][0]                    
===============================================================================
Total params: 11,558,548
Trainable params: 11,546,900
Non-trainable params: 11,648
_______________________________________________________________________________

Found 3345 images belonging to 20 classes.
..
..
Evaluation Loss: 1.67691540718
Evaluation Top K accuracy: 0.828101634979
Evaluation precision: 0.616197168827
Evaluation recall: 0.366816133261

Execute tlt-evaluate on a DetectNet_v2 model.

tlt-evaluate detectnet_v2 [-h] -e <experiment_spec> 
                               -m <model_file> 
                               -k <key> 
                               [--use_training_set] 
                               

Required arguments:

• -e, --experiment_spec_file: Experiment spec file to set up the evaluation experiment. This should be the same as training spec file.
• -m, --model: Path to the model file to use for evaluation.
• -k, -–key : Provide the encryption key to decrypt the model.

Optional arguments

• -h, --help : show this help message and exit.
• --use_training_set: Set this flag to run evaluation on training + validation dataset.

If you have followed the example in Training a detection model, you may now evaluate the model using the following command.

tlt-evaluate detectnet_v2 -e <path to training spec file>
                          -m <path to the model> 
                          -k <key to load the model>

Use these steps to evaluate on a test set with ground truth labeled:

1. Create tfrecords for this training set by following the steps listed in the data input section.
2. Update the dataloader configuration part of the training spec file to include the newly generated tfrecords. For more information on the dataset config, please refer to Create an experiment spec file.

dataset_config {
  data_sources: {
    tfrecords_path: "<path to training tfrecords root>/<tfrecords_name*>"
    image_directory_path: "<path to training data root>"
  }
  image_extension: "jpg"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "automobile"
      value: "car"
  }
  ..
  ..
  ..
  target_class_mapping {
      key: "person"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "rider"
      value: "cyclist"
  }
  validation_data_source: {
    tfrecords_path: "<path to testing tfrecords root>/<tfrecords_name*>"
    image_directory_path: "<path to testing data root>"
  }
}

The rest of the experiment spec file remains the same as the training spec file.

Sample output log

Here's an example of the output:

Using TensorFlow backend.
..
..
packages/iva/detectnet_v2/evaluation/build_evaluator.pyc: Found 1802 samples in validation set
_______________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
===============================================================================
input_1 (InputLayer)            (None, 3, 544, 960)  0                                           
_______________________________________________________________________________
conv1 (Conv2D)                  (None, 64, 272, 480) 9472        input_1[0][0]                    
_______________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 64, 272, 480) 256         conv1[0][0]                      
_______________________________________________________________________________
activation_1 (Activation)       (None, 64, 272, 480) 0           bn_conv1[0][0]                   
_______________________________________________________________________________
..
..
________________________________________________________________________________
activation_17 (Activation)      (None, 512, 34, 60)  0               add_8[0][0]                      
________________________________________________________________________________
dropout_1 (Dropout)             (None, 512, 34, 60)  0       activation_17[0][0]              
________________________________________________________________________________
output_bbox (Conv2D)            (None, 12, 34, 60)   6156        dropout_1[0][0]                  
________________________________________________________________________________
output_cov (Conv2D)             (None, 3, 34, 60)    1539        dropout_1[0][0]                  
================================================================================
Total params: 11,555,983
Trainable params: 11,544,335
Non-trainable params: 11,648
________________________________________________________________________________

INFO:tensorflow:Graph was finalized.
2018-10-22 19:55:24,136 [INFO] tensorflow: Graph was finalized.
..
..
Validation cost: 0.000268
Mean average_precision (in %): 73.9490

class name      average precision (in %)
------------  --------------------------
person                         83.5255
bag                            54.1475
face                           84.1741
Time taken to run /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/scripts/train.pyc:main: 0:45:45.071311.

To run evaluation for a faster_rcnn model use this command:

tlt-evaluate faster_rcnn [-h] -e <experiment_spec>

Here's a sample output log:

Using TensorFlow backend.

2019-05-29 07:59:14.442525: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-29 07:59:14.687355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 

name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:06:00.0
totalMemory: 15.75GiB freeMemory: 15.44GiB

2019-05-29 07:59:14.687423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-29 07:59:15.241007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-29 07:59:15.241067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-05-29 07:59:15.241075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-05-29 07:59:15.242055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14935 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 7.0)
2019-05-29 07:59:15,261 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: {0: u'Car', 1: u'Pedestrian', 2: u'Cyclist', 3: u'background'}

2019-05-29 07:59:15,262 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Loading kpi test model...

WARNING:tensorflow:From /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/pip_deps2__tensorflow_gpu_1_13_1/extracted/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:

Colocations handled automatically by placer.

2019-05-29 07:59:15,330 [WARNING] tensorflow: From /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/pip_deps2__tensorflow_gpu_1_13_1/extracted/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:

Colocations handled automatically by placer.

2019-05-29 07:59:17,649 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Done!
2019-05-29 07:59:17,748 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: 0/1000
2019-05-29 07:59:24.976197: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-05-29 07:59:27,428 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Elapsed time = 9.67983293533
2019-05-29 07:59:28,407 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: 1/1000
2019-05-29 07:59:28,534 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Elapsed time = 0.126852035522
2019-05-29 07:59:28,615 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: 2/1000
2019-05-29 07:59:28,731 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Elapsed time = 0.116088151932
2019-05-29 07:59:28,794 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: 3/1000
...
...
2019-07-03 02:38:19,946 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: 999/1000
2019-07-03 02:38:20,049 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Elapsed time = 0.103152036667
2019-07-03 02:38:20,053 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Cyclist AP: 0.68731839316, precision: 0.7, recall: 0.72850678733, TP: 161.0, FP: 69.0, FN: 60.0
2019-07-03 02:38:20,072 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Car AP: 0.837039752906, precision: 0.853330184223, recall: 0.847724073205, TP: 3613.0, FP: 621.0, FN: 649.0
2019-07-03 02:38:20,074 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: Pedestrian AP: 0.564051624343, precision: 0.674321503132, recall: 0.606003752345, TP: 323.0, FP: 156.0, FN: 210.0
2019-07-03 02:38:20,075 [INFO] /app/iva/common/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/test.py: mAP = 0.696136590137

To run evaluation for an SSD model use this command:

tlt-evaluate ssd [-h] -e <experiment_spec_file> -m <model_file> -k <key>

Here's a sample output log:

Using TensorFlow backend.
2019-07-23 18:05:23.625666: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
...
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
...
...
2019-07-23 18:06:03,638 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/evaluate.pyc: Number of images in the validation dataset:   2696
Number of images in the evaluation dataset: 2696
()
Producing predictions batch-wise:   0%|                  | 0/22 [00:00<?, ?it/s]2019-07-23 18:06:12.764148: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
Producing predictions batch-wise: 100%|#########| 22/22 [00:22<00:00,  1.88it/s]
Matching predictions to ground truth, class 1/3.: 100%|#| 245/245 [00:00<00:00, 26717.40it/s]
Matching predictions to ground truth, class 2/3.: 100%|#| 25954/25954 [00:00<00:00, 41923.85it/s]
Matching predictions to ground truth, class 3/3.: 100%|#| 120686/120686 [00:06<00:00, 19488.45it/s]
Computing precisions and recalls, class 1/3
Computing precisions and recalls, class 2/3
Computing precisions and recalls, class 3/3
Computing average precision, class 1/3
Computing average precision, class 2/3
Computing average precision, class 3/3
2019-07-23 18:06:36,688 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/evaluate.pyc: bicycle       AP    0.001
2019-07-23 18:06:36,688 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/evaluate.pyc: car           AP    0.0
2019-07-23 18:06:36,689 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/evaluate.pyc: person        AP    0.07
2019-07-23 18:06:36,689 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/scripts/evaluate.pyc:               mAP   0.02

Execute tlt-infer on a classification model trained on the Transfer Learning Toolkit.

tlt-infer classification [-h] 
                          -m <model> 
                          -i <image> 
                          -d <image  dir>
                         [-b <batch size>] 
                          -k <key> 
                          -cm <classmap>

Here are the parameters of the tlt-infer tool:

Required arguments

• -m, --model : Path to the pretrained model (TLT model).
• -i, --image : A single image file for inference.
• -d, --image_dir : The directory of input images for inference.
• -k, --key : Key to load model.
• -cm, --class_map : The json file that specifies the class index and label mapping.

Optional arguments

• --batch_size : Inference batch size, default: 1

• -h, --help : show this help message and exit

Sample output using single image mode

Single Image Mode
_____________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
=============================================================================
input_1 (InputLayer)            (None, 3, 224, 224)  0                                           
_____________________________________________________________________________
conv1 (Conv2D)                  (None, 16, 112, 112) 2368        input_1[0][0]                    
_____________________________________________________________________________
...
...            
_____________________________________________________________________________
2018-11-05 18:46:16,248 [INFO] root: Current predictions: [[2.0956191e-04 4.7424308e-08 6.0529976e-07 1.5379728e-05 4.9668059e-05
  2.3047665e-05 8.3990363e-07 2.1063986e-06 3.9042366e-06 9.8465785e-07
  7.9830796e-05 8.4068454e-08 1.3434786e-06 1.6271177e-05 1.1729119e-06
  9.9955863e-01 2.9604094e-05 2.6558594e-06 3.4933796e-06 7.3329272e-07]]
2018-11-05 18:46:16,248 [INFO] root: Class label = 15
2018-11-05 18:46:16,248 [INFO] root: Class name = mercedes
 

Execution using -d or directory mode

A result.csv file is created and stored in the directory you specify using -d. The result.csv has the following format, where the second column shows the file path and third shows the predicted class name.

0,/home/tmp/1.jpg,A
0,/home/tmp/2.jpg,B
0,/home/tmp/3.jpg,C

The tlt-infer tool for object detection networks which may be used to visualize bboxes, or generate frame by frame kitti format labels on a single image or a directory of images. An example of the command for this tool is shown here:

tlt-infer detectnet_v2 [-h] -m <path to model file> -i <path to inference input> -o <path to output directory>
					   -bs <batch size> -cp <path to cluster params file> -k <encryption key>
					   [--kitti_dump] [-lw LINE_WIDTH] 
					   [-g <gpu to run inference>] [--disable_overlay]
					   [--output_nodes <output_cov_blob,output_bbox_blob>] 

Required parameters

• -m, --model: TLT model file path
• -i, --inference_input: The directory of input images or a single image for inference.
• -o, --inference_output: The directory to the output images and labels. The annotated images are in inference_output/images_annotated and labels are in inference_output/labels
• -bs, --batch_size: Inference batch size
• -cp, --cluster_params_file: Bbox post processing json file.
• -lw, --line_width: Overlay linewidth
• -k, --enc_key: Key to load model

Optional parameters

• -g, --gpu_set: GPU index to choose. The default is 0.
  Note: Inference is not a multiple GPU process. This process only allows the user to choose which GPU to run inference on, in case there are multiple GPU's in the machine.
• --output_nodes: Comma separated list of output nodes, default=output_cov,output_bbox
• --kitti_dump: Flag to enable KITTI dump
• --disable_overlay : Flag to disable image overlay

This clusterfile is suitable for use with our uploaded pretrained models in NGC.

The tool automatically generates bbox rendered images in output_path/images_annotated. In order to get the bbox labels in KITTI format, please set the --kitti-dump flag. This will generate the output in output_path/labels.

Here's a sample output log:

Using TensorFlow backend.

2018-11-05 16:56:08.557935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: 

name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582

pciBusID: 0000:02:00.0

..

..

Layer (type)                 Output Shape              Param #   

=================================================================

input_1 (InputLayer)         (None, 3, 384, 1240)      0        

..

..

0it [00:00, ?it/s]

  0%|                                                    | 0/32 [00:00<?, ?it/s]

  3%|█▍                                          | 1/32 [00:00<00:04,  7.50it/s]

  ..

100%|███████████████████████████████████████████| 23/23 [00:03<00:00,  7.18it/s]

1it [00:10, 10.85s/it]

  0%|                                                    | 0/32 [00:00<?, ?it/s]

  3%|█▍                                          | 1/32 [00:00<00:03,  7.92it/s]

  ..

100%|███████████████████████████████████████████| 32/32 [00:04<00:00,  6.87it/s]

2it [00:19,  9.67s/it]

..

..

5it [00:40,  8.07s/it]

2018-11-05 16:56:52,571 [INFO] iva.detectnet_v2.scripts.inference: Inference complete

The tlt-infer tool for FasterRCNN networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. You can execute this tool from the command line as shown here:

tlt-infer faster_rcnn [-h] -e <experiment_spec>

Required arguments:

• -e, --experiment_spec_file: Path to the experiment specification file for FasterRCNN training.

Here's a sample output log:

Using TensorFlow backend.
2019-05-29 08:19:42.667096: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-29 08:19:42.927812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:85:00.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-05-29 08:19:42.927857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-29 08:19:43.446058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-29 08:19:43.446106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-05-29 08:19:43.446114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-05-29 08:19:43.446984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14935 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0)
2019-05-29 08:19:43,459 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: {0: u'Car', 1: u'Pedestrian', 2: u'Cyclist', 3: u'background'}
2019-05-29 08:19:43,460 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Loading test model...
WARNING:tensorflow:From /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/pip_deps2__tensorflow_gpu_1_13_1/extracted/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-05-29 08:19:43,495 [WARNING] tensorflow: From /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/pip_deps2__tensorflow_gpu_1_13_1/extracted/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-05-29 08:19:45,819 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Done!
2019-05-29 08:20:02,271 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: 000008.png
2019-05-29 08:20:09.102160: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-05-29 08:20:09,768 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Elapsed time = 7.49691820145
2019-05-29 08:20:09,798 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Image 000008.png processed.
2019-05-29 08:20:09,798 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: 000010.png
2019-05-29 08:20:09,918 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Elapsed time = 0.120166063309
2019-05-29 08:20:09,946 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Image 000010.png processed.
2019-05-29 08:20:09,946 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: 000012.png
2019-05-29 08:20:10,082 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Elapsed time = 0.13534784317
2019-05-29 08:20:10,111 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: Image 000012.png processed.
2019-05-29 08:20:10,111 [INFO] /app/iva/faster_rcnn/launcher/py_image.binary.runfiles/ai_infra/iva/faster_rcnn/scripts/inference.py: 000035.png

The tlt-infer tool for SSD networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. Here's an example of using this tool:

tlt-infer ssd  -i <input directory> 
               -o <output annotated image directory> 
               -e <experiment spec file> 
               -m <model file> 
               [-l <output label directory>]
               [-t <visualization threshold>] 
               -k <key>

Required arguments

• -m, --model : Path to the pretrained model (TLT model).
• -i, --in_image_dir : The directory of input images for inference.
• -o, --out_image_dir : The directory path to output annotated images.
• -k, --key : Key to load model.
• -e, --config_path : Path to an experiment spec file for training.

Optional arguments

• -t, --draw_conf_thres : Threshold for drawing a bbox. default: 0.3
• -h, --help : Show this help message and exit
• -l, --out_label_dir : The directory to output KITTI labels.

Here's a sample output log:

Using TensorFlow backend.
2019-05-29 08:19:42.667096: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-29 08:19:42.927812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:85:00.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-05-29 08:19:42.927857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-29 08:19:43.446058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-29 08:19:43.446106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-05-29 08:19:43.446114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-05-29 08:19:43.446984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14935 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0)
...
...
anchor_reshape_5 (Reshape)      (None, 24, 1, 8)     0           ssd_anchor_5[0][0]               
________________________________________________________________________________
mbox_conf_sigmoid (Activation)  (None, 32760, 1, 20) 0           mbox_conf[0][0]                  
________________________________________________________________________________
mbox_loc (Concatenate)          (None, 32760, 1, 4)  0           loc_reshape_0[0][0]              
                                                                 loc_reshape_1[0][0]              
                                                                 loc_reshape_2[0][0]              
                                                                 loc_reshape_3[0][0]              
                                                                 loc_reshape_4[0][0]              
                                                                 loc_reshape_5[0][0]              
________________________________________________________________________________
mbox_priorbox (Concatenate)     (None, 32760, 1, 8)  0           anchor_reshape_0[0][0]           
                                                                 anchor_reshape_1[0][0]           
                                                                 anchor_reshape_2[0][0]           
                                                                 anchor_reshape_3[0][0]           
                                                                 anchor_reshape_4[0][0]           
                                                                 anchor_reshape_5[0][0]           
________________________________________________________________________________
concatenate_3 (Concatenate)     (None, 32760, 1, 32) 0           mbox_conf_sigmoid[0][0]          
                                                                 mbox_loc[0][0]                   
                                                                 mbox_priorbox[0][0]              
________________________________________________________________________________
ssd_predictions (Reshape)       (None, 32760, 32)    0           concatenate_3[0][0]              
================================================================================
Total params: 7,961,848
Trainable params: 7,958,376
Non-trainable params: 3,472
________________________________________________________________________________
WARNING:tensorflow:From ./ssd/box_coder/output_decoder_layer.py:83: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version
Instructions for updating:
Use tf.cast instead.
2019-08-04 00:01:14,444 [WARNING] tensorflow: From ./ssd/box_coder/output_decoder_layer.py:83: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
100%|##########| 4952/4952 [03:35<00:00, 22.99it/s]