Using Inference on a Model ========================== The :code:`tlt-infer` command runs the inference on a specified set of input images. In the classification mode, :code:`tlt-infer` provides class label output over the command line for a single image or a csv file containing the image path and the corresponding labels for multiple images. In DetectNet_v2, SSD, RetinaNet, DSSD, YOLOV3, or FasterRCNN mode, :code:`tlt-infer` produces output images with bounding boxes rendered on them after inference. Optionally, you can also serialize the output meta-data in kitti_format. In MaskRCNN, :code:`tlt-infer` produces annotated images with bounding boxes and masks rendered on them after inference. TensorRT python inference can also be enabled. Running Inference on a Classification Model ------------------------------------------- Execute :code:`tlt-infer` on a classification model trained on the Transfer Learning Toolkit. .. code:: tlt-infer classification [-h] -m -i -d [-b ] -k -cm Here are the parameters of the :code:`tlt-infer` tool: Required arguments ^^^^^^^^^^^^^^^^^^ * :code:`-m, --model`: Path to the pretrained model (TLT model). * :code:`-i, --image`: A single image file for inference. * :code:`-d, --image_dir`: The directory of input images for inference. * :code:`-k, --key`: Key to load model. * :code:`-cm, --class_map`: The json file that specifies the class index and label mapping. Optional arguments ^^^^^^^^^^^^^^^^^^ * :code:`--batch_size`: Inference batch size, default: 1 * :code:`-h, --help`: show this help message and exit .. Note:: The inference tool requires a cluster_params.json file to configure the post processing block. When executing with :code:`-d`, or directory mode, a :code:`result.csv` file will be created and stored in the directory you specify using :code:`-d`. The :code:`result.csv` has the file path in the first column and predicted labels in the second. .. Note:: In both single image and directory modes, a classmap (:code:`-cm`) is required, which should be a byproduct (:code:`-classmap.json`) of your training process. Running Inference on a DetectNet_v2 Model ----------------------------------------- The tlt-infer tool for object detection networks which may be used to visualize bboxes, or generate frame by frame kitti format labels on a single image or a directory of images. An example of the command for this tool is shown here: .. code:: tlt-infer detectnet_v2 [-h] -e \ -i \ -o \ -k Required Parameters ^^^^^^^^^^^^^^^^^^^ * :code:`-e, --inference_spec`: Path to an inference spec file. * :code:`-i, --inference_input`: The directory of input images or a single image for inference. * :code:`-o, --inference_output`: The directory to the output images and labels. The annotated images are in :code:`inference_output/images_annotated` and labels are in :code:`inference_output/labels`. * :code:`-k, --enc_key`: Key to load model The tool automatically generates bbox rendered images in :code:`output_path/images_annotated`. In order to get the bbox labels in KITTI format, please configure the :code:`bbox_handler_config` spec file using the :code:`kitti_dump parameter` as mentioned `here`_. This will generate the output in :code:`output_path/labels`. .. _here: https://docs.nvidia.com Running Inference on a FasterRCNN Model --------------------------------------- The :code:`tlt-infer` tool for FasterRCNN networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. You can execute this tool from the command line as shown here: .. code:: tlt-infer faster_rcnn [-h] -e [-k ] Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-e, --experiment_spec_file`: Path to the experiment specification file for FasterRCNN training. Optional Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-h, --help`: Print help log and exit. * :code:`-k, --enc_key`: The encoding key, can override the one in the spec file. Two Modes for tlt-infer ^^^^^^^^^^^^^^^^^^^^^^^ The :code:`tlt-infer` command line for FasterRCNN has two modes. It can run with either TLT backend or TensorRT backend. This behavior is also controlled via the spec file. The :code:`inference_config` in the spec file can have an optional :code:`trt_inference` sub-field that specifies which backend the :code:`tlt-infer` will run with. By default(if the :code:`trt_inference` sub-field is not present in :code:`inference_config`), tlt-infer will use TLT as the backend. If the :code:`trt_inference` sub-field is present, it can specify :code:`tlt-infer` to run at TensorRT backend. In that case, the model to do inference can be either the :code:`.etlt` model from :code:`tlt-export` or the TensorRT engine file from :code:`tlt-export` or :code:`tlt-converter`. To use a TensorRT engine file for TensorRT backend based :code:`tlt-infer`, the :code:`trt_inference` sub-field should look like this: .. code:: trt_inference { trt_engine: '/workspace/tlt-experiments/data/faster_rcnn/trt.int8.engine' } To use a :code:`.etlt` model for TensorRT backend based :code:`tlt-infer`, the :code:`trt_inference` sub-field should look like this: .. code:: trt_inference { etlt_model { model: '/workspace/tlt-experiments/data/faster_rcnn/resnet18.epoch12.etlt' calibration_cache: '/workspace/tlt-experiments/data/faster_rcnn/cal.bin' } trt_data_type: 'int8' } If the TensorRT inference data type is not INT8, the :code:`calibration_cache` sub-field that provides the path to the INT8 calibration cache is not needed. In INT8 case, the calibration cache should be generated via the :code:`tlt-export` command line in INT8 mode. See also the documentation of FasterRCNN spec file for the details of the :code:`trt_inference` message structure. Running Inference on an SSD Model --------------------------------- The :code:`tlt-infer` tool for SSD networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. Here's an example of using this tool: .. code:: tlt-infer ssd -i -o -e -m [-l ] [-t ] -k Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-m, --model`: Path to the pretrained model (TLT model). * :code:`-i, --in_image_dir`: The directory of input images for inference. * :code:`-o, --out_image_dir`: The directory path to output annotated images. * :code:`-k, --key`: Key to load model. * :code:`-e, --config_path`: Path to an experiment spec file for training. Optional Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-t, --draw_conf_thres`: Threshold for drawing a bbox. default: 0.3 * :code:`-h, --help`: Show this help message and exit * :code:`-l, --out_label_dir`: The directory to output KITTI labels. Running Inference on a DSSD Model --------------------------------- The :code:`tlt-infer` tool for DSSD networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. Here's an example of using this tool: .. code:: tlt-infer dssd -i -o -e -m [-l ] [-t ] -k Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-m, --model`: Path to the pretrained model (TLT model). * :code:`-i, --in_image_dir`: The directory of input images for inference. * :code:`-o, --out_image_dir`: The directory path to output annotated images. * :code:`-k, --key`: Key to load model. * :code:`-e, --config_path`: Path to an experiment spec file for training. Optional Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-t, --draw_conf_thres`: Threshold for drawing a bbox. default: 0.3 * :code:`-h, --help`: Show this help message and exit * :code:`-l, --out_label_dir`: The directory to output KITTI labels. Running Inference on a YOLOv3 Model ----------------------------------- The tlt-infer tool for YOLOv3 networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. Here's an example of using this tool: .. code:: tlt-infer yolo -i -o -e -m [-l ] [-t ] -k Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-m, --mode`: Path to the pretrained model (TLT model). * :code:`-i, --in_image_dir`: The directory of input images for inference. * :code:`-o, --out_image_dir`: The directory path to output annotated images. * :code:`-k, --key`: Key to load model. * :code:`-e, --config_path`: Path to an experiment spec file for training. Optional Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-t, --draw_conf_thres`: Threshold for drawing a bbox. default: 0.3 * :code:`-h, --help`: Show this help message and exit * :code:`-l, --out_label_dir`: The directory to output KITTI labels. Running Inference on a RetinaNet Model -------------------------------------- The tlt-infer tool for RetinaNet networks can be used to visualize bboxes, or generate frame by frame KITTI format labels on a directory of images. Two modes are supported, namely TLT model model and TensorRT engine mode. You can execute the TLT model mode using the following command: .. code:: tlt-infer retinanet -i -o -e -m [-l ] [-t ] -k Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-m, --model`: Path to the pretrained model (TLT model). * :code:`-i, --in_image_dir`: The directory of input images for inference. * :code:`-o, --out_image_dir`: The directory path to output annotated images. * :code:`-k, --key`: Key to load model. * :code:`-e, --config_path`: Path to an experiment spec file for training. Optional Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-t, --draw_conf_thres`: Threshold for drawing a bbox. default: 0.3 * :code:`-h, --help`: Show this help message and exit * :code:`-l, --out_label_dir`: The directory to output KITTI labels. Alternatively, you can execute the TensorRT engine mode as follows: .. code:: tlt-infer retinanet -i -o -e -p [-t ] -k Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-p, --engine_path`: Path to the TensorRT (TLT exported). * :code:`-i, --in_image_dir`: The directory of input images for inference. * :code:`-o, --out_image_dir`: The directory path to output annotated images. * :code:`-k, --key`: Key to load model. * :code:`-e, --config_path`: Path to an experiment spec file for training. Optional Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-t, --draw_conf_thres`: Threshold for drawing a bbox. default: 0.3 * :code:`-h, --help`: Show this help message and exit * :code:`-l, --out_label_dir`: The directory to output KITTI labels. Alternatively, you can execute the TensorRT engine mode as follows: .. code:: tlt-infer retinanet -i -o -e -p [-t ] -k Required Arguments ^^^^^^^^^^^^^^^^^^ * :code:`-p, --engine_path`: Path to the TensorRT (TLT exported). * :code:`-i, --in_image_dir`: The directory of input images for inference. * :code:`-o, --out_image_dir`: The directory path to output annotated images. * :code:`-k, --key`: Key to load model. * :code:`-e, --config_path`: Path to an experiment spec file for training. Optional arguments ^^^^^^^^^^^^^^^^^^ * :code:`-t, --draw_conf_thres`: Threshold for drawing a bbox. default: 0.3 * :code:`-h, --help`: Show this help message and exit Running Inference on a MaskRCNN Model ------------------------------------- The :code:`tlt-infer` tool for MaskRCNN networks can be used to visualize bboxes, or generate frame by frame COCO format labels on a directory of images. Here's an example of using this tool: .. code:: tlt-infer mask_rcnn -i -o -e -m [-l