LPRNet

LPRNet (License Plate Recognition Net) takes images as network input and predicts sequences of license plates characters.

The dataset for LPRNet contains cropped license plates images and corresponding label files.

The data structure must be in the following format:

            

            
/Dataset_01
    /images
        0000.jpg
        0001.jpg
        0002.jpg
        ...
        ...
        ...
        N.jpg
    /labels
        0000.txt
        0001.txt
        0002.txt
        ...
        ...
        ...
        N.txt
/characters_list.txt
        

Each cropped license plate image has a corresponding label text file that contains one line of characters in the specific license plate. There is a characters_list.txt which has all the characters found in license plate dataset. Each character occupies one line.

The spec file for LPRNet includes the random_seed, lpr_config, training_config, eval_config, augmentation_config, and dataset_config parameters. Here is an example for training on the NVIDIA license plate dataset:

            

            
random_seed: 42
lpr_config {
  hidden_units: 512
  max_label_length: 8
  arch: "baseline"
  nlayers: 10
}
training_config {
  batch_size_per_gpu: 32
  num_epochs: 100
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 1e-6
    max_learning_rate: 1e-4
    soft_start: 0.001
    annealing: 0.7
  }
  }
  regularizer {
    type: L2
    weight: 5e-4
  }
}
eval_config {
  validation_period_during_training: 5
  batch_size: 1
}
augmentation_config {
  output_width: 96
  output_height: 48
  output_channel: 3
  max_rotate_degree: 5
  rotate_prob: 0.5
  gaussian_kernel_size: 5
  gaussian_kernel_size: 7
  gaussian_kernel_size: 15
  blur_prob: 0.5
  reverse_color_prob: 0.5
  keep_original_prob: 0.3
}
dataset_config {
  data_sources: {
    label_directory_path: "/path/to/train/labels"
    image_directory_path: "/path/to/train/images"
  }
  characters_list_file: "/path/to/lp_characters"
  validation_data_sources: {
    label_directory_path: "/path/to/test/labels"
    image_directory_path: "/path/to/test/images"
  }
}
        

lpr_config

The lpr_config parameter provides options to change the LPRNet architecture.

            

            
lpr_config {
  hidden_units: 512
  max_label_length: 8
  arch: "baseline"
  nlayers: 10
}
        

training_config

The training_config parameter defines the hyperparameters of the training process.

            

            
training_config {
  checkpoint_interval: 5
  batch_size_per_gpu: 32
  num_epochs: 100
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 1e-6
    max_learning_rate: 1e-4
    soft_start: 0.001
    annealing: 0.7
  }
  }
  regularizer {
    type: L2
    weight: 5e-4
  }
}
        

visualizer

Visualization during training is configured using the visualizer parameter. The parameters are described in the table below.

If visualization is enabled, the TensorBoard log will be produced during training, including the graphs for learning rate, training loss, and validation accuracy. The augmented images will also be produced in the TensorBoard.

early_stopping

The parameters for early stopping are described in the table below.

eval_config

The eval_config parameter defines the hyperparameters of the evaluation process. The metric for evaluation is the license plate recognition accuracy. The recognition will be regarded as correct if all the characters in a license plated are classified correctly.

            

            
eval_config {
  validation_period_during_training: 5
  batch_size: 1
}
        

augmentation_config

The augmentation_config parameter contains the hyperparameters for augmentation during training. It also defines the spatial size of the network input.

            

            
augmentation_config {
  output_width: 96
  output_height: 48
  output_channel: 3
  max_rotate_degree: 5
  rotate_prob: 0.5
  gaussian_kernel_size: 5
  gaussian_kernel_size: 7
  gaussian_kernel_size: 15
  blur_prob: 0.5
  reverse_color_prob: 0.5
  keep_original_prob: 0.3
}
        

dataset_config

The dataset_config parameter defines the path to training dataset, validation dataset, and characters list file.

            

            
dataset_config {
  data_sources: {
    label_directory_path: "/path/to/train/labels"
    image_directory_path: "/path/to/train/images"
  }
  characters_list_file: "/path/to/lp_characters"
  validation_data_sources: {
    label_directory_path: "/path/to/test/labels"
    image_directory_path: "/path/to/test/images"
  }
}
        

Use the following command to run LPRNet training:

            

            
tao model lprnet train -e <experiment_spec_file>
                 -r <results_dir>
                 -k <key>
                [--gpus <num_gpus>]
                [--gpu_index <gpu_index>]
                [--use_amp]
                [--log_file <log_file>]
                [-m <resume_model_path>]
                [--initial_epoch <initial_epoch>]
        

Required Arguments

• -e, --experiment_spec_file: The path to the experiment spec file

• -r, --results_dir: The path to a folder where the experiment outputs should be written.

• -k, --key: The user-specific encoding key to save or load a .tlt model.

Optional Arguments

• --gpus: The number of GPUs to be used in the training in a multi-GPU scenario (default: 1).

• --gpu_index: The GPU indices used to run the training. We can specify the GPU indices used to run training when the machine has multiple GPUs installed.

• --use_amp: A flag to enable AMP training.

• --log_file: The path to the log file. Defaults to stdout.

• -m, --resume_model_weights: The path to a pretrained model or a model to continue training.

• --initial_epoch: The epoch to start training.

Here’s an example of using the LPRNet training command:

            

            
tao model lprnet train --gpu_index=0 -e $DEFAULT_SPEC -r $RESULTS_DIR -k $YOUR_KEY
        

The evaluation metric of LPRNet is recognition accuracy. A recognition will be regarded accurate if all the characters in the license plate are correct.

Use the following command to run LPRNet evaluation:

            

            
tao model lprnet evaluate -m <model>
                    -e <experiment_spec_file>
                    [-k <key>]
                    [--gpu_index <gpu_index>]
                    [--log_file <log_file>]
                    [--trt]
        

Required Arguments

• -m, --model: .tlt model or TRT engine to be evaluated.

• -e, --experiment_spec_file: Experiment spec file to set up the evaluation experiment. This should be the same as a training spec file.

Optional Arguments

• -h, --help: Show this help message and exit.

• -k, --key：The encoding key for the .tlt model.

• --gpu_index: The GPU index used to run the evaluation. You can specify the GPU index used to run evaluation when the machine has multiple GPUs installed. Note that evaluation can only run on a single GPU.

• --log_file: The path to the log file. Defaults to stdout.

• --trt: Evaluate the TRT engine.

Here’s an example of using the LPRNet evaluation command:

            

            
tao model lprnet evaluate --gpu_index=0 -m $TRAINED_TLT_MODEL -e $DEFAULT_SPEC -k $YOUR_KEY
        

Use the following command to run inference on LPRNet with .tlt model or TensorRT engine:

            

            
tao model lprnet inference -m <model>
                     -i <in_image_path>
                     -e <experiment_spec>
                     [-k <key>]
                     [--gpu_index <gpu_index>]
                     [--log_file <log_file>]
                     [--trt]
        

Required Arguments

• -m, --model: The .tlt model or TensorRT engine to do inference with

• -i, --in_image_path: The path to the license plate images to do inference with.

• -e, --experiment_spec: The experiment spec file to set up export. Can be the same as the training spec.

Optional Arguments

• -h, --help: Show this help message and exit.

• -k, --key：The encoding key for the .tlt model.

• --gpu_index: The GPU index used to run the inference. We can specify the GPU index used to run inference when the machine has multiple GPUs installed. Note that inference can only run on a single GPU.

• --log_file: Path to the log file. Defaults to stdout.

• --trt: Run inference with the TRT engine.

Here’s an example of using the LPRNet inference command:

            

            
tao model lprnet inference --gpu_index=0 -m $SAVED_TRT_ENGINE -i $PATH_TO_TEST_IMAGES -e $DEFAULT_SPEC --trt
        

Use the following command to export LPRNet to .etlt format for deployment:

            

            
tao model lprnet export -m <model>
                  -k <key>
                  -e <experiment_spec>
                  [--gpu_index <gpu_index>]
                  [--log_file <log_file>]
                  [-o <output_file>]
                  [--data_type {fp32,fp16}]
                  [--max_workspace_size <max_workspace_size>]
                  [--max_batch_size <max_batch_size>]
                  [--engine_file <engine_file>]
                  [-v]
        

Required Arguments

• -m, --model: The .tlt model to be exported.

• -k, --key: The encoding key of the .tlt model.

• -e, --experiment_spec: Experiment spec file to set up export. Can be the same as the training spec.

Optional Arguments

• --gpu_index: The index of (discrete) GPUs used for exporting the model. We can specify the GPU index to run export if the machine has multiple GPUs installed. Note that export can only run on a single GPU.

• --log_file: The path to the log file. Defaults to stdout.

• -o, --output_file: The path to save the exported model to. The default is ./<input_file>.etlt.

• --data_type: The desired engine data type to generate the calibration cache if in INT8 mode. The options are fp32 or fp16. The default value is fp32.

You can use the following optional arguments to save the TRT engine that is generated to verify export:

• --max_batch_size: The maximum batch size of TensorRT engine. The default value is 16.

• --max_workspace_size: The maximum workspace size of the TensorRT engine. The default value is 1073741824(1<<30).

• --engine_file: The path to the serialized TensorRT engine file. Note that this file is hardware specific and cannot be generalized across GPUs. Useful to quickly test your model accuracy using TensorRT on the host. As the TensorRT engine file is hardware specific, you cannot use this engine file for deployment unless the deployment GPU is identical to the training GPU.

Here’s an example for using the LPRNet export command:

            

            
tao model lprnet export --gpu_index=0 -m $TRAINED_TAO_MODEL -e $DEFAULT_SPEC -k $YOUR_KEY
        

The deep learning and computer vision models that you trained can be deployed on edge devices, such as a Jetson Xavier, Jetson Nano, or Tesla, or in the cloud with NVIDIA GPUs.

DeepStream SDK is a streaming analytic toolkit to accelerate building AI-based video analytic applications. TAO Toolkit is integrated with DeepStream SDK, so models trained with TAO Toolkit will work out of the box with Deepstream.

Using tao-converter

The tao-converter is a tool that is provided with the TAO Toolkit to facilitate the deployment of TAO Toolkit trained models on TensorRT and/or Deepstream. For deployment platforms with an x86 based CPU and discrete GPUs, the tao-converter is distributed within the TAO docker. Therefore, it is suggested to use the docker to generate the engine. However, this requires that the user adhere to the same minor version of TensorRT as distributed with the docker. The TAO docker includes TensorRT version 7.1. In order to use the engine with a different minor version of TensorRT, copy the converter from /opt/nvidia/tools/tao-converter to the target machine and follow the instructions for x86 to run it and generate a TensorRT engine.

For the aarch64 platform, the tao-converter is available to download in the dev zone.

Here is a sample command to generate LPRNet engine through tao-converter:

            

            
tao-converter <etlt_model> -k <key_to_etlt_model> -p image_input,1x3x48x96,4x3x48x96,16x3x48x96 -e <path_to_generated_trt_engine>
        

Through this command, an optimized TensorRT engine with dynamic input shape will be generated. (Dynamic shape of this engine: min_shape=[1x3x48x96], opt_shape=[4x3x48x96], max_shape=[16x3x48x96]. The shape format is NCHW.)

Deploying the LPRNet in the DeepStream sample

Once you get the TensorRT engine of LPRNet, you could deploy it into DeepStream’s LPDR sample. This sample is a complete solution including car detection, license plate detection, and license plate recognition. A configurable CTC decoder is provided in this sample.