Gesture Recognition

GestureNet is an NVIDIA-developed gesture classification model that is included in the TAO Toolkit. GestureNet supports the following tasks:

• dataset_convert

• train

• evaluate

• inference

• export

These tasks can be invoked from the TAO Toolkit Launcher using the following convention on the command line:

            

            
tao gesturenet <sub_task> <args_per_subtask>
        

where args_per_subtask are the command-line arguments required for a given subtask. Each subtask is explained in detail below.

The GestureNet App requires the data images and labels to be in a specific format. Once it is prepared, the TAO Toolkit includes dataset_convert to prepare the data for model training.

See the Data Annotation Format page for more information about the data format for gesture recognition.

Dataset Extraction Config

The dataset_config spec specifies the parameters neededed to crop hand bounding box and prepare dataset.

Here’s a sample spec:

            

            
{
    "org_dataset": "data",
    "mount_dir_path" :"/workspace/tao-experiments/gesturenet/",
    "org_data_dir" : "original",
    "post_data_dir" : "extracted",
    "kpi_users": ["uid_1", "uid_2"],
    "sampling_rate": 1,
    "convert_bbox_square": true,
    "bbox_enlarge_ratio": 1.1,
    "annotation_config":{
        "annotation_path": "annotation"
    }
}
        

The following table describes the parameters:

Dataset Experiment Config

The dataset_experiment_config spec specifies the parameters neededed to combine different datasets. It allows user to provide user IDs that are set aside for validation or test set. It also allows different sampling strategies based on meta data and class counts.

Here’s a sample spec:

            

            
{
    "mount_dir_path" :"/workspace/tao-experiments/gesturenet",
    "org_data_dir" : "original",
    "post_data_dir" : "extracted",
    "set_list": {
        "train_val": [
            "data"
        ],
        "kpi": [
            "data"
        ]
    },
    "uid_list": {
        "uid_name": "user_id",
        "predefined_val_users": false,
        "val_fraction": 0.25,
        "validation": [
        ],
        "kpi": [
            "uid_1"
        ]
    },
    "image_feature_filter": {
        "train_val":{
        "*":[
            {
                "location":"outdoor"
            }
        ]
        },
        "kpi":{
        }
    },
    "sampling": {
        "sampling_mode": "average",
        "use_class_weights": false,
        "class_weights": {
            "thumbs_up": 0.5,
            "v": 0.5
        }
    }
}
        

The following table describes the set_list parameters:

The following table describes the uid_list parameters:

The following table describes the image_feature_filter parameters:

The following table describes the sampling parameters:

Sample Usage of the Dataset Converter Tool

TAO has built in commands to run prepare datset for GestureNet model and is given below.

            

            
tao gesturenet dataset_convert  --dataset_spec <dataset_spec_path>
                                --experiment_spec <experiment_spec_path>
                                --k_folds <num_folds>
                                --output_filename <output_filename>
                                --experiment_name <experiment_name>
        

Required Arguments

• --dataset_spec: The path to dataset spec.

• --experiment_spec: The path to dataset experiment spec.

• --k_folds: Number of folds.

• --output_filename: Output json that is ingested by GestureNet training pipeline.

• --experiment_name: Name of experiment.

Sample Usage

Here is an example using a GestureNet model.

            

            
tao gesturenet dataset_convert --dataset_spec $SPECS_DIR/dataset_config.json \
                               --k_folds 0 \
                               --experiment_spec $SPECS_DIR/dataset_experiment_config.json \
                               --output_filename $USER_EXPERIMENT_DIR/data.json \
                               --experiment_name v1
        

To do training, evaluation, and inference for GestureNet, several components need to be configured, each with their own parameters. The gesturenet train, gesturenet evaluate, and gesturenet inference commands for a GestureNet experiment share the same configuration file.

The main components of the configuration file is given below.

• Trainer

• Model

• Evaluator

Trainer Config

The trainer configuration allows you to configure how you want to train your model. The two main components are top_training and finetuning. num_workers allows you to specify how many workers to use to train your model. Details on the top_training configuration file is explained in detail in the next section.

Top Training Config

The top training configuration allows you to customize how your model trains. There are 5 main components to the top_training configuration and they are as follows:

Fine Tuning Config

Each fine tuning configuration file has 9 different options to perform fine tuning and is listed below.

Model Config

The model configuration file allows you to customize the architecture you want to use and the hyperparameters. The key options available are given below.

Evaluator Config

Evaluator configuration is the configuration options for evaluating your GestureNet.

TAO Toolkit has built-in commands to train a GestureNet model and is given below.

            

            
tao gesturenet train -e <spec_file>
                     -k <key>
        

Required Arguments

• -e, --experiment_spec_filename: Path to spec file.

• -k, –key: User specific encoding key to save or load a .tlt model.

Sample Usage

Here’s an example of using the train command on GestureNet:

            

            
tao gesturenet train -e $SPECS_DIR/train_spec.json \
                     -k $KEY
        

TAO Toolkit has built in commands to evaluate a GestureNet model and is given below.

            

            
tao gesturenet evaluate -e <spec_file>
                        -m <model_file>
                        -k <key>
        

Required Arguments

• -e, --experiment_spec_filename: Experiment spec file to set up the evaluation experiment. This should be the same as training spec file.

• -m, --model: Path to the model file to use for evaluation.

• -k, -–key: Provide the encryption key to decrypt the model. This is a required argument only with a .tlt model file.

Sample Usage

Here’s an example of using the evaluation command on a GestureNet model.

            

            
tao gesturenet evaluate -e $USER_EXPERIMENT_DIR/model/train_spec.json \
                        -m $USER_EXPERIMENT_DIR/model/model.tlt \
                        -k $KEY
        

TAO Toolkit has built in commands to run inference on a GestureNet model and is given below.

            

            
tao gesturenet inference -e <spec_file>
                         -m <model_full_path>
                         -k <key>
                         --image_root_path <root_path>
                         --data_json <json_path>
                         --data_type <data_type>
                         -results_dir <results_dir>
        

Required Arguments

• -e, --experiment_spec_filename: Experiment spec file to set up the evaluation experiment. The model parameters should be the same as training spec file.

• -m, --model: Path to the model file to use for evaluation.

• -k, -–key: Provide the encryption key to decrypt the model. This is a required argument. only with a .tlt model file.

• --image_root_path: The root directpry that dataset is mounted at.

• --data_json: The json spec with image path and hand bounding box.

• --data_type: The dataset type within data_json that inference is to be run on.

• --results_dir: Directory where the results are saved.

Sample Usage

Here is an example of running inference using a GestureNet model.

            

            
tao gesturenet inference -e $USER_EXPERIMENT_DIR/model/train_spec.json \
                         -m $USER_EXPERIMENT_DIR/model/model.tlt \
                         -k $KEY \
                         --image_root_path /workspace/tao-experiments/gesturenet \
                         --data_json /workspace/tao-experiments/gesturenet/data.json \
                         --data_type kpi_set \
                         --results_dir $USER_EXPERIMENT_DIR/model
        

TAO Toolkit provides a utility for exporting a trained model to an encrypted onnx format or a TensorRT deployable engine format.

The export sub-task optionally generates the calibration cache for TensorRT INT8 engine calibration.

Exporting the model decouples the training process from deployment and allows for conversion to TensorRT engines outside the TAO environment. TensorRT engines are specific to each hardware configuration and should be generated for each unique inference environment. This may be interchangeably referred to as a .trt or .engine file. The same exported TAO model may be used universally across training and deployment hardware. This is referred to as the .etlt file, or encrypted TAO file. During model export, the TAO model is encrypted with a private key, which is required when you deploy this model for inference.

INT8 Mode Overview

TensorRT engines can be generated in INT8 mode to run with lower precision,thus improving performance. This process requires a cache file that contains scale factors for the tensors to help combat quantization errors, which may arise due to low-precision arithmetic. The calibration cache is generated using a calibration tensorfile when export is run with the --data_type flag set to int8. Pre-generating the calibration information and caching it removes the need for calibrating the model on the inference machine. Moving the calibration cache is usually much more convenient than moving the calibration tensorfile since it is a much smaller file and can be moved with the exported model. Using the calibration cache also speeds up engine creation, as building the cache can take several minutes to generate depending on the size of the Tensorfile and the model itself.

The export tool can generate an INT8 calibration cache by ingesting a sampled subset of training data. You need to create a sub-sampled directory of random images that best represent your test dataset. We recommend using at least 10-20% of the training data. The more data provided during calibration, the closer int8 inferences are to fp32 inferences. A helper script is provided with the sample notebook to select the subset data from the given training data.

Based on the evaluation results of the INT8 model, you might need to adjust the number of sampled images or the kind of selected to images to better represent test dataset. You can also use a portion of data from the test data for calibration to improve the results.

FP16/FP32 Model

The calibration.bin is only required if you need to run inference at INT8 precision. For FP16/FP32 based inference, the export step is much simpler. All that is required is to provide a model from the train step to export to convert it into an encrypted TAO model.

Sample Usage of the Export tool

            

            
tao gesturenet export -m <Trained TAO Model Path> -k <Encode Key> -o <Output file .etlt>
        

• -m: The path to the trained model to be exported

• -k: The encryption key for model loading

• -o: The path to the output .etlt file (.etlt is appended to model path otherwise)

• -t: The target opset value for onnx conversion. The default value is 10

• --cal_data_file: The path to the calibration data file (.tensorfile)

• --cal_image_dir The path to a directory with calibration image samples

• --cal_cache_file The path to the calibration file (.bin)

• --data_type: The data type for the TensorRT export. The options are fp32 and int8.

• --batches: The number of images per batch. The default value is 1.

• --max_batch_size: The maximum batch size for the TensorRT engine builder. The default value is 1.

• --max_workspace_size: The maximum workspace size to be set for the TensorRT engine builder

• --batch_size: The number of batches to calibrate over. The default value is 1.

• --engine_file: The path to the exported TRT engine. Generates an engine file if specified.

• --input_dims: Input dims in channels first(CHW) or channels last (HWC) format as comma separated integer values. Default 3,160,160.

• --backend: The model type to export to.

INT8 Export Mode Required Arguments

• --cal_image_dir: The directory of images that is preprocessed and used for calibration.

• --cal_data_file: The tensorfile generated using images in cal_image_dir for calibrating the engine. If this already exists, it is directly used to calibrate the engine. The INT8 tensorfile is a binary file that contains the preprocessed training samples.

INT8 Export Optional Arguments

• --cal_cache_file: The path to save the calibration cache file to. The default value is ./cal.bin. If this file already exists, the calibration step is skipped.

• --batches: The number of batches to use for calibration and inference testing. The default value is 10.

• --batch_size: The batch size to use for calibration. The default value is 1.

• --max_batch_size: The maximum batch size of the TensorRT engine. The default value is 1.

• --max_workspace_size: The maximum workspace size of the TensorRT engine. The default value is 2 * (1 << 30).

• --experiment_spec: The experiment_spec used for training. This argument is used to obtain the parameters to preprocess the data used for calibration.

• --engine_file: The path to the serialized TensorRT engine file. Note that this file is hardware specific and cannot be generalized across GPUs. Use this argument to quickly test your model accuracy using TensorRT on the host. As the TensorRT engine file is hardware specific, you cannot use this engine file for deployment unless the deployment GPU is identical to the training GPU.

Deploying to DeepStream 6.0

The pretrained model for GestureNet provided through NGC is available by default with DeepStream 6.0.

For more details, refer to DeepStream TAO Integration for GestureNet.