Gesture Recognition

GestureNet is an NVIDIA-developed gesture classification model that is included in the Transfer Learning Toolkit. GestureNet supports the following tasks:

• dataset_convert

• train

• evaluate

• inference

• export

These tasks can be invoked from the TLT launcher using the following convention on the command line:

            

            
tlt gesturenet <sub_task> <args_per_subtask>
        

where args_per_subtask are the command-line arguments required for a given subtask. Each subtask is explained in detail below.

The GestureNet App requires the data images and labels to be in a specific format. Once it is prepared, the Transfer Learning Toolkit includes dataset_convert to prepare the data for model training.

See the Data Annotation Format page for more information about the data format for gesture recognition.

Dataset Extraction Config

The dataset_config spec specifies the parameters neededed to crop hand bounding box and prepare dataset.

Here’s a sample spec:

            

            
{
    "org_dataset": "data",
    "mount_dir_path" :"/workspace/tlt-experiments/gesturenet/",
    "org_data_dir" : "original",
    "post_data_dir" : "extracted",
    "kpi_users": ["uid_1", "uid_2"],
    "sampling_rate": 1,
    "convert_bbox_square": true,
    "bbox_enlarge_ratio": 1.1,
    "annotation_config":{
        "annotation_path": "annotation"
    }
}
        

The following table describes the parameters:

Dataset Experiment Config

The dataset_experiment_config spec specifies the parameters neededed to combine different datasets. It allows user to provide user IDs that are set aside for validation or test set. It also allows different sampling strategies based on meta data and class counts.

Here’s a sample spec:

            

            
{
    "mount_dir_path" :"/workspace/tlt-experiments/gesturenet",
    "org_data_dir" : "original",
    "post_data_dir" : "extracted",
    "set_list": {
        "train_val": [
            "data"
        ],
        "kpi": [
            "data"
        ]
    },
    "uid_list": {
        "uid_name": "user_id",
        "predefined_val_users": false,
        "val_fraction": 0.25,
        "validation": [
        ],
        "kpi": [
            "uid_1"
        ]
    },
    "image_feature_filter": {
        "train_val":{
        "*":[
            {
                "location":"outdoor"
            }
        ]
        },
        "kpi":{
        }
    },
    "sampling": {
        "sampling_mode": "average",
        "use_class_weights": false,
        "class_weights": {
            "thumbs_up": 0.5,
            "v": 0.5
        }
    }
}
        

The following table describes the set_list parameters:

The following table describes the uid_list parameters:

The following table describes the image_feature_filter parameters:

The following table describes the sampling parameters:

Sample Usage of the Dataset Converter Tool

TLT has built in commands to run prepare datset for GestureNet model and is given below.

            

            
tlt gesturenet dataset_convert  --dataset_spec <dataset_spec_path>
                                --experiment_spec <experiment_spec_path>
                                --k_folds <num_folds>
                                --output_filename <output_filename>
                                --experiment_name <experiment_name>
        

Required Arguments

• --dataset_spec: The path to dataset spec.

• --experiment_spec: The path to dataset experiment spec.

• --k_folds: Number of folds.

• --output_filename: Output json that is ingested by GestureNet training pipeline.

• --experiment_name: Name of experiment.

Sample Usage

Here is an example using a GestureNet model.

            

            
tlt gesturenet dataset_convert --dataset_spec $SPECS_DIR/dataset_config.json \
                               --k_folds 0 \
                               --experiment_spec $SPECS_DIR/dataset_experiment_config.json \
                               --output_filename $USER_EXPERIMENT_DIR/data.json \
                               --experiment_name v1
        

To do training, evaluation, and inference for GestureNet, several components need to be configured, each with their own parameters. The gesturenet train, gesturenet evaluate, and gesturenet inference commands for a GestureNet experiment share the same configuration file.

The main components of the configuration file is given below.

• Trainer

• Model

• Evaluator

Trainer Config

The trainer configuration allows you to configure how you want to train your model. The two main components are top_training and finetuning. num_workers allows you to specify how many workers to use to train your model. Details on the top_training configuration file is explained in detail in the next section.

Top Training Config

The top training configuration allows you to customize how your model trains. There are 5 main components to the top_training configuration and they are as follows:

Fine Tuning Config

Each fine tuning configuration file has 9 different options to perform fine tuning and is listed below.

Model Config

The model configuration file allows you to customize the architecture you want to use and the hyperparameters. The key options available are given below.

Evaluator Config

Evaluator configuration is the configuration options for evaluating your GestureNet.

TLT has built-in commands to train a GestureNet model and is given below.

            

            
tlt gesturenet train -e <spec_file>
                     -k <key>
        

Required Arguments

• -e, --experiment_spec_filename: Path to spec file.

• -k, –key: User specific encoding key to save or load a .tlt model.

Sample Usage

Here’s an example of using the train command on GestureNet:

            

            
tlt gesturenet train -e $SPECS_DIR/train_spec.json \
                     -k $KEY
        

TLT has built in commands to evaluate a GestureNet model and is given below.

            

            
tlt gesturenet evaluate -e <spec_file>
                        -m <model_file>
                        -k <key>
        

Required Arguments

• -e, --experiment_spec_filename: Experiment spec file to set up the evaluation experiment. This should be the same as training spec file.

• -m, --model: Path to the model file to use for evaluation.

• -k, -–key: Provide the encryption key to decrypt the model. This is a required argument only with a .tlt model file.

Sample Usage

Here’s an example of using the evaluation command on a GestureNet model.

            

            
tlt gesturenet evaluate -e $USER_EXPERIMENT_DIR/model/train_spec.json \
                        -m $USER_EXPERIMENT_DIR/model/model.tlt \
                        -k $KEY
        

TLT has built in commands to run inference on a GestureNet model and is given below.

            

            
tlt gesturenet inference -e <spec_file>
                         -m <model_full_path>
                         -k <key>
                         --image_root_path <root_path>
                         --data_json <json_path>
                         --data_type <data_type>
                         -results_dir <results_dir>
        

Required Arguments

• -e, --experiment_spec_filename: Experiment spec file to set up the evaluation experiment. The model parameters should be the same as training spec file.

• -m, --model: Path to the model file to use for evaluation.

• -k, -–key: Provide the encryption key to decrypt the model. This is a required argument. only with a .tlt model file.

• --image_root_path: The root directpry that dataset is mounted at.

• --data_json: The json spec with image path and hand bounding box.

• --data_type: The dataset type within data_json that inference is to be run on.

• --results_dir: Directory where the results are saved.

Sample Usage

Here is an example of running inference using a GestureNet model.

            

            
tlt gesturenet inference -e $USER_EXPERIMENT_DIR/model/train_spec.json \
                         -m $USER_EXPERIMENT_DIR/model/model.tlt \
                         -k $KEY \
                         --image_root_path /workspace/tlt-experiments/gesturenet \
                         --data_json /workspace/tlt-experiments/gesturenet/data.json \
                         --data_type kpi_set \
                         --results_dir $USER_EXPERIMENT_DIR/model
        

The command to export the GestureNet model to a TensorRT plan can be found below. It only supports FP16 at the moment.

            

            
tlt gesturenet export -m <model_full_path>
                      -k <key>
                      -o <out_file>
                      -t <export_type>
                      -ll <log_level>
        

Required Arguments

• -m, --model_filename: The full path to the model to export.

• -k, --key: Encryption key used to train the model.

• -o, --out_file: Place to save the exported model.

Optional Arguments

• -t, --export_type: Type of export to use. Options are onnx or tfonnx.

• -ll, --log_level: The log level to use.

Sample Usage

Here is an example of exporting a GestureNet model.

            

            
tlt gesturenet export -m $USER_EXPERIMENT_DIR/model/model.tlt \
                      -k $KEY \
                      -o $USER_EXPERIMENT_DIR/model/model.etlt \
                      -t 'tfonnx'
        

The pretrain model for gesture classification provided through NGC is available by default to use inside the TLT CV Inference Pipeline. You can also deploy a model trained through TLT workflow to the TLT CV Inference Pipeline. Refer to TLT CV Quick Start Scripts section for instructions of both options.