ActionRecognitionNet

ActionRecognitionNet takes a sequence of images as network input and predicts the people’s action in those images. TAO provides the network backbones in 2D/3D with the following input options: RGB-only input, optical flow (OF) only input, and two-stream joint input (RGB+OF).

Action Recognition Net Architecture

ActionRecognitionNet requires RGB video frames for the RGB input stream and optical flow vectors for the OF input stream. The x-axis and y-axis of the raw optical flow vectors should be mapped to grayscale images for training. We provide a tool to preprocess sample. This tool converts the video to frames and generate optical flow images based on the NVIDIA Optical Flow (NVOF) SDK.

Organize the data in the following structure:

            

            
/Dataset
    /class_a
        /video_1
            /rgb
                000000.png
                000001.png
                ...
                N.png
            /u
                000000.jpg
                000001.jpg
                ...
                N-1.jpg
            /v
                000000.jpg
                000001.jpg
                ...
                N-1.jpg
        

The root directory of dataset contains multiple sub-directories for different classes. Each class directory has sub-folders for different videos, and each of these subfolders contain rgb, u and v folders that respectively hold RGB frames, optical flow x-axis grayscale images, and optical flow y-axis grayscale images. The u and v folders can be empty if you want to train an RGB-only model. A script is provided to generate RGB frames only.

The common data process pipeline can be depicted with the following diagrams:

RGB-Only data process pipeline

OF-Only data process pipeline

The spec file for ActionRecognitionNet includes model, train, and dataset parameters. Here is an example spec for training a 3D RGB-only model with a resnet18 backbone on a dataset that contains 5 classes: “walk”, “sits”, “squat”, “fall”, “bend”:

            

            
model:
  model_type: rgb
  backbone: resnet18
  rgb_seq_length: 3
  input_type: 3d
  sample_rate: 1
  dropout_ratio: 0.0
dataset:
  train_dataset_dir: /data/train
  val_dataset_dir: /data/test
  label_map:
    walk: 0
    sits: 1
    squat: 2
    fall: 3
    bend: 4
  output_shape:
  - 224
  - 224
  batch_size: 32
  workers: 8
  clips_per_video: 15
  augmentation_config:
    train_crop_type: no_crop
    horizontal_flip_prob: 0.5
    rgb_input_mean: [0.5]
    rgb_input_std: [0.5]
    val_center_crop: False
train:
  optim:
    lr: 0.0005
    momentum: 0.9
    weight_decay: 0.0005
    lr_scheduler: MultiStep
    lr_decay: 0.1
    lr_steps: [15, 25]
    patience: 1
    min_lr: 0.0001
  num_epochs: 10
  checkpoint_interval: 5
  validation_interval: 5
  clip_grad_norm: 0.0
  num_gpus: 1
  gpu_ids: [0]
  seed: 1234
        

model

The model parameter provides options to change the ActionRecognitionNet architecture.

            

            
model:
  model_type: rgb
  backbone: resnet18
  rgb_seq_length: 3
  input_type: 3d
  sample_rate: 1
  dropout_ratio: 0.0
        

train

The train parameter defines the hyperparameters of the training process.

            

            
train:
  optim:
    lr: 0.0005
    momentum: 0.9
    weight_decay: 0.0005
    lr_scheduler: MultiStep
    lr_decay: 0.1
    lr_steps: [15, 25]
    patience: 1
    min_lr: 0.0001
  num_epochs: 10
  checkpoint_interval: 5
  validation_interval: 5
  clip_grad_norm: 0.0
  num_gpus: 1
  gpu_ids: [0]
  seed: 1234
        

optim

The optim parameter defines the config for the SGD optimizer in training, including the learning rate, learning scheduler, and weight decay.

            

            
optim:
  lr: 0.0005
  momentum: 0.9
  weight_decay: 0.0005
  lr_scheduler: MultiStep
  lr_decay: 0.1
  lr_steps: [15, 25]
  patience: 1
  min_lr: 0.0001
        

dataset

The dataset parameter defines the dataset source, training batch size, and augmentation.

            

            
dataset:
  train_dataset_dir: /data/train
  val_dataset_dir: /data/test
  label_map:
    walk: 0
    sits: 1
    squa: 2
    fall: 3
    bend: 4
  output_shape:
  - 224
  - 224
  batch_size: 32
  workers: 8
  clips_per_video: 15
  augmentation_config:
    train_crop_type: no_crop
    horizontal_flip_prob: 0.5
    rgb_input_mean: [0.5]
    rgb_input_std: [0.5]
    val_center_crop: False
        

augmentation_config

The augmentation_config parameter contains hyperparameters for augmentation.

            

            
augmentation_config:
  train_crop_type: no_crop
  horizontal_flip_prob: 0.5
  rgb_input_mean: [0.5]
  rgb_input_std: [0.5]
  val_center_crop: False
        

Use the following command to run ActionRecognitionNet training:

            

            
tao model action_recognition train [-h] -e <experiment_spec_file>
                             [results_dir=<global_results_dir>]
                             [model.<model_option>=<model_option_value>]
                             [dataset.<dataset_option>=<dataset_option_value>]
                             [train.<train_option>=<train_option_value>]
                             [train.gpu_ids=<gpu indices>]
                             [train.num_gpus=<number of gpus>]
        

Required Arguments

The only required argument is the path to the experiment spec:

• -e, --experiment_spec: The experiment specification file to set up the training experiment

Optional Arguments

You can set optional arguments to override the option values in the experiment spec file.

• -h, --help: Show this help message and exit.

• model.<model_option>: The model options.

• dataset.<dataset_option>: The dataset options.

• train.<train_option>: The train options.

• train.optim.<optim_option>: The optimizer options

Checkpointing and Resuming Training

At every train.checkpoint_interval, a PyTorch Lightning checkpoint is saved. It is called model_epoch_<epoch_num>.pth. These are saved in train.results_dir, like so:

            

            
$ ls /results/train

'model_epoch_000.pth'
'model_epoch_001.pth'
'model_epoch_002.pth'
'model_epoch_003.pth'
'model_epoch_004.pth'
        

The latest checkpoint is also saved as ar_model_latest.pth. Training automatically resumes from ar_model_latest.pth, if it exists in train.results_dir. This is superseded by train.resume_training_checkpoint_path, if it is provided.

The major implication of this logic is that, if you wish to trigger fresh training from scratch, either:

• Specify a new, empty results directory (Recommended)

• Remove the latest checkpoint from the results directory

The evaluation metric of ActionRecognitionNet is recognition accuracy. Two modes of video sampling strategies are provided for evaluation on a video: center and conv.

The center evaluation inference is performed on the middle part of frames in a video clip. For example, if the model requires 32 frames as input and a video clip has 128 frames, then the frames from index 48 to index 79 will be used to perform inference.

The conv evaluation inference is performed on a number of segments out of a video clip. For example, a video clip is divided uniformly into 10 parts; the center of each segments is treated as a starting point from which 32 consecutive frames are chosen to form an inference segment. In this manner, an inference segment is generated for every part the video was divided into. And the final label of the video is determined by the average score of those 10 segments.

Use the following command to run ActionRecognitionNet evaluation:

            

            
tao model action_recognition evaluate -e <experiment_spec_file>
                             evaluate.checkpoint=<model to be evaluated>
                             [evaluate.batch_size=<batch size>]
                             [evaluate.test_dataset_dir=<path to test dataset>]
                             [evaluate.video_eval_mode=<evaluation mode for the video>]
                             [evaluate.video_num_segments=<number of segments for ``conv`` mode>]
                             [evaluate.gpu_ids=<gpu indices>]
                             [evaluate.num_gpus=<number of gpus>]
        

Multi-GPU evaluation is currently not supported for Action Recognition.

Required Arguments

• -e, --experiment_spec_file: THe xperiment spec file to set up the evaluation experiment. This should be the same as a training spec file.

• evaluate.checkpoint: The .pth model.

Optional Arguments

• evaluate.batch_size: The batch size to perform inference in evaluation. The default value is 1.

• evaluate.test_dataset_dir: The path to the test dataset. If not set, the validation dateset in the experiment_spec will be used.

• evaluate.video_eval_mode: The evaluation mode for the video:

  • center: Evaluation inference is performed on the middle part of frames in the video clip. This is the default mode.

  • conv: Evaluation inference is performed on a number of segments out of a video clip. The final prediction is averaged among all the segments.

• evaluate.video_num_segments: The number of segments sampled in a video clip for conv evaluation mode. The default value is 10.

• evaluate.results_dir: The results directory. Defaults to /results/evaluate.

Use the following command to run inference on ActionRecognitionNet with the .pth model.

            

            
tao model action_recognition inference -e <experiment_spec>
                             inference.checkpoint=<inference model>
                             inference.inference_dataset_dir=<path to dataset to be inferenced>
                             [inference.batch_size=<batch size>]
                             [inference.video_inf_mode=<inference >]
                             [inference.video_num_segments]
                             [inference.gpu_ids=<gpu indices>]
                             [inference.num_gpus=<number of gpus>]
        

The output is formatted as [video_sample_path] [labels list of inference segments in this video].

Multi-GPU inference is currently not supported for Action Recognition.

Required Arguments

• -e, --experiment_spec: The experiment spec file to set up inference. This can be the same as the training spec.

• inference.checkpoint: The .pth model to perform inference with.

• inference.inference_dataset_dir: The path to the dataset to perform inference with. It should be a class-level directory, as described in the Preparing the Dataset section.

Optional Arguments

• inference.batch_size: The batch size to perform inference in evaluation. The default value is 1.

• inference.video_inf_mode: The inference mode for the video:

  • center: Inference is performed on the middle part of frames in the video clip. This is the default mode.

  • conv: Inference is performed on a number of segments in a video clip. All the segment preidctions are kept in a label list.

• inference.video_num_segments: The number of segments sampled in a video clip for the conv inference mode.

• inference.results_dir: The results directory. Defaults to /results/inference.

The expected output for the fall class would be as follows:

            

            
/path/to/fall/video_1 [fall]
/path/to/fall/video_2 [fall]
...
        

Use the following command to export ActionRecognitionNet to .etlt format for deployment:

            

            
tao model action_recognition export -e <experiment_spec>
                             export.checkpoint=<tlt checkpoint to be exported>
                             [export.gpu_id=<gpu index>]
                             [export.onnx_file=<path to exported file>]
        

Required Arguments

• -e, --experiment_spec: The experiment spec file to set up export. This can be the same as the training spec.

• export.checkpoint: The .pth model to be exported.

Optional Arguments

• export.gpu_id: The GPU index used to run the export. We can specify the GPU index used to run export when the machine has multiple GPUs installed. Note that export can only run on a single GPU

• export.onnx_file: The path to save the exported model to. The default path is in the same directory of \*.pth model.

The deep learning and computer vision models that you trained can be deployed on edge devices, such as a Jetson Xavier, Jetson Nano, or Tesla, or in the cloud with NVIDIA GPUs. The exported \*.etlt model can be used in a stand-alone TensorRT inference sample or in DeepStream.

DeepStream SDK is a streaming analytic toolkit to accelerate building AI-based video analytic applications. TAO is integrated with DeepStream SDK, so models trained with TAO will work out of the box with Deepstream.

Deploying the ActionRecognitionNet in the DeepStream Sample

Once you get the .etlt ActionRecognitionNet model, you can deploy it into the DeepStream 3d-action-recognition sample app. Refer to the sample applications documentation for detailed steps to run action recogintion in DeepStream.

Running ActionRecognitionNet Inference on the Stand-Alone Sample

A stand-alone TensorRT inference sample is also provided. It consumes the TensorRT engine and supports running with 2D/3D input on images. The sample can be found on Github.

To use this sample, you need to generate the TensorRT engine out of a \*.etlt model using trtexec.

Using trtexec

For instructions on generating TensorRT engine using trtexec command, refer to trtexec guide for ActionRecognitionNet.

Usage of Inference Sample

After you get the TensorRT engine, you can deploy the engine in the stand-alone sample. Use the following command to run inference:

            

            
python ar_trt_inference.py --input_images_folder <path to input images folder> \
                           --trt_engine <path to tensorrt engine> \
                           [--center_crop] \
                           [--input_2d]
        

Required Arguments

• --input_images_folder: The path to input images folder. It should be a video_<n> level directory as described in the Preparing the Dataset. section.

• --trt_engine: The path to the TensorRT engine.

Optional Arguments

• --center_crop: Resizes the input images with a short side to 256 and center crops to a 224x224 area. If this flag is not set, the input images will be directly resized to 224x224.

• --input_2d: Set this flag if the engine is generated from a 2D model.

            

            
[frame_0, frame_1, frame_2, ..., frame_31]
[frame_1, frame_2, frame_3, ..., frame_32]
....