Simulation & Synthetic Data

Simulation and synthetic data generation (SDG) are increasingly central techniques to augment the dataset of perception models and to provide a digital twin for end-to-end development & validation of applications.

A powerful and developer-friendly tool is Isaac Sim coupled with the Omni.Replicator.Character (ORC) extension.

Briefly, you would need to do the following steps to make a use of synthetic data in Multi-Camera Tracking use cases (MTMC):

  • Generate synthetic data using the ORC extension in Isaac Sim.

  • Use the synthetic data generated to fine-tune the detection/re-identification models using TAO.

  • Deploy the fine-tuned models in the mdx-perception container to generate better MTMC results.

Best Practices for SDG

Ensure cameras are positioned to cover the entire floorplan comprehensively, maintaining continuous observation of all tracked targets. It is advisable to capture each person from 3 to 4 different camera perspectives for optimal data quality.

Videos should ideally last more than one minute. For looped playback data, it’s beneficial to have subjects return to their starting points at the end of the videos to enhance the consistency of the data.

Synthetic Data Augmentation

Augmenting your dataset with synthetic data is a scalable way to improve its diversity & robustness.

As an example, here’s a brief guide on fine-tuning our provided Re-Identification model on synthetic data.

Refer to the ReIdentification fine-tuning guide guide to retrain/fine-tune your network. The recommended route to fine-tune ReIdentificationNet on synthetic data is via API Notebook. The documentation provided below accompanies the cells in the API notebook and offers guidance specific to synthetic data obtained via ORC:

  1. Data Formatting Configuration cell: The data obtained from the ORC extension has to be converted to the Market-1501 format for ReIdentificationNet training. We recommend to start with a smaller number of samples. The default value of 200 images is suggested to start the training experiments. Additional samples of image can be added, once the accuracy is observed to saturate.

  2. ReID Training Configuration & Train ReID cell: It is extremely easy to overfit on synthetic data due to its uniformity in textures & lightning. For fine-tuning, train_num_epochs is set to 10 epochs. It is essential that the user monitors the training process for overfitting/underfitting by evaluating the final checkpoint on test data. If the overfitting is observed, the user can generate more samples incrementally & resume the training process to validate good results on test data. TAO API also provides finer controls on the configuration that can be modified. For more details about TAO ReIdentificationNet configurations refer to these docs.

For the remaining cells, follow the guide located here.

Once the model is exported successfully, refer to the Re-Identification section under model update to bring your model to MDX-Perception container.

Synthetic Data Testing

To leverage synthetic data for testing within multi-camera tracking frameworks, it is necessary to transform the output images from the ORC extension into video format, while also converting the associated ground-truth labels into the MOTChallenge format. Conversion utilities are available as an NGC resource. (If prompted, select nv-mdx as Org and mdx-v2-0 as Team).

For converting RGB images to videos, execute the following shell script command: bash convert_images_to_videos.sh <root dir> . This script converts images to video at a frame rate of 30 FPS and saves the output as video.mp4 in each video’s directory. Subsequently, compile these files into a dedicated video folder, renaming each to <sensor id>.mp4 ensuring sensor IDs correspond with the calibration file. Verify the successful conversion of all frame images by checking the frame count in each video file with: ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets -of csv=p=0 <video file>.

Additionally, a Python script, process_ground_truth_labels.py, is provided for transforming KITTI labels from the ORC extension into a unified ground-truth file in MOTChallenge format. Update the script with the correct root directory path before running it to generate a ground_truth.txt file, which will be saved in the specified root directory.