Skip to navigation
Vision AgentAgent WorkflowsIndustry-Specific ExamplesWarehouse Operations BlueprintDevelopment WorkflowSimulation and Synthetic Data Generation

Dataset Post-Processing

View as Markdown

Post-processing is also a crucial step to verify the correctness of all generated data and convert the ground truth data into a specific format compatible with the model.

Generated data for MTMC training includes as follows:

  • RGB images.
  • 2D & 3D bboxes.
  • Depth maps.
  • Camera calibration.

Prerequisites

With conda installed, run the following from the repository root on Ubuntu/Debian:

conda create -n postpro python=3.10 -y
conda activate postpro
python -m pip install -r tools/sdg-postprocessing/requirements.txt
sudo apt update && sudo apt install -y ffmpeg
ffmpeg -version

Post-processing is divided into two steps:

Data Sanity Check:

  • Sanity checks for RGB, depth maps, and ground truth focuses on identifying issues such as dropped frames or data corruption.
  • Sanity check for calibration focuses on identifying the correctness of calibration, including intrinsic matrix, extrinsic matrix, point pairs array, projection matrix, and homography matrix.
  • Sanity check for b frame focuses on identifying if the b-frames are present in a video.

Navigate to the tools/sdg-postprocessing/data_sanity_check path:

cd /path/to/video-search-and-summarization/tools/sdg-postprocessing/data_sanity_check/
# RGB image sanity check
python dataset_sanity_check_rgb.py --base_dir /path/to/dataset --total_frames 9000
# Depth map sanity checks
# Raw depth maps (.npy) and converted .png files should both be validated
python dataset_sanity_check_npy.py --base_dir /path/to/dataset --total_frames 9000
python dataset_sanity_check_png.py --base_dir /path/to/dataset --total_frames 9000
# Ground truth sanity check
python dataset_sanity_check_json.py --base_dir /path/to/dataset --total_frames 9000
# Calibration sanity check
mkdir -p /path/to/calibration-sanity-output
python dataset_sanity_check_calibration.py --base_dir /path/to/dataset --calibration /path/to/dataset/calibration.json --output_dir /path/to/calibration-sanity-output
# Video b-frame sanity check
bash dataset_sanity_check_videos.sh /path/to/dataset

Data Conversion:

Conversions for depth maps, RGB images, and ground truth are designed to save storage space and reduce redundant ground truth data, while schema conversion is performed for downstream tasks such as E2E model & pipeline evaluation.

Navigate to the tools/sdg-postprocessing/data_conversion path:

cd /path/to/video-search-and-summarization/tools/sdg-postprocessing/data_conversion/
# Depth map conversion
python convert_npy_to_png_depthmap.py /path/to/dataset
# Video conversion
bash convert_images_to_videos_no_bframes.sh /path/to/dataset
# Ground truth conversion
python convert_ground_truth.py /path/to/dataset --calibration /path/to/dataset/calibration.json --output /path/to/dataset/ground_truth --max_frames 9000 --xform_info /path/to/xform_info.json

Generate xform_info.json only when ground-truth conversion requires parent-Xform rotation overrides:

  1. Load the scene in Isaac Sim.

  2. Open the Script Editor.

    Script Editor location

  3. In the Script Editor, open tools/sdg-postprocessing/utils/export_xform_semantics.py via File > Open, and set USER_OUTPUT_PATH to your desired output directory or JSON file path.

  4. Click Run to export the JSON file to the configured path.

If the scene does not need parent-Xform rotation overrides, omit the --xform_info option from the ground-truth conversion command.