Sparse4D (3D multi-camera)

Model Card

Sparse4D is an advanced 3D Multi-Camera Detection and Tracking Network. We specifically adapt Sparse4D for indoor environments such as warehouses with static camera setups. It generates precise 3D bounding boxes and tracking IDs for a diverse set of objects across multiple camera views. The included model in the Perception Microservice is pre-trained on the MTMC Tracking 2025 subset from the Nvidia PhysicalAI-SmartSpaces dataset. The v2.3 and v3.0 models predict seven warehouse-centric classes, in class-ID order: Person, Fourier_GR1_T2_Humanoid, Agility_Digit_Humanoid, Nova_Carter, Transporter, Forklift, and Pallet_Truck. The v2.3 per-class evaluation tables report six evaluated classes, not the full model output taxonomy. The latest releases are v2.3, a stable model tuned for general and synthetic warehouse data (available in ResNet50 and ResNet101 backbones), and v3.0, which is better adapted to real-world datasets (ResNet50 backbone only). ResNet50 is the default backbone for VSS 3.3 (lower latency, smaller footprint); choose the ResNet101 v2.3 checkpoint when accuracy headroom matters more than throughput.

The Sparse4D model cards on NGC (sparse4d_rn50 and sparse4d_rn101) expand on data formats and evaluation methodology. The remainder of this page covers TAO fine-tuning (BEV grouping, dataset conversion, training, evaluation, inference, and export).

Use the model card’s Input and Raw ONNX/TensorRT outputs sections for the tensor interface of the matching model release. Raw engine outputs are not the decoded boxes_3d / scores_3d dictionaries. The released models use 900 queries and a separate 600-instance temporal cache; the TAO tracking decoder selects the top 300 current queries by classification confidence, with fewer detections possible after threshold filtering. See the model compatibility checklist before swapping models or reusing an engine.

Inference using Perception Microservice

Detailed information can be found in the 3D Multi-Camera Detection and Tracking (Sparse4D) page.

Model versions

The VSS 3.3 Sparse4D model cards list v2.3 and v3.0 as the latest trainable and deployable releases:

For new VSS 3.3 work, start from trainable_v2.3 (or deployable_v2.3) for general and synthetic-data deployments, or trainable_v3.0 (deployable_v3.0) when targeting real-world data; use an older version only to reproduce a prior benchmark. The ResNet50 backbone is the default for VSS 3.3 (lower latency, smaller footprint, built on top of an RT-DETR R50 pre-training initialization); pick the ResNet101 backbone (available for v2.3) when accuracy headroom matters more than throughput.

The Warehouse Operations Docker Compose profile selects ResNet50 v2.3 for DATASET_TYPE=synthetic and v3.0 for DATASET_TYPE=real; see the dataset-specific model paths. This selection does not apply to every deployment entry point: the standalone detection/tracking helper in skills/deployment/vss-deploy-detection-tracking-2d/assets/deploy-defaults.yml remains pinned to deployable_v2.2 and its v2.2 companion paths. Use the Warehouse Operations profile for the v2.3/v3.0 flow documented here; the standalone helper does not select these releases automatically.

Real-Time Inference Throughput & Latency

Inference runs through the DeepStream pipeline on TensorRT with mixed precision (FP16+FP32). The TensorRT columns capture model-only latency; the DeepStream columns are the sustainable stream counts once the DeepStream microservice’s instance-bank pre-/post-processing overhead is accounted for — i.e. the camera count whose model latency fits within roughly 70% of the frame budget. The table summarizes how many cameras each GPU (ResNet50 backbone) supports at 30, 15, and 10 FPS. Values are the conservative minimum across the v2.3 and v3.0 releases, which are latency-equivalent. Numbers in bold are measured TensorRT values. A value of 200 reflects the tested sweep ceiling and may be higher in practice.

Datacenter GPUs

Embedded GPUs

KPI

The key performance indicator is Higher Order Tracking Accuracy (HOTA) per-class and the average HOTA obtained across all classes. We utilize the HOTA evaluation methodology to assess multi-object tracking accuracy.

Higher Order Tracking Accuracy (HOTA) is a metric that balances detection accuracy and association accuracy into a single unified score. It decomposes tracking performance into Detection Accuracy (DetA), which measures how well objects are localized, and Association Accuracy (AssA), which measures how well detections are linked over time into consistent tracks. HOTA is computed as the geometric mean of DetA and AssA, averaged over a range of localization thresholds. This provides a balanced, holistic measure of tracking quality that captures both spatial accuracy and temporal consistency.

The following scores are for v2.3 models trained on the MTMC Tracking 2025 subset. The evaluation set and training set are disjoint. The averages cover the six evaluated classes shown in the tables; Pallet_Truck is an output class but is not included in these reported averages.

ResNet50 backbone (v2.3)

ResNet101 backbone (v2.3)

Results highlighted in the above tables are for the latest v2.3 models on the test set of the MTMC Tracking 2025 dataset. The v3.0 checkpoint is evaluated on real-world scenes; refer to the sparse4d_rn50 and sparse4d_rn101 model cards for those results and more details.

Fine-tuning using NVIDIA TAO Toolkit

Hardware & Software Requirements

Please refer to the Requirements section of the TAO Toolkit Quick Start Guide for more details.

Dataset Requirements

The Sparse4D model can be fine-tuned on existing MTMC Tracking 2025 scenes, your own synthetic dataset generated with Isaac Sim, or real-world imagery. For real-world data without 3D annotations, use the 2D-to-3D geometric distillation workflow to combine 2D supervision with 3D-labeled training data. Please refer to the Simulation and Synthetic Data Generation page for more details on synthetic data generation.

Performing 3D Multi-Camera Detection & Tracking in large regions such as warehouses with high-density cameras becomes computationally expensive. Therefore, we divide a region into a collection of groups called Bird’s Eye View (BEV) groups. Each Bird’s Eye View (BEV) group can contain multiple cameras, and these BEV groups may overlap with one another. Sparse4D is trained on such BEV groups. The below image represents a top down view of a single scene with multiple BEV groups. Each unique color represents a BEV group highlighting its camera coverage in the entire scene.

Requirements for the supervised multi-camera 3D workflow:

1. Number of Cameras in Scene

   • Refer to the camera placement guide for details based on scene size

2. Number of RGB Frames

   • Minimum: 9,000 frames per camera (5 minutes @ 30 FPS)

3. Number of Depth Map Frames

   • Optional but recommended for better model convergence
   • Should match the number of RGB frames if included

4. Number of BEV Groups (Bird’s Eye View)

   • We recommend a minimum of 3 BEV groups for a scene with 12 cameras in total.
   • The TAO Fine-tuning notebook will help generate randomized BEV groups based on the raw calibration data obtained from the SDG step.

5. Number of Cameras per BEV Group

   • Range: 4-12 cameras per group

6. Annotations

   • Ground Truth 3D & 2D bounding boxes in OV coordinates

7. Calibration Data

   • Single calibration file containing all camera parameters (intrinsic & extrinsic)

Refer to Dataset Post-Processing for the file format of the files listed above. An example dataset can be found in the Warehouse_014 scene from the MTMC Tracking 2025 dataset.

2D-to-3D Geometric Distillation for Real-World Datasets

2D-to-3D geometric distillation lets Sparse4D learn from calibrated multi-camera real-world images with 2D annotations or detector pseudo-labels, reducing the need to collect full 3D ground truth. The term 2D-to-3D describes the direction of supervision: 2D targets supervise the 3D model. During training, 3D-to-2D projection maps Sparse4D’s predicted 3D boxes into camera images, where they can be compared with the 2D targets. Sparse4D still predicts 3D boxes and tracking IDs.

For the complete TAO configuration and artifact-preparation procedure, refer to the TAO 2D-to-3D geometric distillation guide.

A projected 3D cuboid encloses all eight corners and can be larger than the object’s tight image-space box. The released geometry-only loose-to-tight (LTT) MLP corrects this difference toward tight, amodal 2D boxes. During Sparse4D fine-tuning, the MLP is frozen and its predicted correction factors are detached; gradients flow through the corrected box projection into Sparse4D. Per-camera matching then compares these boxes with real-world 2D detections, such as offline RT-DETR predictions, to supervise box geometry and classification.

Choose the supervision route

Keep 3D-labeled data in the training mix to retain metric geometry supervision.

Download the released LTT checkpoint

For the released seven-class warehouse taxonomy listed below, reuse _loose_to_tight_mlp.pth from the Sparse4D ResNet-50 trainable_v3.0 bundle on NGC. With the NGC CLI, download just the LTT file:

mkdir -p /data
ngc registry model download-version nvidia/tao/sparse4d_rn50:trainable_v3.0 \
  --file _loose_to_tight_mlp.pth --dest /data

This writes /data/sparse4d_rn50_vtrainable_v3.0/_loose_to_tight_mlp.pth, including the leading underscore in the filename. Make that path visible inside the TAO training environment and use it for model.head.loose_to_tight.mlp_ckpt, as shown below. Keep the checkpoint’s ordered class taxonomy unchanged and validate its corrections on the target data. Continue with teacher-cache preparation and mixed-training data preparation below.

This workflow supports only the released checkpoint’s seven classes in their original order. Custom taxonomies require a matching LTT checkpoint and are outside this guide; changing class names in the configuration does not adapt the released weights.

Retain the external LTT file for fine-tuning and resume: the frozen MLP is excluded from Sparse4D model checkpoints. Standard evaluation, inference, and export do not require it.

Prepare the training data

1. Prepare 2D supervision. Use operation=ltt_2dgt for visible-2D sidecars on 3D-labeled scenes. For calibrated real scenes, run the 2D teacher offline, then use operation=rtdetr_2d to convert its per-camera KITTI labels.tar.gz archives to <scene>__rtdetr2d.npz. This operation converts detections; it does not run RT-DETR. Before building the mixed split, create matching image/calibration info PKLs with the calibrated real-scene conversion command below.
2. Build the mixed training split. List the converted 3D-labeled and calibrated real info PKLs once each. Mark genuinely unlabeled 3D samples with gt_boxes=None, rather than treating them as annotated empty scenes. Run operation=lazy_index after finalizing this split if using dataset.lazy_load: true.

Use the same ordered taxonomy for data preparation class_names, training dataset.classes, the LTT checkpoint, and all caches. For the released seven-class warehouse models, the training names are person, gr1_t2, agility_digit, nova_carter, transporter, forklift, and pallet_truck. Map teacher labels explicitly to this taxonomy; a matching class count alone is insufficient. Camera names, source frame IDs, image dimensions, and coordinate conventions must agree between the info PKLs and their caches.

Prepare calibrated real-scene inputs

Use annotations convert in TAO Data Services with the unlabeled-scene spec. Copy the spec to /specs/aicity2ovpkl_unlabeled.yaml and set its ordered aicity.class_config.CLASS_LIST to match your model and teacher caches. This requires a build that exposes aicity.load_annotations and aicity.fps.

Arrange each scene as /data/real/train/<scene>/calibration.json plus <camera>/rgb/000000000.jpg, 000000001.jpg, and so on. NVSchema calibration must provide camera intrinsics and world-to-camera poses with IDs matching the image directories. The rgb_00000.jpg image naming convention also works. Use synchronized, constant-rate frames, contiguous from zero, and provide the actual capture rate:

annotations convert -e /specs/aicity2ovpkl_unlabeled.yaml \
  aicity.root=/data/real aicity.split=train aicity.fps=30 \
  results_dir=/data/real_infos

The spec sets aicity.load_annotations=false, keeps the world frame without recentering, and uses all calibrated cameras as one group per scene. It writes /data/real_infos/train/<scene>_infos_train.pkl with synchronized image paths, camera geometry, sequence/frame identifiers, timestamps in seconds, and gt_boxes=None. Ground truth and depth files are not required, and anchor initialization is skipped. Reuse the pretrained model’s anchors and keep image paths visible at the same location inside the training environment.

Each PKL stores frame records in infos. Each record retains token, scene_name, timestamp, cams, and gt_boxes=None for unlabeled frames. The cams mapping is keyed by camera ID; each camera entry retains its data_path. Preserve these field names and conventions:

sensor2world_transform is a legacy field name. The loader uses this matrix directly with the camera intrinsics to project 3D points; it does not invert it. If BEV-group recentering is enabled, the matrix must instead map the corresponding group-local 3D coordinates into the camera frame.

For HDF5 image arrays, set aicity.rgb_format=h5 and provide <camera>.h5 files with HWC arrays at rgb/rgb_00000.jpg, rgb/rgb_00001.jpg, and so on. Set dataset.use_h5_file_for_rgb: true for these inputs during training.

The converter rejects empty, gapped, or unequal camera sequences. Synchronize irregular or dropped-frame inputs before conversion and run the 2D teacher on those same frames. Inspect projections and cache joins before training; keep calibration consistent with image dimensions and the metric world frame.

The existing camera-grouping options can generate multiple BEV groups. Use one info PKL per group, with camera names and source frame IDs matching the teacher cache. For example, a PKL scene named RealWarehouse+bev0 uses /data/rtdetr/RealWarehouse__rtdetr2d.npz: the loader strips the BEV-group suffix when locating the cache. Include each PKL once in /data/mixed_train_split.txt alongside the 3D-labeled PKLs, then run operation=lazy_index using the final split.

Configure mixed 3D and real-world training

Merge the following settings into a complete Sparse4D experiment spec for your checkpoint. Replace the paths and RealWarehouse with your real scene-name substring. Retain the matching backbone, anchors, class order, and model input size from the base spec.

model:
  cotrain_param_touch: false
  head:
    loose_to_tight:
      enable: true
      mlp_ckpt: /data/sparse4d_rn50_vtrainable_v3.0/_loose_to_tight_mlp.pth
      num_classes: 0  # Derive from dataset.classes.
      loss_weight: 0.1
      pseudo_enable: true
      pseudo_box_weight: 0.1
      pseudo_cls_weight: 1.0
      min_cams: 1
dataset:
  classes: [person, gr1_t2, agility_digit, nova_carter, transporter, forklift, pallet_truck]
  train_dataset:
    ann_file: /data/mixed_train_split.txt
  lazy_load: true
  ltt_2dgt_sidecar_dir: /data/ltt_2dgt
  rtdetr_2d_cache_dir: /data/rtdetr
  rtdetr_2d_mark_real: true
  sync_route: true
  real_scene_keywords: [RealWarehouse]
  real_block_prob: 0.5
  scene_switch_iters: 100

real_block_prob: 0.5 is an example starting point for the proportion of 2D-route blocks, not an accuracy recommendation. Distributed co-training requires sync_route: true and a positive scene_switch_iters; each batch must have homogeneous 2D/3D supervision. rtdetr_2d_mark_real: true makes any scene with a teacher cache use the 2D-only route, so keep those caches separate from scenes intended for full 3D supervision.

Run the regular sparse4d train -e /specs/experiment_spec.yaml command inside the TAO PyTorch environment. This workflow uses the training task; there is no separate Sparse4D distill task. Refer to the TAO Sparse4D documentation for the standard training, evaluation, and export workflow.

Validate and deploy

Inspect projected boxes on real images before a long run. Verify that expected labeled frames join their caches and that loss_box_2d_pseudo_* and loss_cls_pseudo_* contribute on calibrated real batches. Missing cache joins skip pseudo supervision and must not be interpreted as successfully processed empty detections.

Compare the original checkpoint and the adapted checkpoint on held-out, 3D-labeled real scenes, and retain a synthetic validation set to check for regressions. Report detection and tracking metrics separately; a lower 2D loss alone does not establish better metric 3D accuracy or tracking.

RT-DETR pseudo-label generation and the LTT MLP are training components. Standard Sparse4D evaluation, inference, and export do not require the external LTT checkpoint. Export the adapted Sparse4D model and follow the model compatibility requirements when deploying it. The deployed multi-camera 3D pipeline still needs real camera calibration.

Camera Grouping for Model Fine-tuning

For Sparse4D fine-tuning, cameras need to be organized into BEV groups with overlapping fields of view. Unlike camera clustering (used for deployment, where each camera belongs to exactly one cluster), camera grouping for training allows cameras to appear in multiple groups to maximize training data coverage.

Use create_camera_groups.py to create camera groups with duplication support based on overlapping FOV:

We use the spatialai_data_utils library to create camera groups:

pip install spatialai-data-utils==2.0.1 --extra-index-url=https://edge.urm.nvidia.com/artifactory/api/pypi/sw-metropolis-pypi/simple

The camera grouping tools are shipped with the full spatialai-data-utils package in the VSS repository.

Navigate to the spatialai-data-utils camera grouping tools

export VSS_REPO_DIR=/path/to/video-search-and-summarization
cd $VSS_REPO_DIR/libs/analytics/spatialai-data-utils/tools/camera_grouping/

# Auto mode (recommended for training): one group each of sizes
# 1, 2, ..., min(n_sensors, max_sensors_per_group). Overrides --cameras_per_group.
python create_camera_groups.py data/scene --auto
# Auto mode with a custom cap on the largest group size (default 18)
python create_camera_groups.py data/scene --auto --max_sensors_per_group 12
# Manual mode: 5 groups with 8 cameras each
python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8
# Multiple size types: 2 groups × 3 sizes = 6 total groups
# (2 groups with 5 cameras, 2 groups with 8 cameras, 2 groups with 6 cameras)
python create_camera_groups.py data/scene --n_groups 2 --cameras_per_group 5 8 6
# With visualization (generates separate images per group by default)
python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 --visualize
# Custom overlap threshold for stricter FOV requirements
python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 \
    --min_overlap_threshold 0.3 --visualize
# Use existing FOV polygons from calibration instead of frustum calculation
python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 \
    --prefer_existing_fov

Use --help to list all available arguments.

Key options:

• --auto: Auto mode — set --cameras_per_group to [1, 2, …, min(n_sensors, --max_sensors_per_group)], creating groups for every size (recommended for training). Overrides --cameras_per_group.
• --n_groups: Number of groups to create per size (default: 1).
• --cameras_per_group: Number of cameras per group (required unless --auto is used). Single value: all groups have that size. Multiple values: creates n_groups for EACH size (total = n_groups × count).
• --max_sensors_per_group: Maximum sensors per group (default: 18). In auto mode, caps the largest generated group size.
• --output: Output path for the grouped calibration file.
• --output_suffix: Suffix for output files (default: “grouped”).
• --map_file: Path to map image for visualization (uses black background if not provided).
• --start_camera_index: Starting camera index for seeding the first group (default: 0).
• --min_overlap_threshold: Minimum FOV overlap (0-1) for group membership (default: 0.2).
• --max_distance_threshold: Maximum centroid distance in meters for membership (default: inf).
• --prefer_existing_fov: Use existing FOV polygons instead of calculating from frustum.
• --max_camera_distance: Maximum distance in meters for frustum calculation (default: 30.0).
• --height_range: Height range (min, max) in meters for ground plane intersection (default: 1.0 3.0).
• --image_size: Image dimensions (width, height) in pixels for frustum calculation (default: 1920 1080).
• --dilation: Buffer distance in meters for group bounding boxes (default: 8.0).
• --visualize: Generate visualization of camera groups.
• --vis_no_camera_id_labels: Disable drawing camera IDs on the visualization.
• --vis_combined: Generate a single combined image instead of separate images per group.

Expected time for finetuning

The expected data requirements and time to fine-tune the Sparse4D model on a single scene of the MTMC Tracking 2025 dataset are as follows:

We recommend a multi-gpu training setup for faster training.

TAO Fine-tuning via Notebook Walkthrough & Configuration Changes

Sparse4D can be fine-tuned via the TAO containers and the TAO CLI Notebook.

The documentation provided below accompanies the cells in the TAO fine-tuning notebook and offers guidance on how to execute them. TAO Sparse4D supports the following tasks via the Jupyter notebook:

• dataset convert
• model train
• model evaluate
• model inference
• model export

An experiment specification file (also known as a configuration file) is used for fine-tuning the model. It consists of several main components:

• dataset
• model
• train
• evaluate
• inference
• export
• visualize

The notebook utilizes the Warehouse_014 scene from the MTMC Tracking 2025 dataset for training.

The following is an example spec file for training a Sparse4D model on one scene of the MTMC Tracking 2025 dataset. For field-level detail, use the TAO Sparse4D configuration documentation that matches your TAO Toolkit release.

results_dir: /results
train:
  num_gpus: 1
  gpu_ids:
  - 0
  num_nodes: 1
  seed: 1234
  cudnn:
    benchmark: false
    deterministic: true
  num_epochs: 10
  checkpoint_interval: 0.5
  validation_interval: 0.5
  resume_training_checkpoint_path: null
  results_dir: null
  pretrained_model_path: null
  optim:
    type: adamw
    lr: 5.0e-05
    weight_decay: 0.001
    momentum: 0.9
    paramwise_cfg:
      custom_keys:
        img_backbone:
          lr_mult: 0.2
    grad_clip:
      max_norm: 25
      norm_type: 2
    lr_scheduler:
      policy: cosine
      warmup: linear
      warmup_iters: 500
      warmup_ratio: 0.333333
      min_lr_ratio: 0.001
model:
  type: sparse4d
  embed_dims: 256
  use_grid_mask: true
  use_deformable_func: true
  input_shape:
  - 960
  - 540
  backbone:
    type: resnet_101
  neck:
    type: FPN
    num_outs: 4
    start_level: 0
    out_channels: 256
    in_channels:
    - 256
    - 512
    - 1024
    - 2048
    add_extra_convs: on_output
    relu_before_extra_convs: true
  depth_branch:
    type: dense_depth
    embed_dims: 256
    num_depth_layers: 3
    loss_weight: 0.2
  head:
    type: sparse4d
    num_output: 300
    cls_threshold_to_reg: 0.05
    decouple_attn: true
    return_feature: true
    use_reid_sampling: false
    embed_dims: 256
    reid_dims: 0
    num_groups: 8
    num_decoder: 6
    num_single_frame_decoder: 1
    drop_out: 0.1
    temporal: true
    with_quality_estimation: true
    operation_order:
    - deformable
    - ffn
    - norm
    - refine
    - temp_gnn
    - gnn
    - norm
    - deformable
    - ffn
    - norm
    - refine
    - temp_gnn
    - gnn
    - norm
    - deformable
    - ffn
    - norm
    - refine
    - temp_gnn
    - gnn
    - norm
    - deformable
    - ffn
    - norm
    - refine
    - temp_gnn
    - gnn
    - norm
    - deformable
    - ffn
    - norm
    - refine
    - temp_gnn
    - gnn
    - norm
    - deformable
    - ffn
    - norm
    - refine
    visibility_net:
      type: visibility_net
      embedding_dim: 256
      hidden_channels: 32
    instance_bank:
      num_anchor: 900
      anchor: ''
      num_temp_instances: 600
      confidence_decay: 0.8
      feat_grad: false
      default_time_interval: 0.033333
      embed_dims: 256
      use_temporal_align: false
      grid_size: null
    anchor_encoder:
      type: SparseBox3DEncoder
      vel_dims: 3
      embed_dims:
      - 128
      - 32
      - 32
      - 64
      mode: cat
      output_fc: false
      in_loops: 1
      out_loops: 4
      pos_embed_only: false
    sampler:
      num_dn_groups: 5
      num_temp_dn_groups: 3
      dn_noise_scale:
      - 2.0
      - 2.0
      - 2.0
      - 0.5
      - 0.5
      - 0.5
      - 0.5
      - 0.5
      - 0.5
      - 0.5
      - 0.5
      max_dn_gt: 128
      add_neg_dn: true
      cls_weight: 2.0
      box_weight: 0.25
      reg_weights:
      - 2.0
      - 2.0
      - 2.0
      - 0.5
      - 0.5
      - 0.5
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      - 0.0
      use_temporal_align: false
      gt_assign_threshold: 0.5
    reg_weights:
    - 2.0
    - 2.0
    - 2.0
    - 1.0
    - 1.0
    - 1.0
    - 1.0
    - 1.0
    - 1.0
    - 1.0
    - 1.0
    loss:
      cls:
        type: focal
        use_sigmoid: true
        gamma: 2.0
        alpha: 0.25
        loss_weight: 2.0
      reg:
        type: sparse_box_3d
        box_weight: 0.25
        cls_allow_reverse: []
      id:
        type: cross_entropy_label_smooth
        num_ids: 70
    bnneck:
      type: bnneck
      feat_dim: 256
      num_ids: 70
    deformable_model:
      embed_dims: 256
      num_groups: 8
      num_levels: 4
      attn_drop: 0.15
      use_deformable_func: true
      use_camera_embed: true
      residual_mode: cat
      num_cams: 6
      max_num_cams: 20
      proj_drop: 0.0
      kps_generator:
        embed_dims: 256
        num_learnable_pts: 6
        fix_scale:
        - - 0
          - 0
          - 0
        - - 0.45
          - 0
          - 0
        - - -0.45
          - 0
          - 0
        - - 0
          - 0.45
          - 0
        - - 0
          - -0.45
          - 0
        - - 0
          - 0
          - 0.45
        - - 0
          - 0
          - -0.45
    refine_layer:
      type: sparse_box_3d_refinement_module
      embed_dims: 256
      refine_yaw: true
      with_quality_estimation: true
    valid_vel_weight: -1.0
    graph_model:
      type: MultiheadAttention
      embed_dims: 512
      num_heads: 8
      batch_first: true
      dropout: 0.1
    temp_graph_model:
      type: MultiheadAttention
      embed_dims: 512
      num_heads: 8
      batch_first: true
      dropout: 0.1
    decoder:
      type: SparseBox3DDecoder
      score_threshold: 0.05
    norm_layer:
      type: LN
      normalized_shape: 256
    ffn:
      type: AsymmetricFFN
      in_channels: 512
      pre_norm:
        type: LN
        normalized_shape: 256
      embed_dims: 256
      feedforward_channels: 1024
      num_fcs: 2
      ffn_drop: 0.1
      act_cfg:
        type: ReLU
        inplace: true
  use_temporal_align: false
inference:
  num_gpus: 1
  gpu_ids:
  - 0
  num_nodes: 1
  checkpoint: ???
  trt_engine: null
  results_dir: null
  jsonfile_prefix: sparse4d_pred
  output_nvschema: true
  tracking:
    enabled: true
    threshold: 0.2
evaluate:
  num_gpus: 1
  gpu_ids:
  - 0
  num_nodes: 1
  checkpoint: ${results_dir}/train/sparse4d_model_latest.pth
  trt_engine: null
  results_dir: null
  metrics:
  - detection
  tracking:
    enabled: true
    threshold: 0.2
export:
  results_dir: null
  gpu_id: 0
  checkpoint: ???
  onnx_file: ???
  on_cpu: false
  input_channel: 3
  input_width: 960
  input_height: 544
  opset_version: 17
  batch_size: -1
  verbose: false
  format: onnx
visualize:
  show: true
  vis_dir: ./vis
  vis_score_threshold: 0.25
  n_images_col: 6 # Update this based on the number of cameras in the scene
  viz_down_sample: 3

Dataset Preparation

The data obtained from HuggingFace is stored in AICity format. We need to convert the AICity format to the OVPKL format required for model training. TAO Data Service can be used to convert the AICity format to the OVPKL format required for model training. The following configuration file can be used to convert the AICity format to the OVPKL format.

data:
  input_format: "AICity"
  output_format: "OVPKL"
aicity:
  root: ??? # path to your dataset root directory
  version: "2025"
  split: "train"
  class_config:
    CLASS_LIST:
      - "Person"
      - "FourierGR1T2"
      - "AgilityDigit"
      - "Transporter"
    SUB_CLASS_DICT: {}
    MAP_CLASS_NAMES:
      "Person": "Person"
      "FourierGR1T2": "FourierGR1T2"
      "AgilityDigit": "AgilityDigit"
      "Transporter": "Transporter"
    ATTRIBUTE_DICT:
      "Person": "person.moving"
      "FourierGR1T2": "fourier_gr1_t2.moving"
      "AgilityDigit": "agility_digit.moving"
      "Transporter": "transporter.moving"
    CLASS_RANGE_DICT:
      "Person": 40
      "FourierGR1T2": 40
      "AgilityDigit": 40
      "Transporter": 40
  recentering: true
  rgb_format: "mp4" # For datasets obtained from HuggingFace, RGB images are stored in "mp4" format. For SDG generated dataset, RGB images are stored in "h5" format.
  depth_format: "h5" # Depth maps are stored in "h5" both for HuggingFace & SDG generated datasets.
  camera_grouping_mode: "train"
  num_frames: -1
  anchor_init_config:
    output_file_name: "anchor_init_kmeans900.npy"
results_dir: ??? # Path to where you want to store the data

Once the configurations are modified, launch dataset conversion by running the following command:

Launch Model Fine-tuning

Below are the important configurations that need to be updated to launch training:

dataset:
  use_h5_file: true
  num_frames: 9000
  batch_size: 4 # Depending on your GPU memory
  num_bev_groups: 3 # Depending on your scene size
  num_workers: 4 # Depending on your CPU memory
  num_ids: 70 # Depending on your dataset
  classes: [ # Update this based on your dataset
    "person",
    "gr1_t2",
    "agility_digit",
    "nova_carter",
    "transporter",
    "forklift",
  ]
  data_root: ??? # Update this based on your dataset root directory
  train_dataset:
    ann_file: ??? # Update this based on your training dataset annotations file/folder.
  val_dataset:
    ann_file: ??? # Update this based on your validation dataset annotations file/folder.
  test_dataset:
    ann_file: ??? # Update this based on your test dataset annotations file/folder.

model:
  head:
    instance_bank:
      anchor: ??? # Update this to the anchor file generated by the TAO Fine-tuning notebook.

train:
  num_epochs: 5 # Update this based on your dataset size. 5 epochs is recommended for fine-tuning on a single scene.
  num_nodes: 1 # Update this based on your GPU count
  num_gpus: 1 # Update this based on your GPU count
  validation_interval: 1 # Update this based on your dataset size
  checkpoint_interval: 1 # Update this based on your dataset size
  pretrained_model_path: ??? # Update this to the pretrained model path if you are fine-tuning on a pretrained model.
  optim:
    lr: 0.0001 # Update this based on your dataset size
    paramwise_cfg:
      custom_keys:
        img_backbone:
          lr_mult: 0.25

Once the configurations are modified, launch training by running the following command:

Evaluate the fine-tuned model

The fine-tuned model’s tracking accuracy can be evaluated using the Higher Order Tracking Accuracy (HOTA) metric, decomposed as Detection Accuracy (DetA), Association Accuracy (AssA), and Localization Accuracy (LocA). Please refer to the KPI section above for more details on the metric definition and how to interpret per-class scores.

Below are the important configurations that need to be updated to launch evaluation:

evaluate:
  checkpoint: ${results_dir}/train/sparse4d_model_latest.pth # Update this based on your fine-tuned model path

dataset:
  test_dataset:
    ann_file: ??? # Update this based on your test dataset annotations file/folder.

Run Inference on the fine-tuned model using TAO

The fine-tuned model’s output can be inspected by running inference and visualizing the 3D bounding boxes and their tracking IDs. Below are the important configurations that need to be updated to launch inference:

inference:
  checkpoint: ??? # Update this based on your fine-tuned model path

dataset:
  test_dataset:
    ann_file: ??? # Update this based on your test dataset annotations file/folder.

visualize:
  show: true
  vis_dir: ./vis
  vis_score_threshold: 0.25
  n_images_col: 6 # Update this based on the number of cameras in the scene
  viz_down_sample: 3

Once the configurations are modified, launch inference by running the following command:

Export the fine-tuned model

Below are the important configurations that need to be updated to launch export:

export:
  results_dir: ??? # Update this based on your results directory
  checkpoint: ??? # Update this based on your fine-tuned model path
  onnx_file: ??? # Update this based on your desired model name

Once the configurations are modified, launch export by running the following command:

Using the fine-tuned model in the Perception Microservice

Once the model is fine-tuned and exported to ONNX format, it can be used in the Perception Microservice. Refer to the Integrating a Sparse4D Model Checkpoint section for more details.