> This page is for version 3.3.0 (default). > For other versions, use one of these documentation indexes: > - 3.3.0 (default): https://docs.nvidia.com/vss/3.3.0/llms.txt > - Dev: https://docs.nvidia.com/vss/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/vss/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/vss/_mcp/server. # Sparse4D (3D multi-camera) ## Model Card Sparse4D is an advanced 3D Multi-Camera Detection and Tracking Network. We specifically adapt Sparse4D for indoor environments such as warehouses with static camera setups. It generates precise 3D bounding boxes and tracking IDs for a diverse set of objects across multiple camera views. The included model in the Perception Microservice is pre-trained on the MTMC Tracking 2025 subset from the [Nvidia PhysicalAI-SmartSpaces dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces/tree/main). The v2.3 and v3.0 models predict seven warehouse-centric classes, in class-ID order: `Person`, `Fourier_GR1_T2_Humanoid`, `Agility_Digit_Humanoid`, `Nova_Carter`, `Transporter`, `Forklift`, and `Pallet_Truck`. The v2.3 per-class evaluation tables report six evaluated classes, not the full model output taxonomy. The latest releases are `v2.3`, a stable model tuned for general and synthetic warehouse data (available in `ResNet50` and `ResNet101` backbones), and `v3.0`, which is better adapted to real-world datasets (`ResNet50` backbone only). `ResNet50` is the default backbone for VSS 3.3 (lower latency, smaller footprint); choose the `ResNet101` v2.3 checkpoint when accuracy headroom matters more than throughput. ![TAO Sparse4D Model Output](https://raw.githubusercontent.com/vpraveen-nv/model_card_images/main/cv/purpose_built_models/sparse4d/sample_output.gif) The Sparse4D model cards on NGC ([sparse4d_rn50](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn50) and [sparse4d_rn101](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn101)) expand on data formats and evaluation methodology. The remainder of this page covers TAO fine-tuning (BEV grouping, dataset conversion, training, evaluation, inference, and export). Use the model card's **Input** and **Raw ONNX/TensorRT outputs** sections for the tensor interface of the matching model release. Raw engine outputs are not the decoded `boxes_3d` / `scores_3d` dictionaries. The released models use 900 queries and a separate 600-instance temporal cache; the TAO tracking decoder selects the top 300 current queries by classification confidence, with fewer detections possible after threshold filtering. See the [model compatibility checklist](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/microservice-configuration/perception/3d-multi-camera-detection-and-tracking-sparse4d#model-compatibility-requirements) before swapping models or reusing an engine. ![TAO Sparse4D Model Architecture](/vss/_fern-img/f7cb656a81c0f386cc8f9718de05aebf843d3417762dc10b3104bea9233400b5.webp) ## Inference using Perception Microservice Detailed information can be found in the [3D Multi-Camera Detection and Tracking (Sparse4D)](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/microservice-configuration/perception/3d-multi-camera-detection-and-tracking-sparse4d) page. ## Model versions The VSS 3.3 Sparse4D model cards list **v2.3** and **v3.0** as the latest trainable and deployable releases: | Model version | Release date | Notes | | ------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `v1.0` | 06/13/2025 | First commercial warehouse-focused 3D detection and tracking release | | `v1.1` | 11/30/2025 | Updated 960x540 input resolution, camera embeddings, broader training set, and lower-FPS robustness | | `v2.0` | 01/23/2026 | Expanded BEV groups, added COSMOS-augmented datasets, and improved multi-FPS training | | `v2.1` | 02/27/2026 | Expanded data coverage, SDU v1.2 camera grouping, and corrected forklift orientation annotations | | `v2.2` | 05/12/2026 | `trainable_v2.2` / `deployable_v2.2` artifacts. Released for both backbones (`ResNet50` and `ResNet101`); added `pallet_truck` to the trained class set; expanded the training corpus to all available MTMC Tracking 2025 scenes (3K+ BEV groups); refreshed COSMOS-augmented data; refreshed kmeans anchor priors per backbone. | | `v2.3` | 10/05/2026 | Stable release tuned for general and synthetic warehouse data; available for both `ResNet50` and `ResNet101` backbones. | | `v3.0` | 10/05/2026 | Better adapted to real-world datasets; `ResNet50` backbone only. | For new VSS 3.3 work, start from `trainable_v2.3` (or `deployable_v2.3`) for general and synthetic-data deployments, or `trainable_v3.0` (`deployable_v3.0`) when targeting real-world data; use an older version only to reproduce a prior benchmark. The `ResNet50` backbone is the default for VSS 3.3 (lower latency, smaller footprint, built on top of an RT-DETR R50 pre-training initialization); pick the `ResNet101` backbone (available for v2.3) when accuracy headroom matters more than throughput. The Warehouse Operations Docker Compose profile selects ResNet50 v2.3 for `DATASET_TYPE=synthetic` and v3.0 for `DATASET_TYPE=real`; see the [dataset-specific model paths](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/microservice-configuration/perception/3d-multi-camera-detection-and-tracking-sparse4d#dataset-specific-model-paths). This selection does not apply to every deployment entry point: the standalone detection/tracking helper in `skills/deployment/vss-deploy-detection-tracking-2d/assets/deploy-defaults.yml` remains pinned to `deployable_v2.2` and its v2.2 companion paths. Use the Warehouse Operations profile for the v2.3/v3.0 flow documented here; the standalone helper does not select these releases automatically. ## Real-Time Inference Throughput & Latency Inference runs through the DeepStream pipeline on TensorRT with mixed precision (FP16+FP32). The TensorRT columns capture model-only latency; the DeepStream columns are the sustainable stream counts once the DeepStream microservice's instance-bank pre-/post-processing overhead is accounted for — i.e. the camera count whose model latency fits within roughly 70% of the frame budget. The table summarizes how many cameras each GPU (`ResNet50` backbone) supports at 30, 15, and 10 FPS. Values are the **conservative minimum across the v2.3 and v3.0 releases**, which are latency-equivalent. Numbers in **bold** are measured TensorRT values. A value of `200` reflects the tested sweep ceiling and may be higher in practice. #### Datacenter GPUs | GPU | TensorRT @30 FPS | DeepStream @30 FPS | TensorRT @15 FPS | DeepStream @15 FPS | TensorRT @10 FPS | DeepStream @10 FPS | | ---------------------------------------- | ---------------- | ------------------ | ---------------- | ------------------ | ---------------- | ------------------ | | 1 x A100 SXM4 80GB | **20** | 13 | **39** | 29 | **60** | 41 | | 1 x B200 | **78** | 53 | **157** | 109 | **200** | 165 | | 1 x GB200 | **103** | 70 | **200** | 143 | **200** | 200 | | 1 x GB300 | **111** | 71 | **200** | 148 | **200** | 200 | | 1 x H100 80GB HBM3 | **33** | 22 | **71** | 48 | **108** | 75 | | 1 x H100 NVL | **28** | 19 | **59** | 40 | **89** | 62 | | 1 x H200 | **37** | 25 | **77** | 53 | **117** | 81 | | 1 x L4 | **4** | 3 | **10** | 6 | **15** | 11 | | 1 x L40 | **11** | 7 | **22** | 16 | **33** | 23 | | 1 x L40S | **14** | 9 | **29** | 20 | **41** | 31 | | 1 x RTX 6000 Ada Generation | **10** | 8 | **24** | 17 | **34** | 25 | | 1 x RTX PRO 4500 Blackwell (Server) | **13** | 9 | **27** | 19 | **41** | 29 | | 1 x RTX PRO 6000 Blackwell (Server) | **29** | 20 | **59** | 41 | **90** | 63 | | 1 x RTX PRO 6000 Blackwell (Workstation) | **30** | 21 | **62** | 43 | **93** | 65 | #### Embedded GPUs | GPU | TensorRT @30 FPS | DeepStream @30 FPS | TensorRT @15 FPS | DeepStream @15 FPS | TensorRT @10 FPS | DeepStream @10 FPS | | ---------------------------------- | ---------------- | ------------------ | ---------------- | ------------------ | ---------------- | ------------------ | | 1 x DGX Spark | **5** | 3 | **11** | 7 | **17** | 12 | | 1 x IGX Thor (iGPU) | **5** | 4 | **11** | 8 | **17** | 12 | | 1 x IGX Thor + RTX PRO 6000 (dGPU) | **21** | 14 | **42** | 29 | **63** | 44 | | 1 x Jetson AGX Orin | **3** | 1 | **6** | 4 | **10** | 7 | | 1 x Jetson AGX Thor | **6** | 4 | **12** | 8 | **19** | 13 | ## KPI The key performance indicator is Higher Order Tracking Accuracy (HOTA) per-class and the average HOTA obtained across all classes. We utilize the HOTA evaluation methodology to assess multi-object tracking accuracy. Higher Order Tracking Accuracy (HOTA) is a metric that balances detection accuracy and association accuracy into a single unified score. It decomposes tracking performance into Detection Accuracy (DetA), which measures how well objects are localized, and Association Accuracy (AssA), which measures how well detections are linked over time into consistent tracks. HOTA is computed as the geometric mean of DetA and AssA, averaged over a range of localization thresholds. This provides a balanced, holistic measure of tracking quality that captures both spatial accuracy and temporal consistency. The following scores are for v2.3 models trained on the MTMC Tracking 2025 subset. The evaluation set and training set are disjoint. The averages cover the six evaluated classes shown in the tables; `Pallet_Truck` is an output class but is not included in these reported averages. #### ResNet50 backbone (v2.3) | Object Class | HOTA | DetA | AssA | LocA | | ----------------------- | ----- | ----- | ----- | ----- | | Person | 50.70 | 46.35 | 56.94 | 67.68 | | Fourier_GR1_T2_Humanoid | 48.80 | 42.89 | 58.12 | 68.17 | | Agility_Digit_Humanoid | 53.92 | 50.23 | 59.65 | 68.37 | | Nova_Carter | 53.76 | 50.76 | 59.18 | 68.53 | | Transporter | 36.20 | 31.43 | 45.01 | 64.91 | | Forklift | 47.86 | 41.12 | 67.94 | 65.12 | | Average | 48.54 | 43.80 | 57.81 | 67.13 | #### ResNet101 backbone (v2.3) | Object Class | HOTA | DetA | AssA | LocA | | ----------------------- | ----- | ----- | ----- | ----- | | Person | 51.85 | 47.58 | 58.21 | 68.12 | | Fourier_GR1_T2_Humanoid | 47.24 | 42.30 | 55.14 | 67.38 | | Agility_Digit_Humanoid | 54.70 | 51.48 | 60.03 | 69.18 | | Nova_Carter | 53.83 | 50.88 | 59.11 | 68.45 | | Transporter | 38.24 | 32.54 | 49.25 | 64.94 | | Forklift | 46.43 | 41.08 | 60.77 | 66.36 | | Average | 48.71 | 44.31 | 57.08 | 67.40 | Results highlighted in the above tables are for the latest v2.3 models on the test set of the MTMC Tracking 2025 dataset. The v3.0 checkpoint is evaluated on real-world scenes; refer to the [sparse4d_rn50](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn50) and [sparse4d_rn101](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn101) model cards for those results and more details. ## Fine-tuning using NVIDIA TAO Toolkit ### Hardware & Software Requirements Please refer to the [Requirements section](https://docs.nvidia.com/tao/tao-toolkit/latest/text/getting_started.html) of the TAO Toolkit Quick Start Guide for more details. ### Dataset Requirements The Sparse4D model can be fine-tuned on existing MTMC Tracking 2025 scenes, your own synthetic dataset generated with Isaac Sim, or real-world imagery. For real-world data without 3D annotations, use the [2D-to-3D geometric distillation workflow](#2d-to-3d-geometric-distillation-for-real-world-datasets) to combine 2D supervision with 3D-labeled training data. Please refer to the [Simulation and Synthetic Data Generation](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/development-workflow/simulation-and-synthetic-data-generation) page for more details on synthetic data generation. Performing 3D Multi-Camera Detection & Tracking in large regions such as warehouses with high-density cameras becomes computationally expensive. Therefore, we divide a region into a collection of groups called Bird's Eye View (BEV) groups. Each Bird's Eye View (BEV) group can contain multiple cameras, and these BEV groups may overlap with one another. Sparse4D is trained on such BEV groups. The below image represents a top down view of a single scene with multiple BEV groups. Each unique color represents a BEV group highlighting its camera coverage in the entire scene. ![BEV Groups](/vss/_fern-img/79bbdccbf067b5d177039963f544cb6cfeb98fc4705d55a496c12e43100f4453.webp) **Requirements for the supervised multi-camera 3D workflow:** 1. **Number of Cameras in Scene** * Refer to the camera placement guide for details based on scene size 2. **Number of RGB Frames** * Minimum: 9,000 frames per camera (5 minutes @ 30 FPS) 3. **Number of Depth Map Frames** * Optional but recommended for better model convergence * Should match the number of RGB frames if included 4. **Number of BEV Groups (Bird's Eye View)** * We recommend a minimum of 3 BEV groups for a scene with 12 cameras in total. * The TAO Fine-tuning notebook will help generate randomized BEV groups based on the raw calibration data obtained from the SDG step. 5. **Number of Cameras per BEV Group** * Range: 4-12 cameras per group 6. **Annotations** * Ground Truth 3D & 2D bounding boxes in OV coordinates 7. **Calibration Data** * Single calibration file containing all camera parameters (intrinsic & extrinsic) Refer to [Dataset Post-Processing](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/development-workflow/simulation-and-synthetic-data-generation/dataset-post-processing) for the file format of the files listed above. An example dataset can be found in the [Warehouse_014](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces/tree/main/MTMC_Tracking_2025/train/Warehouse_014) scene from the [MTMC Tracking 2025](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces/tree/main/MTMC_Tracking_2025) dataset. ### 2D-to-3D Geometric Distillation for Real-World Datasets **2D-to-3D geometric distillation** lets Sparse4D learn from calibrated multi-camera real-world images with 2D annotations or detector pseudo-labels, reducing the need to collect full 3D ground truth. The term **2D-to-3D** describes the direction of supervision: 2D targets supervise the 3D model. During training, **3D-to-2D projection** maps Sparse4D's predicted 3D boxes into camera images, where they can be compared with the 2D targets. Sparse4D still predicts 3D boxes and tracking IDs. For the complete TAO configuration and artifact-preparation procedure, refer to the [TAO 2D-to-3D geometric distillation guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/sparse4d/sparse4d.html#sparse4d-geometric-distillation). A projected 3D cuboid encloses all eight corners and can be larger than the object's tight image-space box. The released geometry-only **loose-to-tight (LTT)** MLP corrects this difference toward tight, amodal 2D boxes. During Sparse4D fine-tuning, the MLP is frozen and its predicted correction factors are detached; gradients flow through the corrected box projection into Sparse4D. Per-camera matching then compares these boxes with real-world 2D detections, such as offline RT-DETR predictions, to supervise box geometry and classification. > **Note** > > Use a TAO PyTorch build that includes `model.head.loose_to_tight`. The data preparation commands below also require a TAO Data Services build with `annotations sparse4d_prepare` and `annotations convert` support for `aicity.load_annotations=false`; use the [shipped preparation spec](https://github.com/NVIDIA-TAO/tao-data-services/blob/main/nvidia_tao_ds/annotations/experiment_specs/sparse4d_prepare.yaml). Native artifact preparation commands are documented in the [TAO PyTorch tools guide](https://github.com/NVIDIA-TAO/tao-pytorch/blob/main/nvidia_tao_pytorch/cv/sparse4d/tools/README.md). Check the specs shipped with your build; the optional LTT training losses are disabled by default. #### Choose the supervision route | Training data | Required inputs | Supervision | | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | Synthetic or real scenes with 3D labels | Synchronized images, camera intrinsics/extrinsics, 3D boxes and IDs; visible 2D boxes for the optional LTT loss | Standard 3D losses, optionally supplemented by LTT 2D supervision | | Calibrated real-world scenes without 3D labels | Synchronized images, camera intrinsics/extrinsics, BEV groups, and per-camera 2D detections | Projected 2D box and classification pseudo-losses using LTT | Keep 3D-labeled data in the training mix to retain metric geometry supervision. #### Download the released LTT checkpoint For the released seven-class warehouse taxonomy listed below, reuse `_loose_to_tight_mlp.pth` from the [Sparse4D ResNet-50 `trainable_v3.0` bundle on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/sparse4d_rn50/files?version=trainable_v3.0). With the NGC CLI, download just the LTT file: ```bash mkdir -p /data ngc registry model download-version nvidia/tao/sparse4d_rn50:trainable_v3.0 \ --file _loose_to_tight_mlp.pth --dest /data ``` This writes `/data/sparse4d_rn50_vtrainable_v3.0/_loose_to_tight_mlp.pth`, including the leading underscore in the filename. Make that path visible inside the TAO training environment and use it for `model.head.loose_to_tight.mlp_ckpt`, as shown below. Keep the checkpoint's ordered class taxonomy unchanged and validate its corrections on the target data. Continue with teacher-cache preparation and mixed-training data preparation below. This workflow supports only the released checkpoint's seven classes in their original order. Custom taxonomies require a matching LTT checkpoint and are outside this guide; changing class names in the configuration does not adapt the released weights. Retain the external LTT file for fine-tuning and resume: the frozen MLP is excluded from Sparse4D model checkpoints. Standard evaluation, inference, and export do not require it. #### Prepare the training data 1. **Prepare 2D supervision.** Use `operation=ltt_2dgt` for visible-2D sidecars on 3D-labeled scenes. For calibrated real scenes, run the 2D teacher offline, then use `operation=rtdetr_2d` to convert its per-camera KITTI `labels.tar.gz` archives to `__rtdetr2d.npz`. This operation converts detections; it does not run RT-DETR. Before building the mixed split, create matching image/calibration info PKLs with the [calibrated real-scene conversion command](#prepare-calibrated-real-scene-inputs) below. 2. **Build the mixed training split.** List the converted 3D-labeled and calibrated real info PKLs once each. Mark genuinely unlabeled 3D samples with `gt_boxes=None`, rather than treating them as annotated empty scenes. Run `operation=lazy_index` after finalizing this split if using `dataset.lazy_load: true`. Use the same ordered taxonomy for data preparation `class_names`, training `dataset.classes`, the LTT checkpoint, and all caches. For the released seven-class warehouse models, the training names are `person`, `gr1_t2`, `agility_digit`, `nova_carter`, `transporter`, `forklift`, and `pallet_truck`. Map teacher labels explicitly to this taxonomy; a matching class count alone is insufficient. Camera names, source frame IDs, image dimensions, and coordinate conventions must agree between the info PKLs and their caches. #### Prepare calibrated real-scene inputs Use `annotations convert` in TAO Data Services with the [unlabeled-scene spec](https://github.com/NVIDIA-TAO/tao-data-services/blob/main/nvidia_tao_ds/annotations/experiment_specs/aicity2ovpkl_unlabeled.yaml). Copy the spec to `/specs/aicity2ovpkl_unlabeled.yaml` and set its ordered `aicity.class_config.CLASS_LIST` to match your model and teacher caches. This requires a build that exposes `aicity.load_annotations` and `aicity.fps`. Arrange each scene as `/data/real/train//calibration.json` plus `/rgb/000000000.jpg`, `000000001.jpg`, and so on. NVSchema calibration must provide camera intrinsics and world-to-camera poses with IDs matching the image directories. The `rgb_00000.jpg` image naming convention also works. Use synchronized, constant-rate frames, contiguous from zero, and provide the actual capture rate: ```bash annotations convert -e /specs/aicity2ovpkl_unlabeled.yaml \ aicity.root=/data/real aicity.split=train aicity.fps=30 \ results_dir=/data/real_infos ``` The spec sets `aicity.load_annotations=false`, keeps the world frame without recentering, and uses all calibrated cameras as one group per scene. It writes `/data/real_infos/train/_infos_train.pkl` with synchronized image paths, camera geometry, sequence/frame identifiers, timestamps in seconds, and `gt_boxes=None`. Ground truth and depth files are not required, and anchor initialization is skipped. Reuse the pretrained model's anchors and keep image paths visible at the same location inside the training environment. Each PKL stores frame records in `infos`. Each record retains `token`, `scene_name`, `timestamp`, `cams`, and `gt_boxes=None` for unlabeled frames. The `cams` mapping is keyed by camera ID; each camera entry retains its `data_path`. Preserve these field names and conventions: | Location | Field | Meaning | | ---------------------- | ------------------------ | -------------------------------------------------------------------------------------------- | | Frame record | `frame_idx` | Zero-based source frame ID, matching the 2D teacher cache; starts at zero for each sequence. | | Camera entry in `cams` | `cam_intrinsic` | 3 × 3 camera intrinsic matrix for the source image dimensions. | | Camera entry in `cams` | `sensor2world_transform` | 4 × 4 **world-to-camera** extrinsic matrix for the default unrecentered spec. | `sensor2world_transform` is a legacy field name. The loader uses this matrix directly with the camera intrinsics to project 3D points; it does **not** invert it. If BEV-group recentering is enabled, the matrix must instead map the corresponding group-local 3D coordinates into the camera frame. For HDF5 image arrays, set `aicity.rgb_format=h5` and provide `.h5` files with HWC arrays at `rgb/rgb_00000.jpg`, `rgb/rgb_00001.jpg`, and so on. Set `dataset.use_h5_file_for_rgb: true` for these inputs during training. The converter rejects empty, gapped, or unequal camera sequences. Synchronize irregular or dropped-frame inputs before conversion and run the 2D teacher on those same frames. Inspect projections and cache joins before training; keep calibration consistent with image dimensions and the metric world frame. The existing camera-grouping options can generate multiple BEV groups. Use one info PKL per group, with camera names and source frame IDs matching the teacher cache. For example, a PKL scene named `RealWarehouse+bev0` uses `/data/rtdetr/RealWarehouse__rtdetr2d.npz`: the loader strips the BEV-group suffix when locating the cache. Include each PKL once in `/data/mixed_train_split.txt` alongside the 3D-labeled PKLs, then run `operation=lazy_index` using the final split. #### Configure mixed 3D and real-world training Merge the following settings into a complete Sparse4D experiment spec for your checkpoint. Replace the paths and `RealWarehouse` with your real scene-name substring. Retain the matching backbone, anchors, class order, and model input size from the base spec. ```yaml model: cotrain_param_touch: false head: loose_to_tight: enable: true mlp_ckpt: /data/sparse4d_rn50_vtrainable_v3.0/_loose_to_tight_mlp.pth num_classes: 0 # Derive from dataset.classes. loss_weight: 0.1 pseudo_enable: true pseudo_box_weight: 0.1 pseudo_cls_weight: 1.0 min_cams: 1 dataset: classes: [person, gr1_t2, agility_digit, nova_carter, transporter, forklift, pallet_truck] train_dataset: ann_file: /data/mixed_train_split.txt lazy_load: true ltt_2dgt_sidecar_dir: /data/ltt_2dgt rtdetr_2d_cache_dir: /data/rtdetr rtdetr_2d_mark_real: true sync_route: true real_scene_keywords: [RealWarehouse] real_block_prob: 0.5 scene_switch_iters: 100 ``` `real_block_prob: 0.5` is an example starting point for the proportion of 2D-route blocks, not an accuracy recommendation. Distributed co-training requires `sync_route: true` and a positive `scene_switch_iters`; each batch must have homogeneous 2D/3D supervision. `rtdetr_2d_mark_real: true` makes any scene with a teacher cache use the 2D-only route, so keep those caches separate from scenes intended for full 3D supervision. Run the regular `sparse4d train -e /specs/experiment_spec.yaml` command inside the TAO PyTorch environment. This workflow uses the training task; there is no separate Sparse4D `distill` task. Refer to the [TAO Sparse4D documentation](https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/sparse4d/sparse4d.html) for the standard training, evaluation, and export workflow. #### Validate and deploy Inspect projected boxes on real images before a long run. Verify that expected labeled frames join their caches and that `loss_box_2d_pseudo_*` and `loss_cls_pseudo_*` contribute on calibrated real batches. Missing cache joins skip pseudo supervision and must not be interpreted as successfully processed empty detections. Compare the original checkpoint and the adapted checkpoint on held-out, 3D-labeled real scenes, and retain a synthetic validation set to check for regressions. Report detection and tracking metrics separately; a lower 2D loss alone does not establish better metric 3D accuracy or tracking. RT-DETR pseudo-label generation and the LTT MLP are training components. Standard Sparse4D evaluation, inference, and export do not require the external LTT checkpoint. Export the adapted Sparse4D model and follow the [model compatibility requirements](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/microservice-configuration/perception/3d-multi-camera-detection-and-tracking-sparse4d#model-compatibility-requirements) when deploying it. The deployed multi-camera 3D pipeline still needs real camera calibration. ### Camera Grouping for Model Fine-tuning For Sparse4D fine-tuning, cameras need to be organized into BEV groups with overlapping fields of view. Unlike camera clustering (used for deployment, where each camera belongs to exactly one cluster), camera grouping for training allows cameras to appear in multiple groups to maximize training data coverage. Use `create_camera_groups.py` to create camera groups with duplication support based on overlapping FOV: We use the `spatialai_data_utils` library to create camera groups: ```bash pip install spatialai-data-utils==2.0.1 --extra-index-url=https://edge.urm.nvidia.com/artifactory/api/pypi/sw-metropolis-pypi/simple ``` The camera grouping tools are shipped with the full [spatialai-data-utils](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.3.0/libs/analytics/spatialai-data-utils) package in the VSS repository. **Navigate to the spatialai-data-utils camera grouping tools** **`Navigate to the spatialai-data-utils camera grouping tools`** ```bash title="Navigate to the spatialai-data-utils camera grouping tools" export VSS_REPO_DIR=/path/to/video-search-and-summarization cd $VSS_REPO_DIR/libs/analytics/spatialai-data-utils/tools/camera_grouping/ ``` ```bash # Auto mode (recommended for training): one group each of sizes # 1, 2, ..., min(n_sensors, max_sensors_per_group). Overrides --cameras_per_group. python create_camera_groups.py data/scene --auto # Auto mode with a custom cap on the largest group size (default 18) python create_camera_groups.py data/scene --auto --max_sensors_per_group 12 # Manual mode: 5 groups with 8 cameras each python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 # Multiple size types: 2 groups × 3 sizes = 6 total groups # (2 groups with 5 cameras, 2 groups with 8 cameras, 2 groups with 6 cameras) python create_camera_groups.py data/scene --n_groups 2 --cameras_per_group 5 8 6 # With visualization (generates separate images per group by default) python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 --visualize # Custom overlap threshold for stricter FOV requirements python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 \ --min_overlap_threshold 0.3 --visualize # Use existing FOV polygons from calibration instead of frustum calculation python create_camera_groups.py data/scene --n_groups 5 --cameras_per_group 8 \ --prefer_existing_fov ``` Use `--help` to list all available arguments. Key options: * `--auto`: Auto mode — set `--cameras_per_group` to \[1, 2, ..., min(n_sensors, `--max_sensors_per_group`)], creating groups for every size (recommended for training). Overrides `--cameras_per_group`. * `--n_groups`: Number of groups to create per size (default: 1). * `--cameras_per_group`: Number of cameras per group (required unless `--auto` is used). Single value: all groups have that size. Multiple values: creates n_groups for EACH size (total = n_groups × count). * `--max_sensors_per_group`: Maximum sensors per group (default: 18). In auto mode, caps the largest generated group size. * `--output`: Output path for the grouped calibration file. * `--output_suffix`: Suffix for output files (default: "grouped"). * `--map_file`: Path to map image for visualization (uses black background if not provided). * `--start_camera_index`: Starting camera index for seeding the first group (default: 0). * `--min_overlap_threshold`: Minimum FOV overlap (0-1) for group membership (default: 0.2). * `--max_distance_threshold`: Maximum centroid distance in meters for membership (default: inf). * `--prefer_existing_fov`: Use existing FOV polygons instead of calculating from frustum. * `--max_camera_distance`: Maximum distance in meters for frustum calculation (default: 30.0). * `--height_range`: Height range (min, max) in meters for ground plane intersection (default: 1.0 3.0). * `--image_size`: Image dimensions (width, height) in pixels for frustum calculation (default: 1920 1080). * `--dilation`: Buffer distance in meters for group bounding boxes (default: 8.0). * `--visualize`: Generate visualization of camera groups. * `--vis_no_camera_id_labels`: Disable drawing camera IDs on the visualization. * `--vis_combined`: Generate a single combined image instead of separate images per group. > **Note** > > For deployment and inference use cases where cameras should be partitioned into non-overlapping clusters, refer to the [Camera Clustering](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/development-workflow/deployment-customization/3d-vision-ai-profile#camera-clustering) section in the 3D Vision AI Profile documentation. ### Expected time for finetuning The expected data requirements and time to fine-tune the Sparse4D model on a single scene of the MTMC Tracking 2025 dataset are as follows: | Backbone Type | GPU Type | Image Size | No. of BEV groups | No. of cameras in each BEV group | No. of frames in each camera | Total No. of Epochs | Total Training Time | | ------------- | -------------------------- | ---------- | ---------------------- | -------------------------------- | ---------------------------- | ------------------- | ------------------- | | Resnet101 | 8 x Nvidia H100 - 80GB SXM | 3x540x960 | 3 (Minimum BEV groups) | 4-12 | 9000 (5 mins @ 30 FPS) | 5 | 10 hours | We recommend a multi-gpu training setup for faster training. ### TAO Fine-tuning via Notebook Walkthrough & Configuration Changes Sparse4D can be fine-tuned via the TAO containers and the TAO CLI Notebook. The documentation provided below accompanies the cells in the TAO fine-tuning notebook and offers guidance on how to execute them. TAO Sparse4D supports the following tasks via the Jupyter notebook: * dataset `convert` * model `train` * model `evaluate` * model `inference` * model `export` An experiment specification file (also known as a configuration file) is used for fine-tuning the model. It consists of several main components: * `dataset` * `model` * `train` * `evaluate` * `inference` * `export` * `visualize` The notebook utilizes the [Warehouse_014](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces/tree/main/MTMC_Tracking_2025/train/Warehouse_014) scene from the [MTMC Tracking 2025](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces/tree/main/MTMC_Tracking_2025) dataset for training. The following is an example spec file for training a Sparse4D model on one scene of the [MTMC Tracking 2025](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces/tree/main/MTMC_Tracking_2025) dataset. For field-level detail, use the TAO Sparse4D configuration documentation that matches your TAO Toolkit release. ```yaml results_dir: /results train: num_gpus: 1 gpu_ids: - 0 num_nodes: 1 seed: 1234 cudnn: benchmark: false deterministic: true num_epochs: 10 checkpoint_interval: 0.5 validation_interval: 0.5 resume_training_checkpoint_path: null results_dir: null pretrained_model_path: null optim: type: adamw lr: 5.0e-05 weight_decay: 0.001 momentum: 0.9 paramwise_cfg: custom_keys: img_backbone: lr_mult: 0.2 grad_clip: max_norm: 25 norm_type: 2 lr_scheduler: policy: cosine warmup: linear warmup_iters: 500 warmup_ratio: 0.333333 min_lr_ratio: 0.001 model: type: sparse4d embed_dims: 256 use_grid_mask: true use_deformable_func: true input_shape: - 960 - 540 backbone: type: resnet_101 neck: type: FPN num_outs: 4 start_level: 0 out_channels: 256 in_channels: - 256 - 512 - 1024 - 2048 add_extra_convs: on_output relu_before_extra_convs: true depth_branch: type: dense_depth embed_dims: 256 num_depth_layers: 3 loss_weight: 0.2 head: type: sparse4d num_output: 300 cls_threshold_to_reg: 0.05 decouple_attn: true return_feature: true use_reid_sampling: false embed_dims: 256 reid_dims: 0 num_groups: 8 num_decoder: 6 num_single_frame_decoder: 1 drop_out: 0.1 temporal: true with_quality_estimation: true operation_order: - deformable - ffn - norm - refine - temp_gnn - gnn - norm - deformable - ffn - norm - refine - temp_gnn - gnn - norm - deformable - ffn - norm - refine - temp_gnn - gnn - norm - deformable - ffn - norm - refine - temp_gnn - gnn - norm - deformable - ffn - norm - refine - temp_gnn - gnn - norm - deformable - ffn - norm - refine visibility_net: type: visibility_net embedding_dim: 256 hidden_channels: 32 instance_bank: num_anchor: 900 anchor: '' num_temp_instances: 600 confidence_decay: 0.8 feat_grad: false default_time_interval: 0.033333 embed_dims: 256 use_temporal_align: false grid_size: null anchor_encoder: type: SparseBox3DEncoder vel_dims: 3 embed_dims: - 128 - 32 - 32 - 64 mode: cat output_fc: false in_loops: 1 out_loops: 4 pos_embed_only: false sampler: num_dn_groups: 5 num_temp_dn_groups: 3 dn_noise_scale: - 2.0 - 2.0 - 2.0 - 0.5 - 0.5 - 0.5 - 0.5 - 0.5 - 0.5 - 0.5 - 0.5 max_dn_gt: 128 add_neg_dn: true cls_weight: 2.0 box_weight: 0.25 reg_weights: - 2.0 - 2.0 - 2.0 - 0.5 - 0.5 - 0.5 - 0.0 - 0.0 - 0.0 - 0.0 - 0.0 use_temporal_align: false gt_assign_threshold: 0.5 reg_weights: - 2.0 - 2.0 - 2.0 - 1.0 - 1.0 - 1.0 - 1.0 - 1.0 - 1.0 - 1.0 - 1.0 loss: cls: type: focal use_sigmoid: true gamma: 2.0 alpha: 0.25 loss_weight: 2.0 reg: type: sparse_box_3d box_weight: 0.25 cls_allow_reverse: [] id: type: cross_entropy_label_smooth num_ids: 70 bnneck: type: bnneck feat_dim: 256 num_ids: 70 deformable_model: embed_dims: 256 num_groups: 8 num_levels: 4 attn_drop: 0.15 use_deformable_func: true use_camera_embed: true residual_mode: cat num_cams: 6 max_num_cams: 20 proj_drop: 0.0 kps_generator: embed_dims: 256 num_learnable_pts: 6 fix_scale: - - 0 - 0 - 0 - - 0.45 - 0 - 0 - - -0.45 - 0 - 0 - - 0 - 0.45 - 0 - - 0 - -0.45 - 0 - - 0 - 0 - 0.45 - - 0 - 0 - -0.45 refine_layer: type: sparse_box_3d_refinement_module embed_dims: 256 refine_yaw: true with_quality_estimation: true valid_vel_weight: -1.0 graph_model: type: MultiheadAttention embed_dims: 512 num_heads: 8 batch_first: true dropout: 0.1 temp_graph_model: type: MultiheadAttention embed_dims: 512 num_heads: 8 batch_first: true dropout: 0.1 decoder: type: SparseBox3DDecoder score_threshold: 0.05 norm_layer: type: LN normalized_shape: 256 ffn: type: AsymmetricFFN in_channels: 512 pre_norm: type: LN normalized_shape: 256 embed_dims: 256 feedforward_channels: 1024 num_fcs: 2 ffn_drop: 0.1 act_cfg: type: ReLU inplace: true use_temporal_align: false inference: num_gpus: 1 gpu_ids: - 0 num_nodes: 1 checkpoint: ??? trt_engine: null results_dir: null jsonfile_prefix: sparse4d_pred output_nvschema: true tracking: enabled: true threshold: 0.2 evaluate: num_gpus: 1 gpu_ids: - 0 num_nodes: 1 checkpoint: ${results_dir}/train/sparse4d_model_latest.pth trt_engine: null results_dir: null metrics: - detection tracking: enabled: true threshold: 0.2 export: results_dir: null gpu_id: 0 checkpoint: ??? onnx_file: ??? on_cpu: false input_channel: 3 input_width: 960 input_height: 544 opset_version: 17 batch_size: -1 verbose: false format: onnx visualize: show: true vis_dir: ./vis vis_score_threshold: 0.25 n_images_col: 6 # Update this based on the number of cameras in the scene viz_down_sample: 3 ``` ### Dataset Preparation The data obtained from HuggingFace is stored in AICity format. We need to convert the AICity format to the OVPKL format required for model training. TAO Data Service can be used to convert the AICity format to the OVPKL format required for model training. The following configuration file can be used to convert the AICity format to the OVPKL format. ```yaml data: input_format: "AICity" output_format: "OVPKL" aicity: root: ??? # path to your dataset root directory version: "2025" split: "train" class_config: CLASS_LIST: - "Person" - "FourierGR1T2" - "AgilityDigit" - "Transporter" SUB_CLASS_DICT: {} MAP_CLASS_NAMES: "Person": "Person" "FourierGR1T2": "FourierGR1T2" "AgilityDigit": "AgilityDigit" "Transporter": "Transporter" ATTRIBUTE_DICT: "Person": "person.moving" "FourierGR1T2": "fourier_gr1_t2.moving" "AgilityDigit": "agility_digit.moving" "Transporter": "transporter.moving" CLASS_RANGE_DICT: "Person": 40 "FourierGR1T2": 40 "AgilityDigit": 40 "Transporter": 40 recentering: true rgb_format: "mp4" # For datasets obtained from HuggingFace, RGB images are stored in "mp4" format. For SDG generated dataset, RGB images are stored in "h5" format. depth_format: "h5" # Depth maps are stored in "h5" both for HuggingFace & SDG generated datasets. camera_grouping_mode: "train" num_frames: -1 anchor_init_config: output_file_name: "anchor_init_kmeans900.npy" results_dir: ??? # Path to where you want to store the data ``` Once the configurations are modified, launch dataset conversion by running the following command: ![TAO Sparse4D Dataset Conversion](/vss/_fern-img/011d3e530b929e4058fe0144a2b4cea957dddc3574cbb0db7ffa261fed4315f8.webp) ### Launch Model Fine-tuning Below are the important configurations that need to be updated to launch training: ```yaml dataset: use_h5_file: true num_frames: 9000 batch_size: 4 # Depending on your GPU memory num_bev_groups: 3 # Depending on your scene size num_workers: 4 # Depending on your CPU memory num_ids: 70 # Depending on your dataset classes: [ # Update this based on your dataset "person", "gr1_t2", "agility_digit", "nova_carter", "transporter", "forklift", ] data_root: ??? # Update this based on your dataset root directory train_dataset: ann_file: ??? # Update this based on your training dataset annotations file/folder. val_dataset: ann_file: ??? # Update this based on your validation dataset annotations file/folder. test_dataset: ann_file: ??? # Update this based on your test dataset annotations file/folder. ``` ```yaml model: head: instance_bank: anchor: ??? # Update this to the anchor file generated by the TAO Fine-tuning notebook. ``` ```yaml train: num_epochs: 5 # Update this based on your dataset size. 5 epochs is recommended for fine-tuning on a single scene. num_nodes: 1 # Update this based on your GPU count num_gpus: 1 # Update this based on your GPU count validation_interval: 1 # Update this based on your dataset size checkpoint_interval: 1 # Update this based on your dataset size pretrained_model_path: ??? # Update this to the pretrained model path if you are fine-tuning on a pretrained model. optim: lr: 0.0001 # Update this based on your dataset size paramwise_cfg: custom_keys: img_backbone: lr_mult: 0.25 ``` Once the configurations are modified, launch training by running the following command: ![TAO Sparse4D Finetuning](/vss/_fern-img/58a81fcc87a68ef02ee1a553da8d731f776d27c10de74e1c0ea88cfef542bc40.webp) ### Evaluate the fine-tuned model The fine-tuned model's tracking accuracy can be evaluated using the Higher Order Tracking Accuracy (HOTA) metric, decomposed as Detection Accuracy (DetA), Association Accuracy (AssA), and Localization Accuracy (LocA). Please refer to the [KPI](/system-components/models/sparse4d-3d-multi-camera#kpi) section above for more details on the metric definition and how to interpret per-class scores. Below are the important configurations that need to be updated to launch evaluation: ```yaml evaluate: checkpoint: ${results_dir}/train/sparse4d_model_latest.pth # Update this based on your fine-tuned model path ``` ```bash dataset: test_dataset: ann_file: ??? # Update this based on your test dataset annotations file/folder. ``` ![TAO Sparse4D Evaluation](/vss/_fern-img/60d1ce550d535069fd3907789f41cb4cce302c0531112d6e7669a1d37301bda5.webp) ### Run Inference on the fine-tuned model using TAO The fine-tuned model's output can be inspected by running inference and visualizing the 3D bounding boxes and their tracking IDs. Below are the important configurations that need to be updated to launch inference: ```yaml inference: checkpoint: ??? # Update this based on your fine-tuned model path ``` ```yaml dataset: test_dataset: ann_file: ??? # Update this based on your test dataset annotations file/folder. ``` ```yaml visualize: show: true vis_dir: ./vis vis_score_threshold: 0.25 n_images_col: 6 # Update this based on the number of cameras in the scene viz_down_sample: 3 ``` Once the configurations are modified, launch inference by running the following command: ![TAO Sparse4D Inference](/vss/_fern-img/554b36f9665ffb1217d6b5675f1d6e0df600234517ef7ad89db163ea230d16ef.webp) ### Export the fine-tuned model Below are the important configurations that need to be updated to launch export: ```yaml export: results_dir: ??? # Update this based on your results directory checkpoint: ??? # Update this based on your fine-tuned model path onnx_file: ??? # Update this based on your desired model name ``` Once the configurations are modified, launch export by running the following command: ![TAO Sparse4D Export](/vss/_fern-img/2a9e2fb7100f3684f326f1c29c7d8e5cd186b46905733030744a3bf045696ea5.webp) ### Using the fine-tuned model in the Perception Microservice Once the model is fine-tuned and exported to ONNX format, it can be used in the Perception Microservice. Refer to the [Integrating a Sparse4D Model Checkpoint](/vision-agent/agent-workflows/industry-specific-examples/warehouse-operations-blueprint/microservice-configuration/perception/3d-multi-camera-detection-and-tracking-sparse4d#integrating-a-sparse4d-model-checkpoint) section for more details.