Sparse4D#

Sparse4D is a Multi-Camera 3D Detection and Tracking model with 4D (spatial-temporal) capabilities. It takes synchronized input images from multiple cameras and calibration matrices and outputs the 3D bounding boxes and temporally consistent tracking IDs. Sparse4D uses a ResNet backbone. Select the backbone that matches your pretrained checkpoint.

Each batch in Sparse4D is trained on a group of cameras. Each group is called as a bird’s-eye view (BEV) group. A BEV group is a collection of multiple overlapping cameras.

The expected data requirements and time to fine-tune the Sparse4D model on a single scene of the MTMC Tracking 2025 dataset are as follows:

Estimated time for fine-tuning Sparse4D on a single scene of the MTMC Tracking 2025 dataset#
Backbone type	GPU type	Image size	No. of BEV groups	No. of cameras in each BEV group	No. of frames in each camera	Total no. of epochs	Total training time
Resnet101	8 x Nvidia H100 - 80GB SXM	3x512x1408	3 (Minimum BEV groups)	4-12	9000 (5 mins @ 30 frames per second, FPS)	5	10 hours

Sparse4D supports the following tasks:

train
evaluate
inference
export
quantize

Data Input for Sparse4D#

Sparse4D uses synchronized multi-camera images and calibration for training, validation, and testing.

Refer to Sparse4D Detection and Tracking OVPKL Format for the converted dataset format and the source MTMC Tracking 2025 dataset. The dataset is converted into pickle format and stored in the data/sparse4d/ directory. For calibrated real scenes without 3D annotations, refer to Convert Calibrated Real Scenes Without 3D Annotations.

Creating an Experiment Specification File#

The specification file for Sparse4D includes model, dataset, train parameters, visualize parameters, evaluate parameters and inference parameters. The following is an example specification file for training a Sparse4D model on one scene of the MTMC Tracking 2025 dataset. We will utilize the Warehouse_014 scene from the MTMC Tracking 2025 dataset for training.

The experiment specification consists of several main components:

dataset
model
train
evaluate
inference
export
visualize

dataset#

The dataset parameter defines the dataset source, training batch size, and augmentation. An example dataset is provided below. This section describes the main parameters of the Omniverse3DDetTrackDatasetConfig.

dataset:
  use_h5_file_for_rgb: false
  use_h5_file_for_depth: true
  num_frames: 9000
  batch_size: 2
  num_bev_groups: 1
  num_workers: 2
  num_ids: 70
  eval_dist_fcn: "iou_3d"
  eval_hota: false
  classes: [
    "person",
    "gr1_t2",
    "agility_digit",
    "nova_carter",
  ]
  type: "omniverse_3d_det_track"
  data_root: ???
  train_dataset:
    ann_file: ???
    test_mode: false
    use_valid_flag: true
    with_seq_flag: true
    sequences_split_num: 100
    keep_consistent_seq_aug: true
    same_scene_in_batch: true
  val_dataset:
    ann_file: ???
    test_mode: true
    use_valid_flag: true
    tracking: true
    tracking_threshold: 0.2
  test_dataset:
    ann_file: ???
    test_mode: true
    use_valid_flag: true
    tracking: true
    tracking_threshold: 0.2
  augmentation:
    resize_lim: [0.7, 0.77]
    final_dim: [512, 1408]
    bot_pct_lim: [0.0, 0.0]
    rot_lim: [-5.4, 5.4]
    image_size: [1080, 1920]
    rand_flip: true
    rot3d_range: [-0.3925, 0.3925]
  normalize:
    mean: [123.675, 116.28, 103.53]
    std: [58.395, 57.12, 57.375]
    to_rgb: true
  sequences:
    split_num: 100
    keep_consistent_aug: true
    same_scene_in_batch: true

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	Dataset type	omniverse_3d_det_track
`batch_size`	int	Batch size	2	1	inf
`use_h5_file_for_rgb`	bool	Read RGB camera images from Hierarchical Data Format 5 (HDF5) files instead of image directories	False
`use_h5_file_for_depth`	bool	Use HDF5 files for depth maps	True
`lazy_load`	bool	Defer `.pkl` file loading to `__getitem__` using a pre-built index	False
`lazy_load_cache_size`	int	Maximum `.pkl` files held in the lazy-loading LRU cache	50	1	inf
`pkl_sample_size`	int	Number of `.pkl` files sampled per epoch; zero disables sampling. Requires `lazy_load=True`	0	0	inf
`pkl_cam_counts_path`	string	Path to the mapping from `.pkl` files to camera counts used for balanced sampling
`fps_drop_prob`	float	Probability of downsampling a training scene to a lower FPS	0.0	0.0	1.0
`target_fps_choices`	list	Candidate target FPS values for FPS-drop augmentation	[30, 20, 15, 10, 6, 5, 3, 2, 1]				FALSE
`max_cameras`	int	Limit training cameras per frame to reduce GPU memory; a non-positive value disables sampling	-1	-1	inf
`eval_dist_fcn`	categorical	Evaluation distance function	iou_3d			center_distance,iou_3d,both
`eval_hota`	bool	Run HOTA tracking evaluation in addition to AMOTA	False
`ltt_2dgt_sidecar_dir`	string	Directory of per-scene LooseToTight 2D ground-truth NPZ sidecars
`ltt_2dgt_frame_regex`	string	Optional regular expression used to extract frame IDs for loose-to-tight (LTT) sidecar joins
`ltt_2dgt_dedup_regex`	string	Regular expression removed from scene names before resolving LTT sidecars	^CT[w.]+?__
`ltt_2dgt_cache_size`	int	Number of per-scene LTT sidecar indices cached per worker	8	1	inf
`ltt_2dgt_warn_on_miss`	bool	Warn once per scene when an LTT sidecar join has no matches	True
`rtdetr_2d_cache_dir`	string	Directory of per-scene RT-DETR 2D pseudo-label NPZ caches
`rtdetr_2d_cache_path`	string	Optional single RT-DETR 2D pseudo-label NPZ cache
`rtdetr_2d_dedup_regex`	string	Regular expression removed from scene names before resolving RT-DETR caches	^CT[w.]+?__
`rtdetr_2d_score_thr`	float	Global score threshold for RT-DETR pseudo-labels	0.0	0.0	1.0
`rtdetr_2d_per_class_score_thr`	collection	Optional class-name to score-threshold overrides	{}				FALSE
`rtdetr_2d_cache_size`	int	Number of RT-DETR scene indices cached per worker	4	1	inf
`rtdetr_2d_mark_real`	bool	Mark samples with an RT-DETR scene cache as lacking 3D ground truth	True
`sync_route`	bool	Keep distributed ranks on the same 3D or 2D supervision route	False
`real_scene_keywords`	list	Scene-name substrings identifying real 2D-supervised data	[]				FALSE
`real_block_prob`	float	Probability in [0, 1], or exactly -1 for automatic scene-count weighting; (-1, 0) is invalid	-1.0	-1.0	1.0
`scene_switch_iters`	int	Iterations per scene before a synchronized route switch; zero uses legacy cadence	0	0	inf
`num_frames`	int	Number of frames	200	1	inf
`num_bev_groups`	int	Number of BEV groups	1	1	inf
`data_root`	string	Path to data root	???
`classes`	list	Classes to detect	[‘person’, ‘gr1_t2’, ‘agility_digit’, ‘nova_carter’]				FALSE
`num_workers`	int	Number of workers	4	0	inf
`num_ids`	int	Number of IDs	70	1	inf
`augmentation`	collection	Augmentation configuration					FALSE
`normalize`	collection	Normalize configuration					FALSE
`sequences`	collection	Sequences configuration					FALSE
`train_dataset`	collection	Train dataset configuration					FALSE
`val_dataset`	collection	Val dataset configuration					FALSE
`test_dataset`	collection	Test dataset configuration					FALSE
`quant_calibration_dataset`	collection	Configurable parameters for quantization calibration dataset.					FALSE

The eval_dist_fcn and eval_hota fields select the evaluation protocol. Their defaults are iou_3d and false. Refer to Evaluation Protocol and Metrics for the metrics that each setting reports.

Dynamic Resampling and Dataset-Loading Options#

The Sparse4D dataloader supports lazy pickle loading, balanced per-epoch dynamic resampling, and frame-rate-drop (FPS-drop) augmentation. Use lazy loading and dynamic resampling when training on large multi-camera annotation sets split across multiple .pkl files.

Set these options directly under dataset, alongside batch_size and the other dataset fields. Current Sparse4D builds declare these fields in the structured dataset configuration. The dataset parameters list their defaults.

Lazy loading. When lazy_load is set to True, the dataloader does not load every annotation .pkl file into memory up front. Instead it loads a pre-built frame index and reads the underlying .pkl files on demand through a least-recently-used (LRU) cache. The lazy_load_cache_size field sets the maximum size of this cache.

Lazy loading requires that the ann_file of the training dataset is either a directory of .pkl files or a .txt file listing one .pkl path per line, and that a lazy-index cache file is present alongside the annotation file before training starts:

If ann_file is a directory, the cache file is expected at <ann_file>/_lazy_index.pkl.
If ann_file is a .txt list, the cache file is expected at the same path with the .txt extension replaced by _lazy_index.pkl (for example, train.txt → train_lazy_index.pkl).

The dataloader does not build this index automatically. If lazy loading is enabled and the cache file is missing, the dataloader raises a FileNotFoundError and training stops, so you must generate the _lazy_index.pkl file yourself beforehand.

Use the maintained TAO Data Services preparation command to build the index:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml \
  operation=lazy_index lazy_index.annotation_source=/data/mixed_train_split.txt

Refer to Prepare Artifacts with TAO Data Services for the preparation specification. The producer records each frame’s source pickle path, local index, scene name, timestamp, and frame ID, and embeds camera counts in pkl_cam_counts. It also writes the sibling _pkl_cam_counts.pkl file. Rebuild the index after changing the split or moving the source files. Keep the recorded paths visible inside the training container.

If ann_file points to a single .pkl file, the dataloader ignores lazy loading and automatically falls back to normal (non-lazy) loading, so no index file is needed in that case.

Dynamic resampling. At each epoch, the dataloader draws a fresh subset of pkl_sample_size .pkl files from the full index instead of training on all files. The subset balances the number of cameras per .pkl file, so that it represents scenes with different camera counts evenly. When pkl_sample_size is 0 (the default), the dataloader disables resampling and lazy loading uses the full index.

New lazy indexes embed the camera counts needed for balanced sampling, so you can leave pkl_cam_counts_path empty. Set that option only to supply an explicit camera-count override or a mapping for a legacy index. The maintained preparation command generates this mapping; you do not need a custom script. The mapping keys must exactly match the pkl_path values in the lazy index.

If pkl_sample_size is greater than 0, training stops at startup with an error in the following cases:

lazy_load is false. The dataloader raises a ValueError.
ann_file points to a single .pkl file. The dataloader raises a ValueError.
pkl_cam_counts_path is empty and the lazy index has no embedded camera counts. The dataloader raises a ValueError.
pkl_cam_counts_path names a file that does not exist. The dataloader raises a FileNotFoundError. Remove the path to use the counts embedded in the index.
The camera-count mapping has no entry for a pkl_path in the lazy index. The dataloader raises a KeyError.

Resampling is driven automatically during training: a PklResampleCallback re-runs the balanced sampling at every epoch boundary, then refreshes the in-batch sequence sampler so it picks up the new sequence/scene grouping. The sampling is seeded reproducibly per epoch, so each epoch sees a different but deterministic subset.

FPS-drop augmentation. fps_drop_prob sets the probability of temporally downsampling a scene during loading to one of the frame rates listed in target_fps_choices, which exposes the model to a range of effective frame rates. Setting fps_drop_prob to 0 (the default) disables this augmentation.

FPS-drop augmentation applies only when the dataloader loads a directory or a .txt split without lazy loading.

Camera subsampling. When max_cameras is greater than 0, each training frame is randomly subsampled to at most that many cameras. The default of -1 keeps all cameras.

The following example enables lazy loading with balanced dynamic resampling of 500 .pkl files per epoch:

dataset:
  type: "omniverse_3d_det_track"
  data_root: /data/sparse4d
  batch_size: 4
  num_workers: 4
  lazy_load: true
  lazy_load_cache_size: 50
  pkl_sample_size: 500
  max_cameras: -1
  train_dataset:
    ann_file: /data/sparse4d/train     # directory of .pkl files (or a .txt list)
    test_mode: false
    use_valid_flag: true
    with_seq_flag: true
    sequences_split_num: 100
    keep_consistent_seq_aug: true
    same_scene_in_batch: true

Train Dataset Configuration (dataset.train_dataset)#

Configuration for the training dataset.

Field	value_type	description	default_value	valid_min	valid_max
`ann_file`	string	Path to annotation file	???
`test_mode`	bool	Test mode	False
`use_valid_flag`	bool	Use valid flag	True
`with_seq_flag`	bool	With sequence flag	True
`sequences_split_num`	int	Number of sequences	100	1	infinity
`keep_consistent_seq_aug`	bool	Keep consistent sequence augmentation	True
`same_scene_in_batch`	bool	Same scene in batch	True

Validation Dataset Configuration (dataset.val_dataset)#

Configuration for the validation dataset.

Field	value_type	description	default_value	valid_min	valid_max
`ann_file`	string	Path to annotation pickle files/folders	???
`test_mode`	bool	Test mode	False
`use_valid_flag`	bool	Use valid flag	True
`tracking`	bool	Tracking	True
`tracking_threshold`	float	Tracking threshold	0.2	0	1
`same_scene_in_batch`	bool	Same scene in batch	True

Test Dataset Configuration (dataset.test_dataset)#

Configuration for the test dataset.

Field	value_type	description	default_value	valid_min	valid_max
`ann_file`	string	Path to annotation pickle files/folders	???
`test_mode`	bool	Test mode	True
`use_valid_flag`	bool	Use valid flag	True
`tracking`	bool	Tracking	True
`tracking_threshold`	float	Tracking threshold	0.2	0	1
`same_scene_in_batch`	bool	Same scene in batch	True

Augmentation Configuration (dataset.augmentation)#

Configuration for data augmentation.

Field	value_type	description	default_value	automl_enabled
`resize_lim`	list	Resize limits	[0.7, 0.77]	false
`final_dim`	list	Final dimensions	[512, 1408]	false
`bot_pct_lim`	list	Bottom percentage limits	[0.0, 0.0]	false
`rot_lim`	list	Rotation limits in degrees	[-5.4, 5.4]	false
`image_size`	list	Original image size	[1080, 1920]	false
`rand_flip`	bool	Random flip	True
`rot3d_range`	list	3D rotation range in radians	[-0.3925, 0.3925]	false

Normalize Configuration (dataset.normalize)#

Configuration for image normalization.

Field	value_type	description	default_value	automl_enabled
`mean`	list	Mean values for normalization	[123.675, 116.28, 103.53]	false
`std`	list	Standard deviation values for normalization	[58.395, 57.12, 57.375]	false
`to_rgb`	bool	Convert to RGB	True

Sequences Configuration (dataset.sequences)#

Configuration for handling image sequences.

Field	value_type	description	default_value	valid_min	valid_max
`split_num`	int	Number of sequence splits	100	1	infinity
`keep_consistent_aug`	bool	Keep consistent augmentation	True
`same_scene_in_batch`	bool	Keep same scene in batch	True

model#

The model parameter provides options to change the Sparse4D architecture.

model:
  type: "sparse4d"
  use_grid_mask: true
  use_deformable_func: true
  use_temporal_align: true
  cotrain_param_touch: false
  input_shape: [1408, 512]
  embed_dims: 256
  neck:
    type: "FPN"
    num_outs: 4
    start_level: 0
    out_channels: 256
    in_channels: [256, 512, 1024, 2048]
    add_extra_convs: "on_output"
    relu_before_extra_convs: true
  depth_branch:
    type: "dense_depth"
    embed_dims: "${model.embed_dims}"
    num_depth_layers: 3
    loss_weight: 0.2
  head:
    type: "sparse4d"
    num_output: 300
    cls_threshold_to_reg: 0.05
    decouple_attn: true
    return_feature: true
    use_reid_sampling: false
    embed_dims: "${model.embed_dims}"
    num_groups: 8
    num_decoder: 6
    num_single_frame_decoder: 1
    drop_out: 0.1
    temporal: true
    with_quality_estimation: true
    instance_bank:
      num_anchor: 900
      anchor: ???
      num_temp_instances: 600
      confidence_decay: 0.8
      feat_grad: false
      default_time_interval: 0.033333
      embed_dims: "${model.embed_dims}"
      use_temporal_align: "${model.use_temporal_align}"
      reset_on_time_gap: false
    anchor_encoder:
      type: 'SparseBox3DEncoder'
      vel_dims: 3
      embed_dims: [128, 32, 32, 64]
      mode: 'cat'
      output_fc: false
      in_loops: 1
      out_loops: 4
    operation_order: [
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm",
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm",
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm",
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm",
      "deformable", "ffn", "norm", "refine", "temp_gnn", "gnn", "norm",
      "deformable", "ffn", "norm", "refine"
    ]
    temp_graph_model:
      type: "MultiheadAttention"
      embed_dims: 512
      num_heads: 8
      batch_first: true
      dropout: 0.1
    graph_model:
      type: "MultiheadAttention"
      embed_dims: "${model.head.temp_graph_model.embed_dims}"
      num_heads: "${model.head.temp_graph_model.num_heads}"
      batch_first: true
      dropout: "${model.head.temp_graph_model.dropout}"
    norm_layer:
      type: "LN"
      normalized_shape: "${model.embed_dims}"
    ffn:
      type: "AsymmetricFFN"
      in_channels: 512
      pre_norm:
        type: "LN"
      embed_dims: 256
      feedforward_channels: 1024
      num_fcs: 2
      ffn_drop: 0.1
      act_cfg:
        type: "ReLU"
        inplace: true
    deformable_model:
      embed_dims: "${model.embed_dims}"
      num_groups: 8
      num_levels: 4
      attn_drop: 0.15
      use_deformable_func: true
      use_camera_embed: false
      residual_mode: "cat"
      kps_generator:
        embed_dims: "${model.embed_dims}"
        num_learnable_pts: 6
        fix_scale:
          - [0, 0, 0]
          - [0.45, 0, 0]
          - [-0.45, 0, 0]
          - [0, 0.45, 0]
          - [0, -0.45, 0]
          - [0, 0, 0.45]
          - [0, 0, -0.45]
    refine_layer:
      type: "SparseBox3DRefinementModule"
      embed_dims: "${model.embed_dims}"
      refine_yaw: true
      with_quality_estimation: true
    sampler:
      num_dn_groups: 5
      num_temp_dn_groups: 3
      dn_noise_scale: [2.0, 2.0, 2.0, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]
      max_dn_gt: 128
      add_neg_dn: true
      cls_weight: 2.0
      box_weight: 0.25
      reg_weights: [2.0, 2.0, 2.0, 0.5, 0.5, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0]
      use_temporal_align: "${model.use_temporal_align}"
    visibility_net:
      type: "visibility_net"
      embedding_dim: 256
      hidden_channels: 32
    loss:
      reg:
        type: "sparse_box_3d"
        box_weight: 0.25
        cls_allow_reverse: [5, 6, 7]
      cls:
        type: "focal"
        use_sigmoid: true
        gamma: 2.0
        alpha: 0.25
        loss_weight: 2.0
      id:
        type: "cross_entropy_label_smooth"
        num_ids: "${dataset.num_ids}"
    bnneck:
      type: "bnneck"
      feat_dim: 256
      num_ids: "${dataset.num_ids}"
    decoder:
      type: "SparseBox3DDecoder"
      score_threshold: 0.05
    reg_weights: [2.0, 2.0, 2.0, 1 ,1, 1, 1, 1, 1, 1, 1]

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`type`	string	Model type	sparse4d
`embed_dims`	int	Embedding dimensions	256	1	inf
`use_grid_mask`	bool	Use grid mask	True
`use_deformable_func`	bool	Use deformable function	True
`input_shape`	list	Input image shape	[1408, 512]			FALSE
`backbone`	collection	Backbone configuration				FALSE
`neck`	collection	Neck configuration				FALSE
`depth_branch`	collection	Depth branch configuration				FALSE
`head`	collection	Head configuration				FALSE
`use_temporal_align`	bool	Use temporal alignment	False
`cotrain_param_touch`	bool	Keep route-specific parameters in the distributed autograd graph	False

Backbone Configuration (model.backbone)#

Configuration for the model’s backbone network. The supported backbone types are resnet_50 and resnet_101.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	Backbone type	resnet_101

Head Configuration (model.head)#

Top-level configuration for the detection and tracking head.

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`type`	string	Head type	sparse4d
`num_output`	int	Number of output instances	300	1	inf
`cls_threshold_to_reg`	float	Classification threshold for regression	0.05	0	1
`decouple_attn`	bool	Decouple attention	True
`return_feature`	bool	Return instance features	True
`use_reid_sampling`	bool	Use Re-ID sampling	False
`embed_dims`	int	Embedding dimensions	256	1	inf
`reid_dims`	int	Re-ID dimensions	0	0	inf
`num_groups`	int	Number of groups	8	1	inf
`num_decoder`	int	Number of decoder layers	6	1	inf
`num_single_frame_decoder`	int	Number of single-frame decoder layers	1	1	inf
`drop_out`	float	Dropout rate	0.1	0	1
`temporal`	bool	Enable temporal modeling	True
`with_quality_estimation`	bool	Enable quality estimation	True
`operation_order`	list	Operation order	[‘deformable’, ‘ffn’, ‘norm’, ‘refine’, ‘temp_gnn’, ‘gnn’, ‘norm’, ‘deformable’, ‘ffn’, ‘norm’, ‘refine’, ‘temp_gnn’, ‘gnn’, ‘norm’, ‘deformable’, ‘ffn’, ‘norm’, ‘refine’, ‘temp_gnn’, ‘gnn’, ‘norm’, ‘deformable’, ‘ffn’, ‘norm’, ‘refine’, ‘temp_gnn’, ‘gnn’, ‘norm’, ‘deformable’, ‘ffn’, ‘norm’, ‘refine’, ‘temp_gnn’, ‘gnn’, ‘norm’, ‘deformable’, ‘ffn’, ‘norm’, ‘refine’]			FALSE
`visibility_net`	collection	Visibility net configuration				FALSE
`instance_bank`	collection	Instance bank configuration				FALSE
`anchor_encoder`	collection	Anchor encoder configuration				FALSE
`sampler`	collection	Sampler configuration				FALSE
`reg_weights`	list	Regression weights	[2.0, 2.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]			FALSE
`loss`	collection	Loss configuration				FALSE
`bnneck`	collection	BN neck configuration				FALSE
`deformable_model`	collection	Deformable model configuration				FALSE
`refine_layer`	collection	Refine layer configuration				FALSE
`valid_vel_weight`	float	Valid velocity weight	-1	-1	inf
`graph_model`	collection	Graph model configuration				FALSE
`temp_graph_model`	collection	Temp graph model configuration				FALSE
`decoder`	collection	Decoder configuration				FALSE
`norm_layer`	collection	Norm layer configuration				FALSE
`ffn`	collection	FFN configuration				FALSE
`loose_to_tight`	collection	Loose-to-tight geometric 2D distillation configuration				FALSE

Deformable Model Configuration (model.head.deformable_model)#

Configuration for the deformable attention mechanism.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`embed_dims`	int	Embedding dimensions	256	1	infinity
`num_groups`	int	Number of groups	8	1	infinity
`num_levels`	int	Number of levels	4	1	infinity
`attn_drop`	float	Attention dropout	0.15	0	1
`use_deformable_func`	bool	Use deformable function	True
`use_camera_embed`	bool	Use camera embedding	False
`residual_mode`	categorical	Residual mode	cat			cat,add
`num_cams`	int	Number of cameras	6	1	infinity
`max_num_cams`	int	Maximum number of cameras	20	1	infinity
`proj_drop`	float	Projection dropout	0.0	0	1
`kps_generator`	collection	KPS generator config					false

Instance Bank Configuration (model.head.instance_bank)#

Configuration for managing object instances over time.

Field	value_type	description	default_value	valid_min	valid_max
`num_anchor`	int	Number of anchors	900	1	inf
`anchor`	string	Path to anchor file
`num_temp_instances`	int	Number of temporal instances	600	0	inf
`confidence_decay`	float	Confidence decay factor	0.8	0	1
`feat_grad`	bool	Enable gradients for features	False
`default_time_interval`	float	Default time interval	0.033333	0	inf
`embed_dims`	int	Embedding dimensions	256	1	inf
`use_temporal_align`	bool	Use temporal alignment	False
`reset_on_time_gap`	bool	Reset recurrent state when timestamps cross a clip boundary	False
`grid_size`	float	Grid size

When reset_on_time_gap is true and the timestamps of all samples in a batch differ from the cached frame by more than the maximum time interval, which is fixed at 2 in the instance bank, the instance bank discards its cached temporal state. Tracking IDs continue to increase and are not reused.

Anchor Encoder Configuration (model.head.anchor_encoder)#

Configuration for encoding anchor information.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	Anchor encoder type	SparseBox3DEncoder
`vel_dims`	int	Velocity dimensions	3	1	infinity
`embed_dims`	list	Embedding dimensions	[128, 32, 32, 64]				false
`mode`	categorical	Mode	cat			cat,add
`output_fc`	bool	Fully Connected Layer	False
`in_loops`	int	In loops	1	1	infinity
`out_loops`	int	Out loops	4	1	infinity
`pos_embed_only`	bool	Pos embed only	False

Sampler Configuration (model.head.sampler)#

Configuration for sampling positive and negative examples during training.

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`num_dn_groups`	int	Number of De-Noising groups	5	1	infinity
`num_temp_dn_groups`	int	Number of temporal DN groups	3	0	infinity
`dn_noise_scale`	list	De-Noising scale	[2.0, 2.0, 2.0, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]			false
`max_dn_gt`	int	Maximum DN ground truth	128	1	infinity
`add_neg_dn`	bool	Add negative DN	True
`cls_weight`	float	Classification weight	2.0	0	infinity
`box_weight`	float	Box weight	0.25	0	infinity
`reg_weights`	list	Regression weights	[2.0, 2.0, 2.0, 0.5, 0.5, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0]			false
`use_temporal_align`	bool	Use temporal alignment	False
`gt_assign_threshold`	float	Ground Truth assign threshold	0.5	0	1

Loss Configuration (model.head.loss)#

This section details the different loss components used in the model head.

Field	value_type	description	automl_enabled
`cls`	collection	Classification loss config	false
`reg`	collection	Regression loss config	false
`id`	collection	ID loss config	false

Classification Loss (model.head.loss.cls)#

Configuration for the classification loss.

Field	value_type	description	default_value	valid_min	valid_max
`type`	string	Classification loss type	focal
`use_sigmoid`	bool	Use sigmoid	True
`gamma`	float	Focal loss gamma	2.0	0	infinity
`alpha`	float	Focal loss alpha	0.25	0	1
`loss_weight`	float	Loss weight	2.0	0	infinity

Regression Loss (model.head.loss.reg)#

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`type`	string	Regression loss type	sparse_box_3d
`box_weight`	float	Box loss weight	0.25	0	infinity
`cls_allow_reverse`	list	Class allow reverse	[]			false

ID Loss (model.head.loss.id)#

Configuration for the ID / Re-ID loss.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	ID loss type	cross_entropy_label_smooth
`num_ids`	int	Number of IDs	70	1	infinity

BNNeck Configuration (model.head.bnneck)#

Field	value_type	description	default_value	valid_min	valid_max
`type`	string	Batch Normalization Neck	bnneck
`feat_dim`	int	Feature dimension	256	1	infinity
`num_ids`	int	Number of IDs	70	1	infinity

KPS Generator Configuration (model.head.deformable_model.kps_generator)#

Configuration for KeyPoint (Sampling) Generator.

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`embed_dims`	int	Embedding dimensions	256	1	infinity
`num_learnable_pts`	int	Number of learnable points	6	1	infinity
`fix_scale`	list	Fixed scale	[[0, 0, 0], [0.45, 0, 0], [-0.45, 0, 0], [0, 0.45, 0], [0, -0.45, 0], [0, 0, 0.45], [0, 0, -0.45]]			false

Refine Layer Configuration (model.head.refine_layer)#

Field	value_type	description	default_value	valid_min	valid_max
`type`	string	Refine layer type	sparse_box_3d_refinement_module
`embed_dims`	int	Embedding dimensions	256	1	infinity
`refine_yaw`	bool	Refine yaw	True
`with_quality_estimation`	bool	With quality estimation	True

Graph Model Configuration (model.head.graph_model and model.head.temp_graph_model)#

Configuration for graph-based modeling (e.g., GNN or attention) used for spatial and temporal relations.

Field	value_type	description	default_value	valid_min	valid_max
`type`	string	Graph model type	MultiheadAttention
`embed_dims`	int	Embedding dimensions	512	1	infinity
`num_heads`	int	Number of heads	8	1	infinity
`batch_first`	bool	Batch first	True
`dropout`	float	Dropout rate	0.1	0	1

Decoder Configuration (model.head.decoder)#

Configuration for the final output decoder.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	Decoder type	SparseBox3DDecoder
`score_threshold`	float	Score threshold	0.05	0	1

Norm Layer Configuration (model.head.norm_layer and model.head.ffn.pre_norm)#

Configuration for normalization layers.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	Norm layer type	LN
`normalized_shape`	int	Normalized shape	256	1	infinity

FFN Configuration (model.head.ffn)#

Configuration for Feed-Forward Networks used in the decoder layers.

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`type`	string	FFN type	AsymmetricFFN
`in_channels`	int	In channels	512	1	infinity
`pre_norm`	collection	Pre-norm config				false
`embed_dims`	int	Embedding dimensions	256	1	infinity
`feedforward_channels`	int	Feedforward channels	1024	1	infinity
`num_fcs`	int	Number of feedforward channels	2	1	infinity
`ffn_drop`	float	FFN dropout	0.1	0	1
`act_cfg`	collection	Activation config				false

Activation Configuration (model.head.ffn.act_cfg)#

Configuration for activation functions.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	string	Activation type	ReLU
`inplace`	bool	Inplace	True

Visibility Net Configuration (model.head.visibility_net)#

Configuration for the visibility prediction network.

Field	value_type	description	default_value	valid_min	valid_max
`type`	string	VisibilityNet type	visibility_net
`embedding_dim`	int	Embedding dimension	256	1	infinity
`hidden_channels`	int	Hidden channels	32	1	infinity

Loose-to-Tight Configuration (model.head.loose_to_tight)#

Configuration for the optional loose-to-tight (LTT) losses used for 2D-to-3D geometric distillation. The LTT losses are disabled by default because enable defaults to false. Refer to 2D-to-3D Geometric Distillation for the full workflow.

Field	value_type	description	default_value	valid_min	valid_max
`enable`	bool	Enable loose-to-tight 2D distillation	False
`mlp_ckpt`	string	Path to the trained frozen loose-to-tight MLP checkpoint
`loss_weight`	float	Weight for supervised loose-to-tight 2D box loss	0.1	0	inf
`num_classes`	int	Number of classes used by the loose-to-tight MLP; zero derives the taxonomy from dataset.classes	0	0	inf
`tight_l1_weight`	float	Weight for the tight-box L1 term	1.0	0	inf
`containment_weight`	float	Weight for the visible-box containment term	1.0	0	inf
`box2d_key`	string	Batch key for visible 2D ground-truth boxes	gt_boxes_2d_visible
`occ_key`	string	Batch key for 2D occlusion weights	gt_occ_weight
`instance_id_key`	string	Batch key for instance identifiers	instance_id
`ego2cam_key`	string	Batch key for ego-to-camera transforms	cam2world_transform
`min_gt_area`	float	Minimum visible 2D ground-truth area in pixels	1.0	0	inf
`eps`	float	Minimum projection depth used for numerical stability	0.1	0	inf
`pseudo_enable`	bool	Enable 2D pseudo-label loss for batches without 3D ground truth	False
`det_box_key`	string	Batch key for per-camera 2D detection boxes	det_boxes_2d
`det_cls_key`	string	Batch key for per-camera 2D detection classes	det_classes_2d
`det_score_key`	string	Batch key for per-camera 2D detection scores	det_scores_2d
`has_3d_gt_key`	string	Batch key indicating whether a sample has 3D ground truth	has_3d_gt
`giou_thr`	float	Minimum generalized IoU for pseudo-label matching	0.3	-1	1
`cost_giou`	float	GIoU component weight in pseudo-label matching cost	2.0	0	inf
`cost_l1`	float	L1 component weight in pseudo-label matching cost	1.0	0	inf
`cost_cls`	float	Classification component weight in pseudo-label matching cost	1.0	0	inf
`det_score_thr`	float	Minimum 2D detection confidence used for matching	0.0	0	1
`min_cams`	int	Minimum number of cameras supporting a pseudo match	1	1	inf
`dedup_dist`	float	BEV distance used to deduplicate pseudo matches; zero disables it	0.0	0	inf
`class_gate`	bool	Require class agreement during pseudo-label matching	True
`pseudo_box_weight`	float	Weight for pseudo-label 2D box loss	0.1	0	inf
`pseudo_cls_weight`	float	Weight for pseudo-label classification loss	1.0	0	inf

Neck Configuration (model.neck)#

Configuration for the model’s neck (Feature Pyramid Network).

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	categorical	Neck - Feature Pyramid Network	FPN			FPN
`num_outs`	int		4	1	infinity
`start_level`	int	Start level for FPN	0	0	infinity
`out_channels`	int	Output channels	256	1	infinity
`in_channels`	list	Input channels	[256, 512, 1024, 2048]				false
`add_extra_convs`	categorical	Type of extra conv	on_output			on_input,on_lateral,on_output,False
`relu_before_extra_convs`	bool	Apply ReLU before extra convs	True

Depth Branch Configuration (model.depth_branch)#

Configuration for the depth estimation branch.

Field	value_type	description	default_value	valid_min	valid_max
`type`	string	Depth branch type	dense_depth
`embed_dims`	int	Embedding dimensions	256	1	infinity
`num_depth_layers`	int	Number of depth layers	3	1	infinity
`loss_weight`	float	Weight for depth loss	0.2	0	infinity

train#

The train config contains the parameters related to training. They are described as follows:

train:
  num_epochs: 5
  num_nodes: 1
  num_gpus: 1
  validation_interval: 1
  checkpoint_interval: 1
  pretrained_model_path: ???
  precision: bf16
  scrub_nan_gradients: false
  optim:
    type: "adamw"
    lr: 0.0001
    weight_decay: 0.001
    paramwise_cfg:
      custom_keys:
        img_backbone:
          lr_mult: 0.25
    grad_clip:
      max_norm: 25
      norm_type: 2
    lr_scheduler:
      policy: "cosine"
      warmup: "linear"
      warmup_iters: 500
      warmup_ratio: 0.333333
      min_lr_ratio: 0.001

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`num_gpus`	int	The number of GPUs to run the train job.	1	1
`gpu_ids`	list	List of GPU IDs to run the training on. The length of this list must be equal to the number of gpus in train.num_gpus.	[0]				FALSE
`num_nodes`	int	Number of nodes to run the training on. If > 1, then multi-node is enabled.	1	1
`seed`	int	The seed for the initializer in PyTorch. If < 0, disable fixed seed.	1234	-1	inf
`cudnn`	collection	cuDNN configuration					FALSE
`num_epochs`	int	Number of epochs to run the training.	10	1	inf
`checkpoint_interval`	int	Checkpoint interval in epochs	1	1	inf
`checkpoint_interval_unit`	categorical	The unit of the checkpoint interval.	epoch			epoch,step
`validation_interval`	int	Validation interval in epochs	1	1	inf
`resume_training_checkpoint_path`	string	Path to the checkpoint to resume training from.
`results_dir`	string	Path to where all the assets generated from a task are stored.
`checkpointer`	collection	Configuration for bounded metric-ranked checkpoint saving					FALSE
`pretrained_model_path`	string	Path to pretrained model
`optim`	collection	Optimizer configuration					FALSE
`precision`	categorical	Precision	bf16			bf16,fp16,fp32
`scrub_nan_gradients`	bool	Opt in to classification-logit clipping, non-finite loss recovery, and NaN-gradient scrubbing after backward. Gradient infinities remain visible to mixed-precision overflow detection	False

Set scrub_nan_gradients: true to clip classification logits to the range -50 to 50, replace a non-finite loss with zero for that batch, and replace NaN gradients with zero. By default, training stops with an error when a loss is non-finite.

optim#

The optim parameter defines the configuration for the optimizer in training (AdamW by default), including the learning rate, learning scheduler, and weight decay.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`type`	categorical	Optimizer type	adamw			adamw,adam,sgd
`lr`	float	Learning rate	5e-05	0	infinity		TRUE
`weight_decay`	float	Weight decay coefficient	0.001
`momentum`	float	Momentum for SGD	0.9
`paramwise_cfg`	collection	Parameter-wise configuration	{‘custom_keys’: {‘img_backbone’: {‘lr_mult’: 0.2}}}				false
`grad_clip`	collection	Gradient clipping configuration	{‘max_norm’: 25, ‘norm_type’: ‘L2’}				false
`lr_scheduler`	collection	Learning rate scheduler configuration	{‘policy’: ‘cosine’, ‘warmup’: ‘linear’, ‘warmup_iters’: 500, ‘warmup_ratio’: 0.333333, ‘min_lr_ratio’: 0.001}				false

evaluate#

The evaluate configuration contains the parameters related to evaluation. Currently, TAO supports evaluation only on a single GPU with batch size 1. The evaluate dataloader always uses batch size 1, so the batch_size parameter in the following table has no effect on Sparse4D. The parameters are described as follows:

evaluate:
  checkpoint: ${results_dir}/train/sparse4d_model_latest.pth

Field	value_type	description	default_value	valid_min	automl_enabled
`num_gpus`	int	The number of GPUs to run the evaluation job.	1	1
`gpu_ids`	list	List of GPU IDs to run the evaluation on. The length of this list must be equal to the number of gpus in evaluate.num_gpus.	[0]		FALSE
`num_nodes`	int	Number of nodes to run the evaluation on. If > 1, then multi-node is enabled.	1	1
`checkpoint`	string	Path to the checkpoint used for evaluation.	???
`trt_engine`	string	Path to the TensorRT engine to be used for evaluation. This only works with `tao-deploy`.
`results_dir`	string	Path to where all the assets generated from a task are stored.
`batch_size`	int	The batch size of the input Tensor. This is important if batch_size > 1 for large dataset.	-1	-1
`metrics`	list	Metrics to evaluate	[‘detection’]		FALSE
`tracking`	collection	Tracking configuration			FALSE

Set the evaluate checkpoint path in the evaluate specification:

visualize#

The visualize config contains the parameters related to visualization. They are described as follows:

Field	value_type	description	default_value	valid_min	valid_max
`show`	bool	Show visualization	False
`vis_dir`	string	Visualization directory	./vis
`vis_score_threshold`	float	Visualization score threshold	0.25	0	1
`n_images_col`	int	Number of images per column	6	1	inf
`viz_down_sample`	int	Visualization down sample	3	1	inf

inference#

The inference configuration contains the parameters related to inference. Currently, TAO supports inference only on a single GPU with batch size 1. The inference dataloader always uses batch size 1, so the batch_size parameter in the following table has no effect on Sparse4D. They are described as follows:

inference:
  checkpoint: ???
  output_nvschema: true
  nvschema_fps: 0.0
  jsonfile_prefix: "sparse4d_pred"

Field	value_type	description	default_value	valid_min	valid_max	automl_enabled
`num_gpus`	int	The number of GPUs to run the inference job.	1	1
`gpu_ids`	list	List of GPU IDs to run the inference on. The length of this list must be equal to the number of gpus in inference.num_gpus.	[0]			FALSE
`num_nodes`	int	Number of nodes to run the inference on. If > 1, then multi-node is enabled.	1	1
`checkpoint`	string	Path to checkpoint file	???
`trt_engine`	string	Path to the TensorRT engine to be used for inference. This only works with `tao-deploy`.
`results_dir`	string	Path to where all the assets generated from a task are stored.
`batch_size`	int	The batch size of the input Tensor. This is important if batch_size > 1 for large dataset.	-1	-1
`jsonfile_prefix`	string	JSON file prefix	sparse4d_pred
`output_nvschema`	bool	Output NVSchema	True
`nvschema_fps`	float	Frame rate used for NVSchema timestamps; zero uses the spatialai-data-utils default	0.0	0.0	inf
`tracking`	collection	Tracking configuration				FALSE

The nvschema_fps parameter sets the frame rate that TAO uses for the timestamps in the NVSchema output. A value of 0.0 uses the default frame rate of the spatialai-data-utils package. TAO applies the value only when output_nvschema is true. Both the inference and evaluate actions read this value from the inference section.

Set the inference checkpoint path in the inference specification:

export#

The export configuration contains the parameters related to export. Currently, TAO supports export only with batch size 1 and a dynamic number of camera sensors. They are described as follows:

export:
  results_dir: ???
  checkpoint: ???
  onnx_file: ???

Field	value_type	description	default_value	valid_min	valid_options
`results_dir`	string	Path to where all the assets generated from a task are stored.
`gpu_id`	int	The index of the GPU to build the TensorRT engine.	0
`checkpoint`	string	Path to the checkpoint file to run export.	???
`onnx_file`	string	Path to the onnx model file.	???
`on_cpu`	bool	Flag to export CPU compatible model.	False
`input_channel`	ordered_int	Number of channels in the input Tensor.	3	1	1,3
`input_width`	int	Width of the input image tensor.	960	32
`input_height`	int	Height of the input image tensor.	544	32
`opset_version`	int	Operator set version of the ONNX model used to generate the TensorRT engine.	17	1
`batch_size`	int	The batch size of the input Tensor for the engine. A value of `-1` implies dynamic tensor shapes.	-1	-1
`verbose`	bool	Flag to enable verbose TensorRT logging.	False
`format`	categorical	File format to export to.	onnx		onnx,xdl

Set the export checkpoint path in the export specification:

Training the Model#

Run Sparse4D training inside the TAO PyTorch environment with a complete experiment specification:

sparse4d train -e /specs/experiment_spec.yaml

For calibrated real scenes with 2D supervision, use the workflow in 2D-to-3D Geometric Distillation.

2D-to-3D Geometric Distillation#

Use 2D-to-3D geometric distillation to adapt Sparse4D to calibrated multi-camera real-world imagery with 2D annotations or detector pseudo-labels while retaining 3D-labeled data in the training mix. The term 2D-to-3D describes the direction of supervision: 2D targets supervise the 3D model. During training, 3D-to-2D projection maps Sparse4D’s predicted 3D boxes into camera images, where they can be compared with the 2D targets. Sparse4D remains a multi-camera 3D detector and tracker; the deployed model outputs 3D predictions.

This guide requires a TAO PyTorch build containing model.head.loose_to_tight and the co-training dataset fields in the experiment specification. The data-service commands require a TAO Data Services build with annotations sparse4d_prepare and annotations convert support for aicity.load_annotations=false. Check that the specifications shipped with your build expose aicity.load_annotations and aicity.fps before using these options. The optional loose-to-tight (LTT) training losses are disabled by default. Refer to the Sparse4D Compatibility Notes for training and deployment behavior that changed with this implementation.

How Geometric Supervision Works#

A 3D cuboid’s eight projected corners define a loose 2D bounding rectangle. The actual object occupies a tighter image-space box. The released LTT multilayer perceptron (MLP) predicts this correction using the class, box extent, camera-relative orientation, projected box, image size, and distance. The released MLP corrects toward the tight amodal 2D box (the object’s full extent), not just its visible fragment.

During Sparse4D training, the MLP is frozen and its four predicted edge scales are detached. Applying those scales to the projected box preserves gradients through the box projection into Sparse4D. On calibrated real scenes without 3D ground truth, corrected projections are matched to per-camera 2D detections using Hungarian matching, followed by cross-camera consistency filtering. Box losses combine generalized intersection over union (GIoU), normalized L1, and visible-box containment; classification losses supervise the matched classes and background queries. The 2D teacher, such as RT-DETR, supplies offline detections.

Run the data preparation commands inside a compatible TAO Data Services environment and the training commands inside a compatible TAO PyTorch environment. Mount your data, specifications, and checkpoints at the paths used in these commands. This procedure uses the container entry points.

Data route	Inputs	Training signal
3D-labeled scenes	Synchronized images, intrinsics and extrinsics, 3D boxes and IDs; optional visible-2D sidecars	Standard 3D losses plus optional LTT visible-2D loss
Calibrated real scenes without 3D labels	Synchronized images, intrinsics and extrinsics, BEV groups, 2D teacher detections	LTT projected-box and classification pseudo-losses

Preserve a 3D-supervised route and validate 3D accuracy independently.

Prepare Compatible Inputs#

Start with a complete Sparse4D fine-tuning specification and pretrained checkpoint. Keep the backbone, input resolution, anchors, and class order compatible with that checkpoint. Use paths visible inside the container for all artifacts.

Use the same ordered class_names during preparation and dataset.classes during training. The released LTT checkpoint retains the seven-class warehouse order, and the NumPy archive (NPZ) caches record the class order that you prepare them with. Training stops when the class names in the LTT checkpoint or in an RT-DETR cache do not match dataset.classes. TAO does not check the class names of the visible-2D sidecars. Map teacher labels to that order through explicit aliases.
Supply separate camera intrinsics and rigid world-to-camera extrinsics. Use consistent units, axes, box centers, and image coordinates. A projection-only cameraMatrix does not provide the separate matrices required for geometric supervision. Inspect projected boxes before producing a full cache.
Align camera names and source frame IDs across images, calibration, info PKLs, and 2D caches. Scene names must not contain the reserved + BEV-group separator. Check joins after conversion or sampling.
Retain image paths, camera geometry, sequence metadata, and timestamps in real info pickle (PKL) files. Use gt_boxes=None when 3D annotations are unavailable; an annotated frame with no objects is different. Prepare these info PKLs with the conversion command below, then prepare the corresponding detection caches with the RT-DETR operation.
Reserve separate scenes or sequences for validation before fine-tuning Sparse4D or selecting teacher thresholds. Do not mix held-out real scenes into pseudo-label training.

Convert Calibrated Real Scenes Without 3D Annotations#

Use the unlabeled-scene specification from a TAO Data Services build containing the annotation-free converter. Copy it to /specs/aicity2ovpkl_unlabeled.yaml.

The specification lists seven classes in aicity.class_config.CLASS_LIST, and the converter stores this list in the metadata.class_names key of each PKL.

Arrange each scene as /data/real/train/<scene>/calibration.json plus <camera>/rgb/000000000.jpg, 000000001.jpg, and so on. NVSchema calibration must include every camera’s intrinsic matrix and world-to-camera pose, with camera IDs matching the image directories. The rgb_00000.jpg naming convention also works.

Use synchronized camera sequences that are contiguous from zero. The converter computes each frame timestamp as the frame index divided by aicity.fps, so set aicity.fps to the capture rate of the sequences:

annotations convert -e /specs/aicity2ovpkl_unlabeled.yaml \
  aicity.root=/data/real aicity.split=train aicity.fps=30 \
  results_dir=/data/real_infos

The shipped specification sets aicity.load_annotations=false, disables recentering, and uses all calibrated cameras as one group per scene. It writes /data/real_infos/train/<scene>_infos_train.pkl with gt_boxes=None, no depth paths, and timestamps in seconds. It requires neither ground_truth.json nor depth files and skips anchor initialization; reuse the pretrained model’s anchors. Labeled conversion still defaults to loading 3D ground truth.

Each PKL stores frame records in infos. Each record retains token, scene_name, timestamp, cams, and gt_boxes=None for unlabeled frames. The cams mapping is keyed by camera ID; each camera entry retains its data_path. For the complete key list, refer to Unlabeled OVPKL Files. Preserve these field names and conventions:

Location	Field	Meaning
Frame record	`frame_idx`	Zero-based source frame ID, matching the 2D teacher cache; starts at zero for each sequence.
Camera entry in `cams`	`cam_intrinsic`	3-by-3 camera intrinsic matrix.
Camera entry in `cams`	`sensor2world_transform`	4-by-4 world-to-camera extrinsic matrix for the default unrecentered specification.

sensor2world_transform is a legacy field name. The loader uses this matrix directly with the camera intrinsics to project 3D points; it does not invert it. If BEV-group recentering is enabled, the matrix must instead map the corresponding group-local 3D coordinates into the camera frame.

The shipped specification sets aicity.rgb_format=jpg. For HDF5 image arrays, set aicity.rgb_format=h5 and provide <camera>.h5 or <camera>.hdf5 files with arrays in height, width, channel (HWC) order at rgb/rgb_00000.jpg, rgb/rgb_00001.jpg, and so on. Use dataset.use_h5_file_for_rgb: true when training on these inputs. Keep image paths visible at the same location inside the training environment.

The converter rejects empty, gapped, or unequal camera sequences. Synchronize irregular or dropped-frame input before conversion, then run the 2D teacher on those same frames. Inspect projections before training. The existing camera-grouping options can also generate multiple BEV groups; keep the calibration and cache joins consistent with each group.

Download the Released LTT Checkpoint#

For the released seven-class warehouse taxonomy used in this guide, reuse _loose_to_tight_mlp.pth from the Sparse4D ResNet-50 trainable_v3.0 Bundle on NGC. With the NGC command line interface (CLI), download just the LTT file:

mkdir -p /data
ngc registry model download-version nvidia/tao/sparse4d_rn50:trainable_v3.0 \
  --file _loose_to_tight_mlp.pth --dest /data

The downloaded path is /data/sparse4d_rn50_vtrainable_v3.0/_loose_to_tight_mlp.pth, including the leading underscore in the filename. Make it visible inside the TAO training environment and set model.head.loose_to_tight.mlp_ckpt to that path, as in the training fragment below. The checkpoint’s ordered class_names must match dataset.classes; validate its corrections on the target data.

This workflow supports only the released checkpoint’s seven classes in their original order. Custom taxonomies require a matching LTT checkpoint and are outside this guide; changing class names in the configuration does not adapt the released weights.

Use the released checkpoint directly. Continue with teacher caches, mixed-training inputs, and optional visible-2D sidecars below.

Retain the external MLP artifact for later training and checkpoint resume: it is frozen and deliberately excluded from Sparse4D model checkpoints. Evaluation, inference, and export do not need this artifact.

Prepare Artifacts with TAO Data Services#

The preparation entry point and shipped sparse4d_prepare specification are included in TAO Data Services. Run the commands in this section in that environment. For the command reference and configuration tables, refer to Preparing Sparse4D Co-Training Artifacts.

The ltt_2dgt operation reads raw scene directories. Each scene directory contains either a ground_truth_final directory of ground_truth_<frame>.json files or a single ground_truth.json file. Each annotation provides object type and 2d bounding box, and can provide object id and 2d bounding box visible. Convert other annotation formats first.

The shipped specification and the schema default list four classes (person, gr1_t2, agility_digit, and nova_carter). The following example therefore sets the released seven-class order explicitly. Save it as /specs/sparse4d_prepare.yaml and replace the example paths, scene names, and label aliases with your dataset’s values. Retain the released class order:

operation: ltt_2dgt
results_dir: /results/sparse4d_prepare
overwrite: false
class_names: [person, gr1_t2, agility_digit, nova_carter, transporter, forklift, pallet_truck]
subclass_map:
  person: [Person, Human]
  gr1_t2: [FourierGR1T2, Fourier_GR1_T2_Humanoid]
  agility_digit: [AgilityDigit, Agility_Digit_Humanoid]
  nova_carter: [NovaCarter, Nova_Carter]
  transporter: [Transporter]
  forklift: [Forklift]
  pallet_truck: [Pallet_Truck]

ltt_2dgt:
  selection:
    scene_dirs: [/data/labeled/SceneA]
  output_dir: /data/ltt_2dgt
  annotation_version: v0.1

rtdetr_2d:
  input_dir: /data/RealWarehouse/rt-detr
  output_path: /data/rtdetr/RealWarehouse__rtdetr2d.npz
  scene_name: RealWarehouse
  camera_map: {camera1: Camera1, camera2: Camera2}
  class_map: {pallet: null}
  confidence_threshold: 0.4

lazy_index:
  annotation_source: /data/mixed_train_split.txt

TAO executes only the block selected by operation. The teacher threshold is an example and must be checked on your real data.

Every sparse4d_prepare command stops with an error when any planned output already exists, including the run summary sparse4d_prepare_summary.json in results_dir. Each of the following commands therefore sets its own results_dir. To replace existing outputs, set overwrite=true.

For ltt_2dgt, each source object type must appear in class_names, match an alias in subclass_map, or appear in ltt_2dgt.drop_names, which defaults to [pallet]. Otherwise the command raises an error that names the class.

For the supervised LTT loss on 3D-labeled scenes, source object id values must match the labeled PKL’s instance_inds. The loader joins those IDs with the frame ID and camera name to attach each visible 2D target. The preparation command accepts annotations without object id and writes instance_id: -1, but that value does not identify a labeled object. Check the loader’s missing-ID or no-match warnings before training. This ID requirement applies to the supervised sidecar join; the separate 2D-only route uses geometric matching with RT-DETR caches and does not require instance IDs.

Prepare the optional visible-2D sidecars as follows:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml operation=ltt_2dgt \
  results_dir=/results/sparse4d_prepare/ltt_2dgt

For each calibrated real scene, run RT-DETR offline and arrange its KITTI label archives as /data/RealWarehouse/rt-detr/<camera>/labels.tar.gz. Map those camera directory names to the exact names in the Sparse4D info PKL.

Each label row has at least nine whitespace-separated fields: the class name is the first field, the box corners (x1, y1, x2, y2) are the fifth to eighth fields, and the confidence score is the last field. TAO ignores shorter rows. Explicitly alias or drop every teacher class outside the target taxonomy. Then convert:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml operation=rtdetr_2d \
  results_dir=/results/sparse4d_prepare/rtdetr_2d

This command normalizes existing detections; it does not run RT-DETR. Run it with the corresponding input, scene name, and output path for each real scene. Do not create teacher caches for 3D-labeled scenes that should stay on the 3D route when dataset.rtdetr_2d_mark_real is enabled.

The operation drops detections whose score is below confidence_threshold and boxes whose width or height is at most 1.0 in the units of the box coordinates. If you omit scene_name, the operation uses the name of the directory that contains input_dir.

Create /data/mixed_train_split.txt with one absolute, container-visible PKL path per line for the existing 3D-labeled data and the prepared calibrated real data. Include each PKL only once. Then build the index for lazy loading:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml operation=lazy_index \
  results_dir=/results/sparse4d_prepare/lazy_index

Rebuild the index after changing the split or moving data. Rebuilding an existing index requires overwrite=true. To control aggregate 2D and 3D sampling, refer to Configure and Train Sparse4D.

Artifact	Consumer
`<scene>__ltt2dgt.npz` (`ltt_2dgt/v1`)	`dataset.ltt_2dgt_sidecar_dir`
`<scene>__rtdetr2d.npz` (`ltt_rtdetr2d/v1`)	`dataset.rtdetr_2d_cache_dir`, or `rtdetr_2d_cache_path` for a single scene
3D-labeled and calibrated real info PKLs	Entries in `dataset.train_dataset.ann_file`
Lazy index beside the split	`dataset.lazy_load: true`
`_pkl_cam_counts.pkl` (written next to the split by default)	`dataset.pkl_cam_counts_path` (optional; read only when `dataset.pkl_sample_size` is greater than 0)

The LTT sidecar’s box2 and box3 arrays both contain 2D boxes: box2 is the full box and box3 is the visible box. When a visible box is missing or invalid, the operation copies the full box to box3 and sets the visibility weight occ to zero.

Training looks up each scene’s sidecar by file name: <scene>__ltt2dgt.npz in dataset.ltt_2dgt_sidecar_dir and <scene>__rtdetr2d.npz in dataset.rtdetr_2d_cache_dir. Before the lookup, TAO removes the +<group> suffix and the matching deduplication regular expression (dataset.ltt_2dgt_dedup_regex or dataset.rtdetr_2d_dedup_regex) from the scene name. Give the rtdetr_2d.output_path file the name <scene>__rtdetr2d.npz, where <scene> is the scene name after that removal.

The producers encode NPZ metadata as JSON in a uint8 _meta array, and TAO reads it with allow_pickle=False. Info PKLs and lazy indexes use the existing trusted TAO annotation format. For the complete key lists of the NPZ files, refer to Sparse4D Co-Training Artifact Formats.

TAO PyTorch also provides module commands that build the lazy index, the LTT sidecars, and the RT-DETR cache without TAO Data Services. Refer to Native Artifact Tools in TAO PyTorch.

Native Artifact Tools in TAO PyTorch#

TAO PyTorch includes Python module commands that build the lazy index, the LTT visible-2D sidecars, and the RT-DETR cache. They do not depend on TAO Data Services. You start each command with python -m nvidia_tao_pytorch.cv.sparse4d.tools.<module>, where <module> is a module name from the following table, or rtdetr_pseudo_labels, of which ltt_rtdetr_pseudo_labels is an alias. They are not console commands and not sparse4d tasks.

The following table lists each module, its output, and its key arguments:

Module	Output	Key arguments
`build_lazy_index`	Lazy index and, by default, `_pkl_cam_counts.pkl`	`ann_file`: positional; a split `.txt` file or a directory of PKL files `--force` or `-f`: rebuild existing data `--workers` or `-w`: the number of worker processes (defaults to the smaller of 32 and the CPU count) `--camera-counts-out`: override the sidecar path `--no-camera-counts-sidecar`: embed camera counts without writing the sidecar
`ltt_build_2dgt`	`<out-dir>/<scene>__ltt2dgt.npz`	`--out-dir`: required `--data-root`, `--scenes`, `--scenes-file`, `--scene-dir`, and `--train-split`: select scenes `--dedup-regex`: defaults to `^CT[\w.]+?__` `--class-config` `--anno-version`: `v0.0` or `v0.1` (defaults to `v0.1`) `--frame-stride`: defaults to 1 `--max-frames-per-scene`: defaults to 0
`ltt_rtdetr_pseudo_labels`	The NPZ file that `--out` names	`--rtdetr-dir` and `--out`: both required `--class-config`, `--class-map`, `--scene-name`, and `--cam-map` `--conf-thr`: defaults to 0.4 `--frame-stride`: defaults to 1 `--max-frames-per-cam`: defaults to 0

The ltt_rtdetr_pseudo_labels module is an alias of the rtdetr_pseudo_labels module and accepts the same arguments. The --class-config argument takes the path to a Python file that defines CLASS_LIST and, optionally, SUB_CLASS_DICT; the command imports this file. The --cam-map argument takes comma-separated <label-directory>=<camera-name> pairs, such as gopro1=GoPro1,gopro3=GoPro3. The --class-map argument takes a JSON object that maps detector labels to class names, and a null value drops that label. A detector label that does not match a taxonomy class and is not mapped by --class-map raises an error. For example, --class-map '{"Human":"person","pallet":null}' aliases Human to person and drops pallet.

The native commands differ from the TAO Data Services operations in these ways:

ltt_build_2dgt and rtdetr_pseudo_labels use the seven-class warehouse taxonomy unless you pass --class-config, while the TAO Data Services default is four classes.
ltt_build_2dgt skips ground-truth object types that are not in the taxonomy without raising an error.
ltt_build_2dgt and rtdetr_pseudo_labels overwrite existing output files without checking and have no overwrite option. build_lazy_index updates an existing index and re-reads every PKL when you pass --force.

For sample commands and the mapping from artifacts to specification fields, refer to Sparse4D Co-Training Artifact Tools.

Configure and Train Sparse4D#

Merge this fragment into a complete specification for the selected pretrained model. Set train.pretrained_model_path and retain the matching model, anchors, image preprocessing, optimizer, validation, and evaluation settings.

model:
  cotrain_param_touch: false
  head:
    loose_to_tight:
      enable: true
      mlp_ckpt: /data/sparse4d_rn50_vtrainable_v3.0/_loose_to_tight_mlp.pth
      num_classes: 0
      loss_weight: 0.1
      tight_l1_weight: 1.0
      containment_weight: 1.0
      pseudo_enable: true
      pseudo_box_weight: 0.1
      pseudo_cls_weight: 1.0
      giou_thr: 0.3
      min_cams: 1
      class_gate: true

dataset:
  classes: [person, gr1_t2, agility_digit, nova_carter, transporter, forklift, pallet_truck]
  train_dataset:
    ann_file: /data/mixed_train_split.txt
  lazy_load: true
  ltt_2dgt_sidecar_dir: /data/ltt_2dgt
  rtdetr_2d_cache_dir: /data/rtdetr
  rtdetr_2d_mark_real: true
  sync_route: true
  real_scene_keywords: [RealWarehouse]
  real_block_prob: 0.5
  scene_switch_iters: 100

Both enable and pseudo_enable must be true for geometric pseudo-label supervision. num_classes: 0 derives the taxonomy size from dataset.classes. min_cams: 1 permits supervision from one supporting view; increase it only when camera overlap and teacher coverage support that requirement. Match thresholds and loss weights need validation on the target data.

With sync_route: true, the sampler reads real_scene_keywords and real_block_prob and treats a scene as real when its name contains any entry of real_scene_keywords. With sync_route: false, the sampler ignores both fields. The keywords must match all 2D-supervised scene names and none of the 3D-supervised scene names. Each batch must have homogeneous 2D or 3D supervision. For distributed mixed training, set sync_route: true and a positive scene_switch_iters so ranks take compatible routes. real_block_prob: 0.5 is an example block probability; -1 selects automatic scene-count weighting. Leave cotrain_param_touch: false to use the training CLI’s find-unused-parameters distributed data parallel (DDP) strategy. Enabling parameter touch can advance optimizer state and apply weight decay to otherwise inactive parameters.

Use the regular training command inside the TAO PyTorch environment:

sparse4d train -e /specs/experiment_spec.yaml

There is no separate Sparse4D distill task. The standard training task selects losses from the batch’s supervision route and the configuration above.

Verify Supervision and Evaluate Adaptation#

Run a short training job before a long adaptation run. Inspect source images, corrected projections, teacher boxes, and sidecar joins. Check these signals:

Route	Expected signal
3D-labeled	Standard classification and regression losses; `loss_box_2d_*` when valid visible-2D sidecars are supplied
Calibrated real 2D	`loss_box_2d_pseudo_` and `loss_cls_pseudo_` on valid matched data

A missing cache, unresolved frame, or incomplete camera join sets has_2d_pseudo=False and skips geometric pseudo supervision for that sample. An explicitly processed frame with no detections is different: validity arrays preserve it and it can supervise background. Investigate join warnings and unexpected zero box losses instead of treating them as convergence. Confirm that teacher coverage is adequate before treating unmatched queries as background.

Evaluate the baseline and adapted checkpoints on the same held-out real scenes with independent 3D and tracking annotations. Report detection and tracking metrics separately and evaluate a synthetic holdout for regressions. Neither 2D box agreement nor a successful short training run establishes a 3D accuracy gain. Select confidence thresholds on validation data and keep test scenes separate.

Use standard Sparse4D evaluation, inference, and export after adaptation. The prediction stages do not construct the training criterion and need neither the external LTT MLP nor the offline RT-DETR teacher. Deploy the exported Sparse4D model with its matching anchors, class order, calibration, preprocessing, and compatible NVIDIA® TensorRT™ and plug-in stack. Refer to the “Temporal State and Deployment” section of the Sparse4D Compatibility Notes.

Behavior Changes in 7.3.0#

TAO 7.3.0 changes how Sparse4D trains, loads data, evaluates, and exports models. The lists below describe each change and what it means for your existing specifications and workflows. The 2D-to-3D distillation workflow has additional requirements; refer to 2D-to-3D Geometric Distillation.

These changes affect training:

The focal classification loss includes background queries, which earlier releases excluded. This changes the loss value and gradients, even when you do not use co-training.
Pixels without a valid depth value keep the value -1, and the dense-depth loss excludes them. Earlier releases clipped these pixels to 0.1 m. This changes the loss_dense_depth values and gradients.
TAO seeds the training sampler with train.seed plus the current epoch. Earlier releases used train.seed alone.
Training stops with a FloatingPointError when a loss is not finite, unless you set train.scrub_nan_gradients: true. Refer to train for what that option enables.

These changes affect data loading:

Setting dataset.pkl_sample_size to a value greater than 0 stops training at startup when the dataloader configuration or the lazy index does not meet the requirements. Refer to Dynamic Resampling and Dataset-Loading Options for the conditions. Rebuild an older index with annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml operation=lazy_index overwrite=true. Alternatively, run the native build_lazy_index module with --force.
The lazy-index builders and the training dataset read split files in the same way. They use the first token on each line, skip empty lines and lines that start with #, and reject duplicate PKL paths. They resolve relative paths against the working directory without resolving symlinks. Use absolute, container-visible paths, and rebuild the index after you move a dataset.

These changes affect evaluation, inference, and export:

The PyTorch CLI default evaluation protocol changes from center_distance to iou_3d. Core-generated specifications also default to iou_3d, with HOTA disabled. Refer to Evaluation Protocol and Metrics for the settings that retain each TAO 7.2.0 evaluation recipe.
Sparse4D resets its temporal and denoising caches when every sample in a batch crosses a group or scene boundary. When only some samples cross a boundary and use_temporal_align is enabled, TAO resets only the ground-truth index mappings of those samples. For the reset_on_time_gap option, refer to Instance Bank Configuration.
Tracking IDs are unique across the scenes that one run processes and do not restart at 0 for each scene.
The Open Neural Network Exchange (ONNX) export replaces each nv::MSDA sampling location with zero when its x or y coordinate is not strictly between 0 and 1. This includes NaN and infinity values. A deployment plug-in for this operator must skip sampling locations that are exactly zero. The exported graph differs from the graph of earlier releases, even for an unchanged 3D checkpoint.

For the complete list of compatibility items, refer to the Sparse4D Compatibility Notes.

Evaluating the Model#

The evaluate action reports detection metrics, including the mean average precision (mAP), and reports tracking metrics when you add tracking to evaluate.metrics. Refer to Evaluation Protocol and Metrics for the evaluation functions and the metric names.

Set evaluate.checkpoint to the selected model checkpoint and dataset.test_dataset.ann_file to the held-out labeled PKL or split. Preserve the checkpoint’s backbone, anchors, class order, and preprocessing in the complete experiment specification. Run the following command inside the TAO PyTorch environment:

sparse4d evaluate -e /specs/evaluate.yaml

Evaluation Protocol and Metrics#

The dataset.eval_dist_fcn field selects the evaluation distance function, which TAO uses to match predictions with ground-truth objects. Set it to center_distance, iou_3d, or both; any other value raises a ValueError. The dataset.eval_hota field adds higher order tracking accuracy (HOTA) evaluation to the tracking metrics that evaluate.metrics selects. TAO 7.3.0 defaults dataset.eval_dist_fcn to iou_3d and dataset.eval_hota to false in both the PyTorch CLI and Core-generated specifications. In TAO 7.2.0, the PyTorch CLI defaults were center_distance and false, while the Core schema defaults were iou_3d and true. Explicit settings in an existing specification remain unchanged.

TAO prefixes each metric name with a namespace, for example img_bbox_IoU3D/mAP with the default protocol. The namespace depends on the evaluation function and the metric group:

`dataset.eval_dist_fcn`	Metric group	Namespace
`center_distance`	Detection and tracking	`img_bbox_NuScenes/`
`center_distance`	HOTA	`img_bbox_HOTA/`
`iou_3d`	Detection and tracking	`img_bbox_IoU3D/`
`iou_3d`	HOTA	`img_bbox_HOTA_IoU3D/`

With both, TAO runs each evaluation function and reports both namespaces for every metric group. The evaluate.metrics field defaults to ['detection']. Add tracking to run tracking evaluation. TAO runs HOTA evaluation only when dataset.eval_hota is true and evaluate.metrics includes tracking, and only in the evaluate action. Training validation reports detection metrics only. If an evaluation run fails, TAO logs a warning and omits the metrics of that run.

To retain the TAO 7.2.0 PyTorch CLI evaluation protocol, set the following values:

dataset:
  eval_dist_fcn: "center_distance"
  eval_hota: false

To retain the TAO 7.2.0 Core tracking recipe, set dataset.eval_dist_fcn: iou_3d and dataset.eval_hota: true, and include tracking in evaluate.metrics.

Note

Metrics from different evaluation functions use different matching criteria, so they are not comparable. Compare results only when they use the same dataset.eval_dist_fcn.

TAO records the metric values in the status log (status.json) in the results directory of the training or evaluation task. The evaluate action also writes the formatted detection results (results_nusc.json) to the img_bbox subdirectory of inference.jsonfile_prefix. TAO writes the detection and tracking evaluation files to that img_bbox subdirectory for center_distance and to its img_bbox_IoU3D subdirectory for iou_3d; with both, TAO writes to both locations. TAO writes the HOTA evaluation files to the img_bbox subdirectory for every evaluation function.

Running Inference on the Model#

Set inference.checkpoint to the selected .pth checkpoint and dataset.test_dataset.ann_file to the input PKL or split in the complete experiment specification. Run the following command inside the TAO PyTorch environment:

sparse4d inference -e /specs/inference.yaml

The output contains JSON logs with object detection and tracking results for each frame.

The expected output is as follows:

{
    "version": "4.0",
    "id": "1", # Frame ID
    "sensorId": "bev-sensor-zone-c4", # BEV Sensor ID
    "timestamp": "2025-01-15T10:30:00.123Z", # Timestamp
    "objects": [
      {
        "id": "1", # Object ID
        "type": "Person", # Object Type
        "confidence": 0.887, # Object Confidence Score
        "coordinate": {
          "x": -1.5, # Object Center X Coordinate
          "y": 3.2, # Object Center Y Coordinate
          "z": 0.75 # Object Center Z Coordinate
        },
        "bbox3d": {
          "coordinates": [
            -1.5, # Object Centeroid X Coordinate
            3.2, # Object Centeroid Y Coordinate
            0.75, # Object Centeroid Z Coordinate
            0.5, # Object Width
            0.5, # Object Length
            0.5, # Object Height
            0.0, # Object Pitch
            0.0, # Object Roll
            1.57 # Object Yaw
          ],
          "embedding": [
            {} # Object Embedding
          ],
          "confidence": 0.887 # Object Confidence Score
        }
      },
      {
        "id": "2",
        "type": "Humanoid",
        "confidence": 0.752,
        "coordinate": {
          "x": 5.1,
          "y": -2.8,
          "z": 0.15
        },
        "bbox3d": {
          "coordinates": [
            5.1,
            -2.8,
            0.15,
            1.2,
            1.0,
            0.2,
            0.0,
            0.0,
            -1.04
          ],
          "embedding": [
            {}
          ],
          "confidence": 0.752
        }
      }
    ]
  }
  {
    # ... more frames
  }

Exporting the Model#

Set export.checkpoint to the selected checkpoint and export.onnx_file to the output path in the complete experiment specification. Keep the matching model architecture and anchors. Run the following command inside the TAO PyTorch environment to export an ONNX model for deployment:

sparse4d export -e /specs/export.yaml

The command stops with an error when the output ONNX file already exists. The exported graph replaces invalid sampling locations with zero; refer to Behavior Changes in 7.3.0 for the deployment plug-in requirement.

Quantization#

Sparse4D supports post-training quantization (PTQ) through TAO Quant. Set quantize.backend and quantize.mode as a pair. Use torchao with weight_only_ptq, or use modelopt.pytorch or modelopt.onnx with static_ptq.

Add a quantize section to your experiment specification. Set quantize.model_path to the checkpoint to quantize, or to an ONNX file for the modelopt.onnx backend. For the schema and the backend options, refer to the TAO Quant documentation. Run the following command inside the TAO PyTorch environment:

sparse4d quantize -e /specs/quantize.yaml

The command saves the quantized model to quantize.results_dir. The torchao backend writes quantized_model_torchao.pth, and the modelopt.pytorch backend writes quantized_model_modelopt.pytorch.pth.

Notes#

For the torchao backend, TAO ignores both the activation quantization settings in the configuration and the algorithm field.

Calibration Dataset (ModelOpt)#

When you use the modelopt.pytorch or modelopt.onnx backend with quantize.mode: static_ptq, provide a calibration dataset using dataset.quant_calibration_dataset. The command builds the calibration data loader only when quantize.mode is not weight_only_ptq and dataset.quant_calibration_dataset.images_dir is set.

Minimal example:

quantize:
  backend: "modelopt.pytorch"
  mode: "static_ptq"
  algorithm: "max"
  model_path: "/path/to/sparse4d_model.pth"
dataset:
  quant_calibration_dataset:
    images_dir: "/path/to/calib/images"

For more information, refer to TAO Quant overview and its Configuration and backend pages.

Deploying with TAO Deploy#

Use TAO Deploy to build a TensorRT engine from the ONNX model that sparse4d export writes and to run inference with that engine. Refer to Sparse4D with TAO Deploy for the gen_trt_engine and inference configuration.

Refer to the NVIDIA Spatial AI documentation page for more information about deploying a Sparse4D model to DeepStream through TensorRT engine generation.