Sparse4D with TAO Deploy#

The sparse4d command of TAO Deploy builds an NVIDIA^®TensorRT™ engine from the Open Neural Network Exchange (ONNX) file that the Sparse4D export action writes. It then runs multi-camera 3D detection and tracking on prepared frames with that engine. For information about training a Sparse4D model, refer to Sparse4D. For the export action, refer to Sparse4D export.

The command provides the gen_trt_engine and inference actions and a default_specs subtask that writes a default specification. You can also reach the two actions through the model_agnostic command, because the shipped specifications set model_name: sparse4d. The prepare_frames helper is a Python module that you run with python -m; it is not a subtask of the sparse4d command.

TAO Deploy supports Sparse4D engines with a fixed batch size of one, a fixed camera count and image size, and a fixed model layout. For the full list of limits, refer to Limitations.

Important

The Sparse4D TAO Deploy specification is deployment-only. It defines only the keys that this page describes, and a specification with other top-level keys, such as a train block, fails against the schema. Do not pass a training specification to sparse4d gen_trt_engine or sparse4d inference.

Prerequisites#

Before you build an engine, set up TensorRT and PyCUDA, supply the plugin library that Sparse4D needs, and export your model to ONNX.

TensorRT and PyCUDA#

Both actions need TensorRT 10.x and PyCUDA. The Sparse4D runtime raises the error Sparse4D currently supports TensorRT 10.x only when the major version of TensorRT is not 10.

Plugin Library#

The Sparse4D ONNX file contains six nv::MSDA operators, so TAO Deploy needs a TensorRT plugin library that implements them. You supply this library; TAO Deploy does not bundle a plugin binary. You identify the library with the plugin.path and plugin.sha256 keys. Both gen_trt_engine and inference enforce the following conditions before they use the library:

The value of plugin.path is an absolute path to an existing file.
The value of plugin.sha256 has 64 characters and equals the Secure Hash Algorithm 256-bit (SHA-256) hash of that file, and TAO Deploy compares the two values without regard to case.

After the checks pass, TAO Deploy initializes the TensorRT plugin registry and then loads the library into the process.

If the path or hash condition fails in gen_trt_engine, the action stops with the error plugin.path must be an absolute path to a trusted MSDA shared library or plugin.sha256 must match the selected MSDA library. The shipped specifications set plugin.sha256 to ???, a placeholder that you must replace with the SHA-256 hash of your library. If you leave ??? in place, the hash check fails and gen_trt_engine stops.

Before every engine build, regardless of data_type, gen_trt_engine also runs conformance probes against the library that you loaded. TAO Deploy builds two small TensorRT test engines, one in 32-bit floating point (FP32) and one in 16-bit floating point (FP16). Each test engine holds a single nv::MSDA operator and receives 900 queries. Queries 0 to 15 are probes: query 0 uses a valid location, and queries 1 to 15 use invalid locations. Every other query uses a valid location. The invalid locations fall into three categories:

Boundary locations, where a coordinate equals 0 or 1
Out-of-range locations, where a coordinate is below 0 or above 1
Non-finite locations, where a coordinate is not a number (NaN) or is infinite

The library conforms only if the test engine output equals the expected result exactly, with the valid queries contributing and queries 1 to 15 contributing nothing. In other words, the plugin must sample a location only when both of its normalized coordinates lie strictly between 0 and 1. If a probe fails, the build stops with an error that names the failing precision.

The ONNX export replaces every sampling location that has a coordinate outside the open range from 0 to 1, or that is not finite, with zero coordinates. For this reason, the plugin must skip locations at exactly zero as well.

The probes run only during gen_trt_engine. The engine sidecar, a JSON file that gen_trt_engine writes beside the engine, records the results. The inference action refuses an engine whose sidecar does not record that both precisions passed. For the sidecar contents and checks, refer to Engine Sidecar Check.

ONNX Model#

The gen_trt_engine action builds from the ONNX file that the Sparse4D export action writes. For the export parameters, refer to Sparse4D export. The export keeps the number of cameras dynamic, so model.num_cameras fixes it when you build the engine. The export takes the image size from model.input_shape of the training specification, as the width followed by the height. Set model.input_width and model.input_height in the deployment specification to the same values.

Before it builds, TAO Deploy checks the ONNX file and stops with an error if the file does not meet all of the following conditions:

The input names are img, projection_mat, image_wh, input_cached_feature, input_cached_anchor, prev_exists, and interval_mask, with or without a timestamp input, which the export can prune.
The output names are exactly the 12 outputs of the six-stage export: classification1, classification2, prediction1 to prediction6, quality1, quality2, output_cached_feature, and output_cached_anchor.
The file contains exactly six nv::MSDA operators.

Specification Structure#

Both actions read one deployment specification with the top-level keys that the following sections describe.

Experiment Configuration#

The following table lists the top-level keys of the specification.

Field	value_type	description	default_value	automl_enabled
`model_name`	string	Backend name that `model_agnostic` reads from the specification to select Sparse4D; the shipped specifications set it to `sparse4d`.	sparse4d
`results_dir`	string	Parent directory for the output of an action; TAO Deploy writes to its `gen_trt_engine` or `trt_inference` subdirectory unless the action block sets its own `results_dir`; set at least one of the two.
`model`	collection	Model dimensions that both actions share. Refer to Model Configuration.		FALSE
`plugin`	collection	Plugin library that both actions load. Refer to Plugin Configuration.		FALSE
`gen_trt_engine`	collection	Settings for the `gen_trt_engine` action. Refer to TensorRT Engine Generation Configuration.		FALSE
`inference`	collection	Settings for the `inference` action. Refer to Inference Configuration.		FALSE

For a string field, an empty default_value cell in the tables on this page means that the default is an empty string. You must set the following fields yourself, and the shipped specifications show each one: plugin.path, plugin.sha256, gen_trt_engine.onnx_file, gen_trt_engine.save_engine, inference.trt_engine, and inference.manifest.

To override a key from the command line, append it as a Hydra key=value argument after -e <specification>, for example inference.save_raw=true.

To start from the default values of this schema, use the default_specs subtask. It needs no specification file. It writes experiment.yaml to the results_dir that you pass as an override, and it replaces an existing experiment.yaml in that directory.

Model Configuration#

The model block describes the exported model and the prepared frames. Its values must agree with both, and inference requires the same values that the engine build used. The defaults for num_cameras, input_height, input_width, and num_classes are schema defaults, not properties of your model, so set them to the values of your exported model.

Field	value_type	description	default_value	valid_min	valid_options
`num_cameras`	int	Number of cameras in each frame; the engine profile fixes this value, so use the same value for `gen_trt_engine` and `inference`.	6	1
`input_height`	int	Height of each camera image in pixels; the `image_wh` array of every prepared frame must equal `[input_width, input_height]`.	256	1
`input_width`	int	Width of each camera image in pixels; together with `input_height`, it sets the shape of the `img` input.	448	1
`num_queries`	int	Number of object queries in the exported model; `gen_trt_engine` accepts only the default value.	900	1
`num_temp_instances`	int	Number of queries that TAO Deploy caches from one frame to the next; the value cannot exceed `num_queries`, and `gen_trt_engine` accepts only the default value.	600	1
`embed_dims`	int	Channel width of each query feature; `gen_trt_engine` accepts only the default value.	256	1
`anchor_dims`	ordered_int	Number of components in each anchor box, including the 3D velocity; TAO Deploy supports only 11.	11		11
`num_classes`	int	Number of classes in the `classification1` and `classification2` outputs; `inference` rejects outputs of any other width.	7	1

Plugin Configuration#

The plugin block identifies the plugin library. Both actions need it. For the conditions that TAO Deploy enforces, refer to Prerequisites.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`path`	string	Absolute path to a trusted plugin library.
`sha256`	string	Required SHA-256 hash of the file at `plugin.path`.

gen_trt_engine#

The gen_trt_engine action validates the ONNX file, loads the plugin, runs the plugin conformance probes, and builds a TensorRT engine with a fixed batch-size-one profile for all seven or eight inputs. Run the action like this:

sparse4d gen_trt_engine -e /specs/gen_trt_engine.yaml

The shipped specification looks like this, with example paths for the model, the plugin, and the engine:

model_name: sparse4d
results_dir: /results/sparse4d
model:
  num_cameras: 6
  input_height: 256
  input_width: 448
plugin:
  path: /plugins/libmsda.so
  sha256: ???
gen_trt_engine:
  onnx_file: /models/sparse4d.onnx
  save_engine: /results/sparse4d/sparse4d.engine
  data_type: fp32
  workspace_size: 4.0

Replace the paths, the plugin hash, the camera count, and the image size with the values for your model. Use a new path for save_engine, and keep the model block identical between gen_trt_engine and inference.

TensorRT Engine Generation Configuration#

The following table lists the parameters of the gen_trt_engine block.

Field	value_type	description	default_value	valid_min	valid_options
`results_dir`	string	Directory that receives `status.json` and `experiment.yaml` for this action; if empty, TAO Deploy uses the `gen_trt_engine` subdirectory of the top-level `results_dir`; the engine goes to `save_engine`, not to this directory.
`gpu_id`	int	Index of the GPU that builds the engine; TAO Deploy exports it as `CUDA_VISIBLE_DEVICES`.	0	0
`onnx_file`	string	Path to the ONNX file that the Sparse4D `export` action writes; the build rejects a file whose input names, output names, or number of `nv::MSDA` operators differ from that export.
`save_engine`	string	Path of the new engine file; TAO Deploy also writes the sidecar file `<save_engine>.json`, and the build stops if either file already exists.
`data_type`	categorical	Precision of the engine; the build disables TensorFloat-32 (TF32) for both values, and `fp16` also sets the TensorRT FP16 flag; the build rejects any other value, including 8-bit integer (INT8).	fp32		fp32,fp16
`workspace_size`	float	Workspace memory pool limit for the build in gibibytes (GiB); the value must be finite and greater than zero.	4.0	0.1

Engine-Build Notes#

Keep the following behaviors in mind when you build a Sparse4D engine:

Profile: The engine has one optimization profile in which the minimum, optimal, and maximum shapes of each input are equal. The batch size is one, and the number of cameras, the image height, and the image width come from the model block. All inputs are float32 except interval_mask, which is Boolean. If the ONNX file has a fixed dimension that differs from the configured value, the build stops.
Precision: The default is fp32. The build clears the TF32 flag for both precisions, and fp16 also sets the TensorRT FP16 flag.
Results directory: TAO Deploy writes status.json and experiment.yaml to the results directory of the action, which is /results/sparse4d/gen_trt_engine with the shipped specification. TAO Deploy writes the engine to the save_engine path and the sidecar to <save_engine>.json.

After it writes the engine, TAO Deploy writes the sidecar file <save_engine>.json beside it. The sidecar records the following values:

The format version and the model name
The TensorRT version
The SHA-256 hash of the engine
The SHA-256 hash of the ONNX file
The SHA-256 hash of the plugin library
The plugin conformance results
The precision
The input shapes
The model block

Important

Inference requires the exact TensorRT version that built the engine, including the patch and build numbers, because TAO Deploy does not build these engines with TensorRT version compatibility. Regenerate the engine after you upgrade TensorRT or change the plugin library. Otherwise, inference refuses the old engine.

Preparing Frames#

The inference action does not read images. It reads a JSON manifest that lists the frames in order and, for each frame, a NumPy archive (NPZ) file that holds already-normalized images and the matching camera calibration. TAO Deploy does not decode, resize, or normalize images.

Manifest Format#

The manifest is a JSON object with a version number and a list of frame records. A minimal example looks like this:

{
  "version": 1,
  "frames": [
    {"scene": "scene_0001", "timestamp": 0.0, "tensors": "000000.npz"},
    {"scene": "scene_0001", "timestamp": 0.1, "tensors": "000001.npz"}
  ]
}

TAO Deploy reads the following keys and stops with an error if the manifest does not meet these rules:

version: The value must equal 1.
frames: The value must be a nonempty list of frame records, and TAO Deploy processes the records in the order that you list them.
scene: The value must be a nonempty string that identifies the scene of the frame.
timestamp: The value must be a finite number in seconds, and TAO Deploy does not accept Boolean values.
tensors: The value must be a nonempty string with the path to the NPZ file of the frame. TAO Deploy resolves a relative path against the directory of the manifest and uses an absolute path unchanged.

Frame Tensor Format#

TAO Deploy loads each NPZ file with pickle disabled. The file must contain the img, projection_mat, and image_wh arrays, and it can also contain T_global. TAO Deploy rejects a file with any other key. Each array must be finite and of type float32. In the following table, C is model.num_cameras, H is model.input_height, and W is model.input_width.

Key	Shape	Description
`img`	(1, C, 3, H, W)	Already-normalized images of the cameras.
`projection_mat`	(1, C, 4, 4)	Camera calibration that matches the images, including any augmentation.
`image_wh`	(1, C, 2)	Width and height of the network image, in that order. Every camera must have the value `[input_width, input_height]`.
`T_global`	(4, 4)	Optional rigid local-to-global transform of the frame. TAO Deploy uses the identity transform when the key is absent.

TAO Deploy stops with the error T_global must be a rigid local-to-global transform if the matrix is not a rigid transform. TAO Deploy uses the T_global transforms of two consecutive frames to project the cached anchors into the current frame.

Generating Frames from a Training Dataset#

The prepare_frames module converts the prediction data of a Sparse4D training specification into the manifest and NPZ files that inference reads. It reuses the validation transforms of the TAO PyTorch data module for image normalization, cropping, resizing, and camera calibration, instead of reimplementing them. Run it in the TAO PyTorch environment, not in the TAO Deploy container.

Warning

Use only trusted training specifications and datasets with this module, because the annotation reader of the training data module uses pickle. The generated NPZ files need no pickle at deployment time.

Run the module like this:

python -m nvidia_tao_deploy.cv.sparse4d.prepare_frames \
    --spec /specs/experiment_spec.yaml \
    --output /data/frames \
    --max-frames 10

The module accepts the following arguments:

--spec: The resolved Sparse4D training specification. This argument is required.
--output: A new output directory. This argument is required, and the module stops if the directory already exists.
--max-frames: The maximum number of frames to write. The value must be positive, and the default is 10.

The module imports nvidia_tao_pytorch, which in turn imports PyTorch and PyTorch Lightning, so you need an environment that has both nvidia_tao_pytorch and nvidia_tao_deploy installed. The gen_trt_engine and inference actions do not import nvidia_tao_pytorch.

The module runs the prediction dataloader of the specification with one frame per batch and no worker processes, and it writes the first --max-frames frames. For each frame, it writes a compressed NPZ file, names it with six digits, such as 000000.npz, and it adds a record with the scene name and timestamp of the frame to frames.json in the output directory. It writes T_global only when the batch carries it.

After it finishes, the module prints Prepared N frames, where N is the number of frames written. Set inference.manifest to the frames.json file.

Running Inference with a TensorRT Engine#

The inference action loads the engine, checks it against its sidecar file, and runs the frames of the manifest in order. Run the action like this:

sparse4d inference -e /specs/inference.yaml

The shipped specification looks like this, with example paths for the plugin, the engine, and the manifest:

model_name: sparse4d
results_dir: /results/sparse4d
model:
  num_cameras: 6
  input_height: 256
  input_width: 448
plugin:
  path: /plugins/libmsda.so
  sha256: ???
inference:
  trt_engine: /results/sparse4d/sparse4d.engine
  manifest: /data/frames.json
  save_raw: false

Keep the model and plugin blocks identical to the ones that you used to build the engine, and replace ??? with the SHA-256 hash of the plugin library. In inference, a wrong or missing plugin.sha256 fails the engine sidecar check first, before the plugin loads. Refer to Engine Sidecar Check.

Engine Sidecar Check#

The <trt_engine>.json sidecar file must sit beside the engine. Before TAO Deploy runs any frame, it compares the sidecar with the current run and refuses to run when a value differs. The following table lists the errors for the value comparisons and their causes:

Error	Cause
`Engine provenance mismatch; regenerate with this TensorRT and MSDA library`	The format version, the model name, the engine hash, the TensorRT version, or the recorded plugin conformance results differ from the expected values, or the plugin hash that the sidecar recorded differs from `plugin.sha256`.
`Inference model settings differ from the engine build`	The `model` block differs from the block that the build recorded.
`Engine profile in sidecar disagrees with model settings`	The input shapes in the sidecar differ from the shapes that the `model` block implies.
`Engine inputs disagree with its sidecar`	The input names of the engine differ from the input names in the sidecar.

The expected values are format version 1, the model name sparse4d, the engine hash that TAO Deploy recomputes, the exact TensorRT version of the current run, and a conformance result that shows that FP32 and FP16 passed. The sidecar also records onnx_sha256 and precision, but inference does not check these two keys.

Inference Configuration#

The following table lists the parameters of the inference block.

Field	value_type	description	default_value	valid_min	valid_max	valid_options	automl_enabled
`results_dir`	string	Directory that receives the output files; if empty, TAO Deploy uses the `trt_inference` subdirectory of the top-level `results_dir`; the action stops if `predictions.jsonl`, `summary.json`, or `raw` already exists there.
`num_gpus`	ordered_int	Number of GPUs; the launcher always runs inference on one GPU.	1			1
`gpu_ids`	list	Indices of the GPUs that the launcher exposes as `CUDA_VISIBLE_DEVICES`; if you list more than one, the launcher keeps only the highest index.	[0]				FALSE
`trt_engine`	string	Path to the engine that `gen_trt_engine` built; the sidecar file `<trt_engine>.json` must sit beside it.
`manifest`	string	Path to the JSON manifest of prepared NPZ frames.
`num_output`	int	Number of queries per frame that TAO Deploy decodes, taken in descending order of class probability; the value must be greater than zero and at most `num_queries`.	300	1
`score_threshold`	float	Minimum raw class probability for a detection; the comparison uses the probability before quality weighting, and the value must be from 0 to 1.	0.05	0.0	1.0
`tracking_threshold`	float	Minimum raw class probability at which a query that has no instance ID receives a new one; the value must be from 0 to 1.	0.2	0.0	1.0
`confidence_decay`	float	Multiplier for the confidence that a cached query had in the previous frame; the larger of the decayed value and the current confidence ranks the query for the next cache; the value must be from 0 to 1.	0.8	0.0	1.0
`max_time_interval`	float	Largest time gap in seconds between consecutive frames of one scene that keeps the cache; a larger gap resets the temporal state; the value must be finite and greater than zero.	2.0	0.0
`save_raw`	bool	Whether to write the raw engine outputs of every frame to the `raw` directory under `results_dir`.	False

Temporal State#

TAO Deploy keeps one temporal state for each inference run. The exported model fuses the queries of a frame with the cached queries of the previous frame, but TAO Deploy selects, projects, and tracks the cache on the host.

In the following cases, TAO Deploy resets the temporal state:

The frame is the first frame of the run.
The scene differs from the scene of the previous frame.
The timestamp is not greater than the previous timestamp.
The gap to the previous timestamp is greater than max_time_interval. A gap equal to max_time_interval keeps the cache.

After a reset, the engine receives an empty cache, and prev_exists is zero.

On a frame that reuses the cache, TAO Deploy moves each cached anchor center forward by its velocity multiplied by the elapsed time, and then it transforms the anchors into the coordinate frame of the current frame. After the engine runs, TAO Deploy keeps the num_temp_instances queries with the highest confidence for the next frame. The confidence of a cached query is the larger of two values: its current raw confidence, or its previous confidence multiplied by confidence_decay. TAO Deploy orders queries with equal confidence by ascending query index.

A query keeps its instance ID while it stays in the cache. A query without an ID receives the next free ID when its raw confidence reaches tracking_threshold.

A reset clears the cache but does not restart the ID counter, so instance IDs stay unique across the scenes of one run. The counter starts again at zero in each run.

Decoding#

For each frame, TAO Deploy takes the sigmoid of classification2 as the class probabilities, assigns each query its most probable class, and keeps the num_output queries with the highest probability. It multiplies the probability of each kept query by the sigmoid of the first channel of quality2 to obtain the score. It sorts the queries by score in descending order and drops every query whose raw class probability is below score_threshold. It builds each box from prediction6.

Outputs#

The inference action writes its output to the results_dir of the inference block. If you leave that key empty, TAO Deploy uses the trt_inference subdirectory of the top-level results_dir, which is /results/sparse4d/trt_inference with the shipped specification. The action stops if predictions.jsonl, summary.json, or raw already exists in that directory, so use a new directory for each run. The directory receives the following files:

predictions.jsonl: The file holds one JSON object per frame, in manifest order.
summary.json: The file holds the totals of the run, the TensorRT version, and the SHA-256 hashes of the engine, the plugin library, and the manifest.
status.json: The file holds the status events of the action, which TAO Deploy writes as JSON lines.
experiment.yaml: The file holds the resolved specification of the run.
raw/: The directory holds the raw engine outputs of each frame. TAO Deploy creates it only when save_raw is true.

Each line of predictions.jsonl has the following keys, in this order:

frame_index: the zero-based position of the frame in the manifest
scene: the scene string of the frame
timestamp: the timestamp of the frame, in seconds
temporal_reset: true when the frame ran with an empty cache, including the first frame
boxes_3d: a list of boxes, each with 10 values in the order [x, y, z, w, l, h, yaw, vx, vy, vz], which are the position, the dimensions, the yaw angle, and the velocity
scores_3d: the score of each box, which is the raw class probability multiplied by the quality score
labels_3d: the class index of each box
cls_scores: the raw class probability of each box
instance_ids: the instance ID of each box

The value -1 in instance_ids marks a box whose query has no ID.

The summary.json file has the following keys:

frames: the number of frames that the run processed
detections: the total number of boxes
temporal_resets: the number of frames with temporal_reset set to true
all_outputs_finite: always true
engine_sha256, plugin_sha256, and manifest_sha256: the SHA-256 hashes of the engine, the plugin library, and the manifest
tensorrt: the TensorRT version of the run

The all_outputs_finite value is always true, because TAO Deploy stops the run when an engine output is not finite. TAO Deploy writes summary.json after the last frame. If a frame fails, the run stops, predictions.jsonl keeps the frames that TAO Deploy already wrote, and no summary.json exists.

When save_raw is true, TAO Deploy writes one compressed NPZ file per frame to raw/ and names each file with six digits, such as raw/000000.npz. Each file holds the 12 named output arrays of the engine.

Limitations#

The TAO Deploy backend for Sparse4D has the following limitations:

The sparse4d command has no evaluate subtask. It provides only gen_trt_engine, inference, and default_specs.
The build rejects INT8 engines and accepts only fp32 and fp16.
Each engine has a fixed camera count and image size, and the build supports only a layout of 900 queries, 600 cached queries, 256 channels, and 11 anchor components.
For inference, the launcher uses one GPU. If you list several indices in inference.gpu_ids, the launcher keeps only the highest index and does not raise an error.
Every engine input has a batch dimension of one, and each run processes the frames of one manifest one after another with a single temporal state.
The inference action reads prepared NPZ frames only, and the prepare_frames module produces them.
The Sparse4D inference action of TAO Deploy writes only the files that Outputs lists. It does not write DeepStream or NVSchema output, and it does not compute Higher Order Tracking Accuracy (HOTA) metrics.