Monocular Depth with TAO Deploy#

To generate an optimized NVIDIA® TensorRT™ engine for NvDepthAnythingV2, the gen_trt_engine action takes an ONNX file previously produced by the NvDepthAnythingV2 export action. For more information about training a NvDepthAnythingV2 model, refer to the section Monocular Depth Estimation.

Converting the NvDepthAnythingV2 .onnx file into TensorRT Engine#

You can reuse the spec from the NvDepthAnythingV2 export action as a starting point.

gen_trt_engine#

The gen_trt_engine parameter defines TensorRT engine generation.

gen_trt_engine:
  onnx_file: /path/to/onnx_file
  trt_engine: /path/to/trt_engine
  tensorrt:
    data_type: fp16
    workspace_size: 1024
    min_batch_size: 1
    opt_batch_size: 4
    max_batch_size: 8

Field

value_type

Description

default_value

valid_min

valid_max

valid_options

automl_enabled

results_dir

string

Path to where all the assets generated from a task are stored.

FALSE

gpu_id

int

Index of the GPU to build the TensorRT engine.

0

FALSE

onnx_file

string

Path to the ONNX model file.

???

FALSE

trt_engine

string

Path where the generated TensorRT engine from gen_trt_engine is stored. This only works with tao-deploy.

FALSE

input_channel

int

Number of channels in the input tensor.

3

3

FALSE

opset_version

int

Operator set version of the ONNX model used to generate the TensorRT engine.

17

1

FALSE

batch_size

int

The batch size of the input tensor for the engine. A value of -1 implies dynamic tensor shapes.

-1

-1

FALSE

verbose

bool

Flag to enable verbose TensorRT logging.

False

FALSE

min_height

int

Minimum input height in the optimization profile for a dynamic-shape engine. Used only when the input ONNX has a dynamic height axis. Refer to the following dynamic-shape section.

14

FALSE

opt_height

int

Optimum input height in the optimization profile for a dynamic-shape engine.

14

FALSE

max_height

int

Maximum input height in the optimization profile for a dynamic-shape engine.

14

FALSE

min_width

int

Minimum input width in the optimization profile for a dynamic-shape engine.

14

FALSE

opt_width

int

Optimum input width in the optimization profile for a dynamic-shape engine.

14

FALSE

max_width

int

Maximum input width in the optimization profile for a dynamic-shape engine.

14

FALSE

tensorrt

collection

Hyperparameters to configure the TensorRT Engine builder.

FALSE

Important

The min/opt/max_height and min/opt/max_width fields apply only to the FastFoundationStereo model. They have no effect for the monocular NvDepthAnythingV2 model (or for FoundationStereo), which do not support dynamic height/width engines. For NvDepthAnythingV2, fix the spatial size at export time and use only the dynamic batch axis. See Building a Dynamic-Shape Engine below.

tensorrt#

The tensorrt parameter defines the TensorRT engine generation.

Field

value_type

Description

default_value

valid_min

valid_max

valid_options

automl_enabled

data_type

string

The precision to be set for building the TensorRT engine.

FP32

FP32,FP16,BF16

FALSE
workspace_size
int
The size in megabytes of the workspace TensorRT has
to run its optimization tactics and generate the
TensorRT engine.
1024









FALSE
min_batch_size
int
The minimum batch size in the optimization profile for
the input tensor of the TensorRT engine.
1






FALSE
opt_batch_size
int
The optimum batch size in the optimization profile for
the input tensor of the TensorRT engine.
1






FALSE
max_batch_size
int
The maximum batch size in the optimization profile for
the input tensor of the TensorRT engine.
1






FALSE

Building a Dynamic-Shape Engine#

A TensorRT engine built with dynamic shapes accepts a range of input sizes at runtime instead of a single fixed size. The dynamic dimensions are baked into the ONNX model at export time (as a -1 in the corresponding axis), and the gen_trt_engine action then attaches a TensorRT optimization profile that supplies the minimum, optimum, and maximum value for each dynamic axis. The optimum value is the size TensorRT tunes its kernels for; the minimum and maximum bound the range that the resulting engine will accept.

There are two independent dynamic axes:

  • Dynamic batch (supported for NvDepthAnythingV2). Set batch_size: -1 in the export spec to mark the batch axis dynamic in the ONNX, then set tensorrt.min_batch_size, tensorrt.opt_batch_size, and tensorrt.max_batch_size in the gen_trt_engine spec to bound the batch range. This is the configuration that the -1 batch_size note in the table above refers to.

  • Dynamic height/width. Set min_height / opt_height / max_height and min_width / opt_width / max_width on the gen_trt_engine parameter to supply the spatial optimization-profile bounds. These fields are consumed only when the input ONNX was exported with a dynamic height/width axis (export.dynamic_hw: true). Each bound must be at least 14, and the values must satisfy min ≤ opt ≤ max for both height and width or engine generation fails with a validation error.

    Note

    Dynamic height/width is supported only for the FastFoundationStereo model. For the monocular NvDepthAnythingV2 model (and for FoundationStereo), the DINOv2 backbone constant-folds the trace-time patch count into its positional-embedding shape arithmetic, so a dynamic-H/W ONNX would silently produce a wrong-shape positional embedding at runtime. Setting export.dynamic_hw: true for these models is therefore ignored with a warning, and the min/opt/max_height/width fields have no effect. For NvDepthAnythingV2, fix the spatial size at export time via export.input_height / export.input_width and use only the dynamic batch axis.

The following gen_trt_engine spec builds an FP16 NvDepthAnythingV2 engine with a dynamic batch axis that accepts batch sizes from 1 to 8, optimized for a batch of 4:

gen_trt_engine:
  onnx_file: /path/to/nv_depth_anything_v2_dynamic_batch.onnx
  trt_engine: /path/to/nv_depth_anything_v2.engine
  batch_size: -1
  tensorrt:
    data_type: fp16
    workspace_size: 1024
    min_batch_size: 1
    opt_batch_size: 4
    max_batch_size: 8

For reference, a dynamic height/width engine (FastFoundationStereo only) adds the spatial bounds alongside the batch bounds. The min/opt/max_height/width fields are siblings of tensorrt under gen_trt_engine:

gen_trt_engine:
  onnx_file: /path/to/dynamic_hw.onnx
  trt_engine: /path/to/dynamic_hw.engine
  batch_size: -1
  min_height: 320
  opt_height: 518
  max_height: 1024
  min_width: 320
  opt_width: 924
  max_width: 1024
  tensorrt:
    data_type: fp16
    workspace_size: 1024
    min_batch_size: 1
    opt_batch_size: 1
    max_batch_size: 4

Ask the agent to run the gen_trt_engine action against your spec. For example:

Build an FP16 TensorRT engine for NvDepthAnythingV2 from the exported
ONNX at ``s3://my-bucket/depth/model.onnx`` using ``trt-spec.yaml``.
Write the engine to ``s3://my-bucket/depth/model.engine``. Run on the local Docker daemon.

Running Evaluation through a TensorRT Engine#

You can reuse the TAO evaluation specification file for evaluation through a TensorRT engine. This is a sample specification file:

evaluate:
  trt_engine: /path/to/engine/file
  input_width: 924
  input_height: 518
dataset:
  test_dataset:
    data_sources:
      - dataset_name: RelativeMonoDataset
        data_file: /data/depth_net/annotation_test.txt
    workers: 4
    batch_size: 4

Ask the agent to run the evaluate action against the engine you built. For example:

Evaluate the NvDepthAnythingV2 TensorRT engine at
``s3://my-bucket/depth/model.engine`` against ``eval-spec.yaml``. Run on
local Docker.

Running Inference through a TensorRT Engine#

You can reuse the TAO inference specification file for inference through a TensorRT engine. This is a sample specification file:

inference:
  trt_engine: /path/to/engine/file
  input_width: 924
  input_height: 518
dataset:
  infer_dataset:
    data_sources:
      - dataset_name: RelativeMonoDataset
        data_file: /data/depth_net/annotation_test.txt
    workers: 4
    batch_size: 4

Ask the agent to run the inference action against the engine you built. For example:

Run NvDepthAnythingV2 inference with the TensorRT engine at
``s3://my-bucket/depth/model.engine`` using ``infer-spec.yaml``. Run on
your chosen backend.

Annotated visualizations are written to images_annotated under the configured results directory, and predictions are written to labels.