Grounding DINO with TAO Deploy

To generate an optimized TensorRT engine, a Grounding DINO .onnx file, which is first generated using tao model grounding_dino export, is taken as an input to tao deploy grounding_dino gen_trt_engine. For more information about training a Grounding DINO model, refer to the Grounding DINO training documentation.

Converting ONNX File into TensorRT Engine

To convert the .onnx file, you can reuse the spec file from the tao model grounding_dino export command.

gen_trt_engine

The gen_trt_engine parameter defines TensorRT engine generation.

FTMS ClientTAO Launcher

Use the following command to get an experiment spec file for ReIdentificationNet:

SPECS=$(tao-client grounding_dino get-spec --action train --job_type experiment --id $EXPERIMENT_ID)

Field	value_type	Description	default_value	valid_min	valid_max	valid_options	automl_enabled
results_dir	string	Path to where all the assets generated from a task are stored.					FALSE
gpu_id	int	The index of the GPU to build the TensorRT engine.	0				FALSE
onnx_file	string	Path to the ONNX model file.	???				FALSE
trt_engine	string	Path where the generated TensorRT engine from gen_trt_engine is stored. This only works with tao-deploy.					FALSE
input_channel	int	Number of channels in the input tensor.	3	3			FALSE
input_width	int	Width of the input image tensor.	960	32			FALSE
input_height	int	Height of the input image tensor.	544	32			FALSE
opset_version	int	Operator set version of the ONNX model used to generate the TensorRT engine.	17	1			FALSE
batch_size	int	The batch size of the input tensor for the engine. A value of -1 implies dynamic tensor shapes.	-1	-1			FALSE
verbose	bool	Flag to enable verbose TensorRT logging.	False				FALSE
tensorrt	collection	Hyper parameters to configure the TensorRT Engine builder.					FALSE

tensorrt

The tensorrt parameter defines the TensorRT engine generation.

Field	value_type	Description	default_value	valid_min	valid_max	valid_options	automl_enabled
data_type	string	The precision to be set for building the TensorRT engine.	FP32			FP32,FP16	FALSE
workspace_size	int	The size (in MB) of the workspace TensorRT has to run it’s optimization tactics and generate the TensorRT engine.	1024				FALSE
min_batch_size	int	The minimum batch size in the optimization profile for the input tensor of the TensorRT engine.	1				FALSE
opt_batch_size	int	The optimum batch size in the optimization profile for the input tensor of the TensorRT engine.	1				FALSE
max_batch_size	int	The maximum batch size in the optimization profile for the input tensor of the TensorRT engine.	1				FALSE

Use the following command to run Grounding DINO engine generation:

FTMS ClientTAO Launcher

GTE_JOB_ID=$(tao-client grounding_dino experiment-run-action --action gen_trt_engine --id $EXPERIMENT_ID --parent_job_id $EXPORT_JOB_ID --specs "$SPECS")

See also

The Export job ID is the job ID of the tao-client grounding_dino experiment-run-action --action export command.

Running Evaluation through a TensorRT Engine

You can reuse the TAO evaluation spec file for evaluation through a TensorRT engine. The following is a sample spec file:

evaluate:
  trt_engine: /path/to/engine/file
  conf_threshold: 0.0
  input_width: 960
  input_height: 544
dataset:
  test_data_sources:
    image_dir: /data/raw-data/val2017/
    json_file: /data/raw-data/annotations/instances_val2017.json
  max_labels: 80
  batch_size: 8

Use the following command to run Grounding DINO engine evaluation:

FTMS ClientTAO Launcher

EVAL_JOB_ID=$(tao-client grounding_dino experiment-run-action --action evaluate --id $EXPERIMENT_ID --parent_job_id $GTE_JOB_ID --specs "$SPECS")

Running Inference through a TensorRT Engine

You can reuse the TAO inference spec file for inference through a TensorRT engine. The following is a sample spec file:

inference:
  conf_threshold: 0.5
  input_width: 960
  input_height: 544
  trt_engine: /path/to/engine/file
  color_map:
    "black cat": green
    car: red
    person: blue
dataset:
  infer_data_sources:
    - image_dir: /path/to/coco/images/val2017/
      captions: ["black cat", "car", "person"]
  max_labels: 80
  batch_size: 8

Use the following command to run Grounding DINO engine inference:

FTMS ClientTAO Launcher

INFER_JOB_ID=$(tao-client grounding_dino experiment-run-action --action inference --id $EXPERIMENT_ID --parent_job_id $GTE_JOB_ID --specs "$SPECS")

The visualization is stored in $RESULTS_DIR/images_annotated, and the KITTI format predictions is stored under $RESULTS_DIR/labels.