TensorRT Ecosystem#

This section introduces the samples, complementary GPU features, software, and tools that work alongside TensorRT for optimized inference deployment.

Samples#

The Sample Support Guide illustrates many of the topics discussed in this section.

Complementary GPU Features#

Beyond engine compilation and runtime APIs, NVIDIA GPUs expose platform features for sharing or partitioning hardware among concurrent workloads. The topics in this section describe two complementary options for TensorRT deployments: Multi-instance GPU (MIG) for dedicated GPU slices and multi-device inference for spreading a single network across multiple GPUs. Refer to Multi-Device Inference for API setup and operator details.

Multi-Instance GPU#

Multi-instance GPU (MIG) is a feature of NVIDIA GPUs with the NVIDIA Ampere architecture or later architectures. It enables user-directed partitioning of a single GPU into multiple smaller GPUs.

The physical partitions provide dedicated compute and memory slices with quality of service. They also support independent execution of parallel workloads on fractions of the GPU.

For TensorRT applications with low GPU utilization, MIG can increase aggregate throughput by running independent workloads on separate slices. The latency effect depends on the GPU, MIG profile, and workload. Measure both latency and throughput with the intended partitioning scheme. Use MIG when strict hardware isolation is required, such as in multi-tenant systems or when GPU memory must be partitioned.

Green Contexts#

A green context (introduced in CUDA 13.1) is a lightweight context associated, from its creation, with a set of specific GPU resources. Users can partition GPU resources, currently streaming multiprocessors (SMs) and work queues (WQs), during green context creation, so that GPU work targeting a green context can only use its provisioned SMs and work queues. Partitioning these resources bounds the compute and work-queue interference that one workload can impose on another. An application can have multiple green contexts.

Green contexts are a lighter-weight alternative to MIG to keep CUDA workloads from affecting each other’s inference compute usage. A primary advantage of green contexts over MIG is that green contexts do not require special hardware support, so they are available on more GPU architectures. Green contexts can also be configured at runtime by the application, where MIG must be configured prior to launching the application. However, unlike MIG, green contexts do not allow you to partition memory and do not provide strict isolation that might be required in multi-tenant environments.

Multi-Device Inference#

Multi-device inference is a feature for partitioning a single network across multiple GPUs, enabling multi-GPU inference. Each GPU runs its own instance of the TensorRT engine as a distinct rank and exchanges intermediate tensors with other ranks using distributed collective primitives.

The multi-device inference feature is exposed through the per-layer nbRanks attribute on IDistCollectiveLayer and IAttention, together with the IDistCollectiveLayer collectives (CollectiveOperation::kALL_REDUCE, kALL_GATHER, kBROADCAST, kREDUCE, kREDUCE_SCATTER, kALL_TO_ALL, kGATHER, kSCATTER). At runtime, the application initializes an NCCL communicator on each participating rank and attaches it to the engine.

For TensorRT applications whose models are too large to fit on a single GPU, or whose latency is dominated by compute that parallelizes well across devices, multi-device inference can lower per-rank memory pressure or single-query latency at the cost of inter-GPU communication. Validate the benefit on the deployed topology. Before deployment, review the multi-device items in the TensorRT Release Notes, including NCCL communicator limitations on Windows and after optimization-profile changes.

Complementary Software#

The following table lists NVIDIA tools that work alongside TensorRT at different stages of a deployment, from serving and data preprocessing through framework integration, model compression, and profiling, so you can reach for the entry that matches the stage you are working on.

Tool

Description

NVIDIA Triton Inference Server

Higher-level library providing optimized inference across CPUs and GPUs with model management, REST, and gRPC endpoints.

NVIDIA DALI

High-performance primitives for preprocessing image, audio, and video data. TensorRT inference can be integrated as a custom operator in a DALI pipeline. Refer to GitHub: DALI for a working example.

Torch-TensorRT

PyTorch-TensorRT compiler that converts PyTorch modules into TensorRT engines. Subgraphs are accelerated through TensorRT while Torch executes the rest natively. Refer to GitHub: Examples.

Model Optimizer

Unified library for quantization, pruning, and distillation. Compresses models for TensorRT-LLM or TensorRT deployment. Replaces the deprecated PyTorch and TensorFlow Quantization Toolkits. To quantize TensorFlow models, export to ONNX first.

NVIDIA Nsight Systems

Profiling tool integrated with TensorRT for performance analysis.

NVIDIA Nsight Deep Learning Designer

IDE for ONNX model editing, performance profiling, and TensorRT engine building.

NVIDIA DRIVE

A restricted subset of TensorRT is certified for NVIDIA DRIVE products. Some APIs are marked for DRIVE use only.

ONNX#

The primary means of importing a trained model from a framework is the ONNX interchange format. TensorRT ships with an ONNX parser library to assist in importing models. The table below summarizes common model entry paths and follow-on tooling.

Entry path

When to use

Next step

ONNX export + ONNX parser

Most training frameworks (PyTorch, TensorFlow, JAX, and others)

Export to the latest supported opset, then parse with the TensorRT ONNX parser (refer to C++ or Python API walkthrough)

PyTorch ONNX export

PyTorch-trained models

PyTorch ONNX export

TensorFlow → ONNX

TensorFlow or Keras models

tf2onnx

Post-export cleanup

Parser errors, unsupported subgraphs, or constant-folding opportunities

Run Polygraphy constant folding; edit with ONNX-GraphSurgeon when needed

TensorRT does not guarantee that every operator and attribute from a single minimum opset is supported. Verify the model’s operators and attributes against the ONNX-TensorRT operator support matrix. The ONNX Model Opset Version Converter can assist in resolving incompatibilities.

The GitHub version can support later opsets than the version shipped with TensorRT. Refer to the ONNX-TensorRT operator support matrix for the latest information on the supported opset and operators. For TensorRT deployment, we recommend exporting to the latest available ONNX opset. Refer to the ONNX Opset Guide.

After exporting a model to ONNX, run constant folding using Polygraphy as a good first step. Constant folding can resolve parser errors caused by foldable constant and shape subgraphs and can simplify the graph before further debugging.

Code Analysis Tools#

For guidance using the Valgrind and Clang sanitizer tools with TensorRT, refer to the Troubleshooting section.