Is this page helpful?

NVIDIA TensorRT Documentation#

NVIDIA® TensorRT™ is an SDK for optimizing deep learning inference on NVIDIA GPUs. It takes trained models from frameworks such as PyTorch and ONNX and compiles them into engines, which are optimized executable artifacts for a specific deployment configuration. TensorRT supports mixed precision (FP32/FP16/BF16/FP8/INT8/FP4/INT4), dynamic input shapes, and specialized optimizations for transformers and large language models (LLMs). Measure latency and throughput on your own model and hardware; refer to Best Practices. For definitions of TensorRT terms, refer to the Glossary.

Quick Start#

  • New to NVIDIA TensorRT? → Install first, then verify, then build your first engine:

    1. Installation Guide Overview and Installing TensorRT (Debian/RPM, tar/zip, or container for the ~10-minute CLI tutorial)

    2. Verify: trtexec --help (non-pip) or import tensorrt; print(tensorrt.__version__) (pip)

    3. Build Your First Engine (requires trtexec, which is not included in pip wheels)

  • Python API only (pip)? → Method 1: Python Package Index (pip) (pip install tensorrt). Pip installs bindings and libraries only, with no trtexec. For the first-engine CLI tutorial, use Debian/RPM, tar/zip, or container instead.

  • C++ or CLI workflows? → Choose Debian/RPM, tar/zip, or container on Installing TensorRT; run trtexec from the package bin directory

  • Ready for the full workflow menu? → After your first engine, use the Quick Start Guide for PyTorch and ONNX export paths, multiple runtimes, dynamic shapes, and quantization

  • Using an open model from Hugging Face? → TensorRT-Model-Connect provides a large collection of pre-implemented open models, taking a supported Hugging Face checkpoint to end-to-end TensorRT inference in two commands with no intermediate ONNX export. It is in public preview, so its APIs and scope may change.

  • Upgrading from 11.3 or earlier? → Refer to What Is New in 11.4.0 below

  • Upgrading from TensorRT 10.x? → Use the NVIDIA TensorRT Migration Guide to plan your API and builder changes

  • Need help with a specific task? → Jump to the Inference Library Overview for API walkthroughs, dynamic shapes, quantization, and more, or the Troubleshooting section

  • Need a TensorRT term defined? → Open the Glossary

  • Optimize inference performance → Best Practices

What is New in NVIDIA TensorRT 11.4.0#

Release Highlights

  • Windows on ARM: Support for Windows on ARM on NVIDIA RTX Spark laptops is newly introduced in this release, starting with TensorRT 11.4.0. Hardware support is limited to DLA-capable models at this time. Refer to DLA Supported Layers for Windows on ARM.

  • DLA support in TensorRT 11.4.0: Enterprise TensorRT 11.4.0 restores DLA on platforms including Windows on ARM, where DLA support is in beta and covers a restricted set of layers. Its supported-layer set and restrictions differ from Jetson and DriveOS: there is no GPU fallback, so every layer must be DLA-capable, and batch size is 1. In 11.4.0, DLA is not supported for Linux SBSA, NVIDIA JetPack, or NVIDIA DriveOS deployments, including NVIDIA DRIVE AGX Thor. Use TensorRT 10.7 if DLA is required on those platforms. DLA engines built with TensorRT 11.4.0 must be rebuilt once DLA support reaches general availability. Refer to DLA Supported Layers for Windows on ARM and Working with DLA.

  • Green Contexts: CUDA green contexts are a lighter-weight alternative to MIG for partitioning streaming multiprocessors (SMs) and work queues among concurrent workloads at runtime. They do not require special hardware support. They do not partition memory and do not provide MIG-style strict isolation. Refer to Green Contexts.

  • Reduced build-time memory usage with weight placeholders: Engine building with weight placeholders now uses less host and device memory. TensorRT 11.3.0 introduced support for null weights in the weight-stripping workflow, and TensorRT 11.4.0 further optimizes it. Refer to Reducing Build-Time Memory Usage with Weight Placeholders.

View 11.4.0 Release Notes

Previous Releases#

Release 11.3.0 Highlights

  • CUDA Toolkit 13.4 dependency upgrade: TensorRT 11.3.0 packages are built against CUDA® Toolkit 13.4; Debian, RPM, tar, and zip package filenames use cuda-13.4. TensorRT 11.3.1 DriveOS packages use CUDA Toolkit 13.4. Refer to the TensorRT Support Matrix for supported CUDA releases per platform and to Prerequisites for installer prerequisites.

  • NVIDIA Vera Rubin support added in TensorRT 11.3.0: NVIDIA Vera Rubin GPUs are supported by TensorRT 11.3.0 Enterprise (general-release). Linux x86, Linux SBSA, and Windows x64 support NVIDIA Vera Rubin GPUs with compute capability version 10.7. Refer to the TensorRT Support Matrix.

  • Refit enhancement in TensorRT 11.3.0: High-precision weights used in FP4 double quantization are now refittable. If you use BuilderFlag::kREFIT, refer to Known Issues in the TensorRT 11.3.0 Release Notes. Refer to Refitting an Engine.

  • Limitations in TensorRT 11.3.0: DLA is not supported in TensorRT 11.3.0 or in TensorRT 11.3.1 for DriveOS (last DLA release is 10.7). NVIDIA JetPack is not supported; Jetson deployments must remain on a TensorRT 10.x release supported by their JetPack version. Refer to the TensorRT 11.3.0 Release Notes.

View 11.3.0 Release Notes

Release 11.2.1 Highlights

  • Platform dependency upgrades: Updates internal build dependencies for TensorRT 11.2.1. TensorRT 11.2.1 packages are built against CUDA 13.3 update 1; Debian, RPM, tar, and zip package filenames continue to use cuda-13.3.

  • GridSample 3D support: Extends GridSample from 2D-only to 3D (rank-5 input) with FP32, FP16, and BF16

  • ONNX DFT operator support: Adds a cuFFT-based plugin for forward and inverse C2C, R2C, and C2R transforms

  • PluginV2 to PluginV3 migration sample: Adds a Python sample with PluginV2-to-PluginV3 method mappings

  • Improved CMake support: Tar and Zip packages include CMake configuration files under the cmake directory

View 11.2.1 Release Notes

Release 11.1.0 Highlights

  • CUDA 13.3 dependency upgrade: Updated CUDA Toolkit baseline across Linux x86-64, Windows x64, and SBSA platforms

  • Ubuntu 26.04 support: Adds Ubuntu 26.04 LTS to the supported Linux x86-64 and SBSA platform lists alongside the existing Ubuntu 22.04/24.04 packages

  • Python 3.14 bindings: Extends the Python wheel matrix to Python 3.14 on supported platforms

  • NVFP4 dual-GEMM fusion for SM121: Fuses the gate and up projection GEMMs in NVFP4 MoE/MLP blocks on NVIDIA DGX Spark (compute capability 12.1)

  • Global Performance Tuner: Automates trtexec build-route search to explore internal builder knobs, benchmark candidate engines, and optionally validate accuracy before selecting a route. Measure the benefit on your model and GPU. Refer to Global Performance Tuning.

View 11.1.0 Release Notes

Release 11.0.0 Highlights

  • Strongly typed networks are now the default: Weak-typing APIs (setPrecision, setDynamicRange, the per-precision BuilderFlag family) and implicit quantization (IInt8Calibrator) have been removed. Use the NVIDIA TensorRT Migration Guide to plan your upgrade

  • The IPluginV2 family is deprecated: IPluginV2 and IPluginV2Ext have been deprecated since TensorRT 8.5, and IPluginV2IOExt, IPluginV2DynamicExt, IPluginCreator, and IPluginV2Layer are deprecated as of TensorRT 10.0. These classes and INetworkDefinition::addPluginV2() are still declared in the 11.x headers, so existing V2 plugins continue to build and run with deprecation warnings. Port them to IPluginV3 with addPluginV3() and plan for removal in a future major release. Refer to the V2 → V3 walkthrough for a side-by-side API mapping

  • Multi-Device Inference is generally available: Preview flag retired, plus new AllToAll, Gather, and Scatter collective ops, automatic NCCL library fallback, and a new context-parallel attention sample. Refer to Multi-Device Inference

  • Ragged batching for attention: IAttention and IKVCacheUpdateLayer now support packed (kPACKED_NHD) layouts so variable-length sequences can be concatenated end-to-end without padding. Refer to Fused Attention

  • MoE inference performance: NVIDIA Blackwell architecture (SM10x/SM110) backend improvements for common MoE patterns; the previous “keep seqLen ≤ 16” guidance no longer applies. Benchmark your own workload. Refer to MoE (Mixture of Experts)

  • Rewritten Best Practices and Benchmarking guide: Reframed as a measure-then-optimize loop with side-by-side ONNX-TRT (trtexec) and Torch-TRT workflows in synchronized tabs covering quantization, dynamic shapes, CUDA graphs, profiling, and Nsight Systems timeline reading. Refer to Performance Benchmarking

  • Platform updates: RHEL 10 / Rocky Linux 10 RPM and tar packages, and a new TensorRT 10.x to 11.x migration path with dedicated DriveOS and Jetson/JetPack chapters

View 11.0.0 Release Notes

Archived Releases

Earlier TensorRT releases with key highlights:

Legacy Versions

Note

For complete version history and detailed changelogs, visit the Release Notes section or the TensorRT GitHub Releases.