Is this page helpful?

NVIDIA TensorRT for RTX Documentation#

NVIDIA® TensorRT™ for RTX builds on the proven performance of the NVIDIA TensorRT inference library and simplifies the deployment of AI models on NVIDIA RTX™ GPUs across desktops, laptops, and workstations. It introduces a Just-In-Time (JIT) optimizer in the runtime that compiles optimized inference engines directly on the end-user NVIDIA RTX-accelerated PC.

The two-phase compilation process (ahead-of-time (AOT) plus JIT) typically completes in under 30 seconds total, eliminating lengthy per-device pre-compilation steps and enabling rapid engine generation with improved application portability. TensorRT for RTX is a compact, under 200 MB, drop-in replacement for NVIDIA TensorRT targeting NVIDIA RTX GPUs from the NVIDIA Turing™ architecture (compute capability 7.5) through the NVIDIA Blackwell architecture (compute capability 12.x) generations.

In the following documentation, TensorRT for RTX is referred to as TensorRT-RTX.

Quick Start#

🆕 New to NVIDIA TensorRT-RTX? → Quick Start Guide verifies your install and walks through download, AOT build, and JIT inference in three commands
📖 Ready to ship an engine? → After Quick Start, use Build Your First Engine for deployment, bundling, portability, and runtime-cache guidance
📦 Install TensorRT-RTX → Start with Installation Guide Overview and Installing TensorRT-RTX
🐍 Python only (PyPI)? → Use the PyPI tab on Installing TensorRT-RTX (pip install tensorrt-rtx)
🛠️ C++ or CLI workflows? → Use the Windows SDK zip or Linux tarball on Installing TensorRT-RTX and add bin to PATH
⬆️ Upgrading from 1.6 or earlier? → Refer to What’s New in 1.7
🔧 Need help with a specific task? → Jump to the Inference Library Overview for API walkthroughs, dynamic shapes, CUDA® graphs, and more
🧪 Looking for samples? → Start with the TensorRT-RTX OSS samples, including the C++ API usage sample and the Python API usage sample. Walkthroughs also link these from C++ API Documentation, Python API Documentation, and Working with RTX CUDA Graphs. For end-to-end samples covering image generation, speech recognition, and computer vision, refer to DIN Deploy, which runs exported ONNX models through ONNX Runtime with the TensorRT-RTX execution provider.
⚡ Optimize inference performance → Best Practices
🔄 Port from TensorRT → Porting Guide for TensorRT Applications

🆕 What’s New in NVIDIA TensorRT-RTX 1.7#

Reduced Build-Time Memory Usage with Weight Placeholders: Build weight-stripped engines from null weight placeholders that declare only the data type and element count of the weights they stand in for. The builder skips loading the weight payload and makes fewer internal build-time allocations. Refer to Reducing Build-Time Memory Usage with Weight Placeholders.
Weights Offloading: Unload an inactive engine’s weight memory so that another engine can use the released GPU memory, then restore those weights later without rebuilding or refitting the engine. Restoration based on CUDA virtual memory maps application-provided GPU memory directly into the engine’s weight address range, so no extra copy is needed. Refer to Offloading and Restoring Engine Weights.
Improved Kernel Performance and Memory Efficiency: Kernel and fusion optimizations across a range of AI workloads. FP8 and INT8 weight-only-quantized convolutions produce smaller engines and lower peak GPU memory usage, and reduced CPU processing time makes engine refitting faster for some workloads.
Edge-LLM Integration: Use TensorRT-RTX as a supported backend for Edge-LLM on x86-64 Linux and Windows, including workloads that use speculative decoding.
Improved JIT Performance and Host-Memory Efficiency: Lower host-side latency when creating execution contexts and when serializing and deserializing the runtime cache. Execution contexts and runtime caches also use less host memory.

View 1.7 Release Notes

Previous Release#

Note

For detailed changelogs, refer to the TensorRT-RTX GitHub Releases.