Is this page helpful?

NVIDIA TensorRT for RTX Documentation#

NVIDIA TensorRT for RTX builds on the proven performance of the NVIDIA TensorRT inference library and simplifies the deployment of AI models on NVIDIA RTX GPUs across desktops, laptops, and workstations. It introduces a Just-In-Time (JIT) optimizer in the runtime that compiles optimized inference engines directly on the end-user’s RTX-accelerated PC.

The two-phase compilation process (AOT + JIT) typically completes in under 30 seconds total, eliminating lengthy per-device pre-compilation steps and enabling rapid engine generation with improved application portability. TensorRT for RTX is a compact, under 200 MB, drop-in replacement for NVIDIA TensorRT targeting NVIDIA RTX GPUs from NVIDIA Turing (compute capability 7.5) through NVIDIA Blackwell (compute capability 12.x) generations.

In the following documentation, TensorRT for RTX is referred to as TensorRT-RTX.

Quick Start#

🆕 What’s New in NVIDIA TensorRT-RTX 1.6#

Release Highlights

  • Windows on ARM Support for NVIDIA RTX Spark: Deploy TensorRT-RTX inference applications to supported ARM-based Windows systems. Learn more about NVIDIA RTX Spark.

  • Linux SBSA Support for NVIDIA DGX Spark: Deploy to compatible ARM64 Linux environments with official support that replaces the experimental support introduced in TensorRT-RTX 1.5.

  • Runtime Cache Compatibility Tightened: The runtime environment must match the cached GPU SKU, TensorRT-RTX version, and Compute-in-Graphics (CiG) state, and the driver version must be greater than or equal to the cached version. Refer to Working with Runtime Cache.

  • CUDA 13.4 Support: Use NVIDIA CUDA Toolkit 13.4 or CUDA 12.9 Update 1. Separate TensorRT-RTX packages are available for each supported CUDA version. Refer to Support Matrix.

  • Zero-Copy I/O for Unified Memory: On NVIDIA RTX Spark and NVIDIA DGX Spark, use mapped input and output buffers to avoid explicit host-to-device and device-to-host copies. Refer to Zero-Copy I/O for Unified Memory.

  • Deferred Weights Loading: Deserialize engines and JIT-compile kernels without allocating engine weights in host or GPU memory. Refer to Deferred Weights Loading.

  • Getting Started Reorganized: Use the new Quick Start Guide and find Build Your First Engine alongside the release notes and support matrix.

  • Interactive Support Matrix: Find support information on a single Support Matrix page with interactive explorers for software and system requirements and for GPU architecture and precision support.

  • Architecture, Performance, and Advanced Topics Expanded: Explore runtime lifetimes, threading, memory, engine compatibility, benchmarking, optimization, refitting, timing-cache migration, and weight streaming in the restructured Architecture, Best Practices, and Advanced Topics content.

  • Runtime Tutorials Modernized: Follow updated C++ and Python walkthroughs that use the current tensor-address and enqueue APIs. Refer to Using the Native Runtime API.

  • Stability Fixes: Resolved a JIT initialization crash involving FP8 convolutions with certain dynamic input shapes and a Windows race condition affecting concurrent inference across multiple engines.

View 1.6 Release Notes

Previous Releases#

📋 Release 1.5 Highlights

  • DGX Spark / Linux SBSA (experimental): New experimental build for NVIDIA DGX Spark (NVIDIA GB10, compute capability 12.1) and ARM64 Linux SBSA platforms (Ubuntu 22.04 / 24.04). Refer to Support Matrix and Prerequisites for the supported hardware and operating systems.

  • CUDA 13.3 Support: Compatible with NVIDIA CUDA 13.3, with continued support for CUDA 12.9 Update 1. Refer to Support Matrix for the full compiler and runtime version list.

  • Qwen3.5 Support: Qwen3.5 dense models are supported through Windows ML with the TensorRT-RTX execution provider. Refer to Architecture Overview for LLM and Windows ML integration details.

  • Operator Support: Added support for the RoiAlign ONNX operator. Refer to Operators for the full ONNX operator catalog.

  • GPU Latency Optimizations: Faster GEMV kernels for dynamic input shapes, reduced CPU overhead between kernel launches, expanded kernel fusion coverage for dynamic shapes, and improved just-in-time kernel generation for additional convolution variants and runtime fusion patterns. Convolution in FP16 precision with batch size 1 has been accelerated with a new backend on Ampere and later. Convolution performance was also improved on the NVIDIA GB10 (sm_121) architecture. Refer to GPU Latency Optimizations for the per-optimization list.

  • Stability and Accuracy Fixes: Resolved YOLO ONNX model builds on Turing (sm_75), fixed FP16 dynamic-shape execution-context errors affecting models such as Stable Diffusion XL UNet and DaVinci Resolve SpeedWarp, fixed a dynamic-shape accuracy regression, enabled BF16 depthwise convolutions and deconvolutions, and enabled 3D deconvolutions with groups and padding. Refer to Fixed Issues for per-issue details.

View 1.5 Release Notes

📋 Release 1.4 Highlights

  • CUDA 13.2 Support: Compatible with NVIDIA CUDA 13.2 Toolkit

  • PyPI Availability: Install TensorRT-RTX Python bindings directly from PyPI with pip install tensorrt-rtx

  • API Capture and Replay: New debugging feature that records TensorRT-RTX API calls during engine building and replays them for issue reproduction without requiring the original application or model source code. Refer to API Capture and Replay documentation. (Linux only)

  • GPU Latency Optimizations: Improved performance with optimized 1D convolution kernels, optimized GEMV kernels, new Windows backend for batch size = 1 convolutions, improved JIT compilation heuristics, faster JIT compilation, and enhanced multi-head attention (MHA) performance

  • Parallel CUDA Graph Capture: Enabled running multiple inference contexts in parallel with CUDA graph capture using unique streams per context

  • Compute-in-Graphics (CiG) Improvements: Fixed performance issues and segmentation faults on Blackwell GPUs, improved MHA kernel shared memory handling

View 1.4 Release Notes

📦 Archived Releases

Earlier TensorRT-RTX 1.x releases with key highlights:

  • 1.3 Release Notes - Thread-Safe Multi-GPU Execution, LLM and Convolution Performance, Blackwell CUDA Graphics Mode, FP8 Performance, CUDA 12.9 / 13.1 Support

  • 1.2 Release Notes - CUDA Graphs Support, User Memory Allocation, CUDA 13.0 Support, Library Reorganization

  • 1.1 Release Notes - Engine Validity API, Faster Compilation

  • 1.0 Release Notes - Reduced Binary Size, Two-Phase Compilation, System Resource Adaptivity, Windows ML Support

Note

For detailed changelogs, refer to the TensorRT-RTX GitHub Releases.