Inference Library Overview#
The Inference Library section documents how to build TensorRT-RTX engines, run inference with the C++ and Python APIs, and apply advanced runtime features for deployment on RTX GPUs. Use this overview to choose the right guide for your integration stage.
About the Inference Library#
TensorRT-RTX exposes the same core builder and runtime concepts as TensorRT: network definition, serialized engines, execution contexts: optimized for portable deployment with ahead-of-time (AOT) and just-in-time (JIT) compilation on end-user RTX hardware.
For installation and first success, start with Installation Guide Overview, Installing TensorRT-RTX, and the Quick Start Guide. After Quick Start succeeds, use Build Your First Engine for deployment planning. For runtime object lifetimes and threading, refer to How TensorRT-RTX Works. When migrating from TensorRT, begin with the Porting Guide for TensorRT Applications.
Integration Path Overview#
Install and verify: Prerequisites → install → Quick Start Guide → Build Your First Engine for bundling and portability
Learn the runtime workflow: Build, deserialize, and run a simple engine with the native API → Using the Native Runtime API
Integrate from PyTorch: Accelerate
torch.compile()workloads with the TensorRT-RTX backend → Using TensorRT-RTX via PyTorchFollow language-specific walkthroughs: Step-by-step C++ or Python build and inference flows → C++ API Documentation or Python API Documentation
Apply advanced features: Quantization, dynamic shapes, CUDA graphs, runtime cache, and more → topics in What’s in This Section below
Optimize and debug: Performance tuning, SCG, CPU-only engines, porting, and capture/replay as needed
For ONNX export before any API integration, refer to ONNX Conversion Guide.
What’s in This Section#
This inference library is organized into the following guides:
Using the Native Runtime API
End-to-end tutorial for building an engine and running inference with the C++ or Python API.
C++ API Documentation
Detailed C++ workflow: network creation, ONNX import, engine build, deserialization, and inference.
Python API Documentation
Python equivalents for parsing ONNX, building engines, and executing inference.
Using TensorRT-RTX via PyTorch
Compile PyTorch models with
torch.compile()and the TensorRT-RTX backend through Torch-TensorRT-RTX.
Advanced Topics
Weight streaming, engine compatibility, refitting, and deprecated timing-cache migration.
Work With Quantized Types
Guidance for explicit quantization, strongly typed networks, and quantized tensor types in TensorRT-RTX.
Working with Dynamic Shapes
Deferred dimension specification, optimization profiles, and runtime shape selection.
Working with Runtime Cache
Persist compiled GPU kernels on disk to reduce JIT startup overhead.
Working with RTX CUDA Graphs
CUDA Graph capture and replay for lower launch overhead, including dynamic shapes.
Simultaneous Compute and Graphics
Run inference alongside graphics workloads (for example, in-game AI features).
CPU-Only AOT and TensorRT-RTX Engines
Build portable engines without a GPU and target specific compute capabilities.
Porting Guide for TensorRT Applications
Differences between TensorRT and TensorRT-RTX, engine strategies, and API migration notes.
TensorRT-RTX API Capture and Replay
Record and replay engine-building API sequences for debugging (Linux).
For ONNX export and installation prerequisites, refer to ONNX Conversion Guide and Prerequisites. For performance tuning, see Best Practices.