Is this page helpful?

Inference Library Overview#

The Inference Library section documents how to build TensorRT-RTX engines, run inference with the C++ and Python APIs, and apply advanced runtime features for deployment on RTX GPUs. Use this overview to choose the right guide for your integration stage.

About the Inference Library#

TensorRT-RTX exposes the same core builder and runtime concepts as TensorRT: network definition, serialized engines, and execution contexts. TensorRT-RTX optimizes these for portable deployment with ahead-of-time (AOT) and just-in-time (JIT) compilation on end-user RTX hardware.

For installation and first success, start with Installation Guide Overview, Installing TensorRT-RTX, and the Quick Start Guide. After Quick Start succeeds, use Build Your First Engine for deployment planning. For runtime object lifetimes and threading, refer to How TensorRT-RTX Works. When migrating from TensorRT, begin with the Porting Guide for TensorRT Applications.

Integration Path Overview#

  1. Install and verify: Prerequisites → install → Quick Start Guide → Build Your First Engine for bundling and portability

  2. Learn the runtime workflow: Build, deserialize, and run a simple engine with the native API → Using the Native Runtime API

  3. Integrate from PyTorch: Accelerate torch.compile() workloads with the TensorRT-RTX backend → Using TensorRT-RTX via PyTorch

  4. Follow language-specific walkthroughs: Step-by-step C++ or Python build and inference flows → C++ API Documentation or Python API Documentation

  5. Apply advanced features: Quantization, dynamic shapes, CUDA graphs, runtime cache, and more → topics in What’s in This Section below

  6. Optimize and debug: Performance tuning, Simultaneous Compute and Graphics (SCG), CPU-only engines, porting, and capture/replay as needed

For ONNX export before any API integration, refer to ONNX Conversion Guide.

What’s in This Section#

The Inference Library section in the sidebar lists every guide. Use the clusters below when you are choosing a starting point rather than scanning the full list.

Cluster

Guides

API walkthroughs

Using the Native Runtime API; C++ API Documentation; Python API Documentation; Using TensorRT-RTX via PyTorch

Working with X

Working with Quantized Types; Working with Dynamic Shapes; Working with Runtime Cache; Working with RTX CUDA Graphs

Advanced

Advanced Topics (weight streaming, engine compatibility, refitting, timing-cache migration); Simultaneous Compute and Graphics; CPU-Only AOT and TensorRT-RTX Engines

Migration

Porting Guide for TensorRT Applications; TensorRT-RTX API Capture and Replay

For ONNX export and installation prerequisites, refer to ONNX Conversion Guide and Prerequisites. For performance tuning, refer to Best Practices.