Inference Library Overview

This section documents how to build TensorRT engines, run inference with the C++ and Python APIs, and apply advanced runtime features for production deployment on NVIDIA GPUs. Use this overview to choose the right guide for your integration stage.

About the Inference Library

The Inference Library covers the builder and runtime workflow: network definition, engine serialization, execution contexts, and inference, plus specialized guides for quantization, dynamic shapes, custom plugins, control flow, DLA, transformers, and debugging tools.

For your first engine, start with Build Your First Engine and the Quick Start Guide. For installation and platform support, refer to Installation Guide Overview and Support Matrix. For runtime object lifetimes and threading, refer to How TensorRT Works.

Integration Path Overview

1. Understand capabilities: Build/runtime model, precision support, and links to deeper topics → TensorRT Capabilities

2. Plan engine compatibility: Version and hardware compatibility, compatibility checks → Engine Compatibility

3. Follow language-specific walkthroughs: Step-by-step ONNX import, engine build, and inference → C++ API Documentation or Python API Documentation

4. Explore samples: Build and run shipped samples with Sample Explorer → Sample Support Guide

5. Apply advanced configuration: Refitting, precision, formats, engine tools, weight streaming, and multi-device → Advanced Topics

6. Optimize accuracy and performance: Quantization, dynamic shapes, transformers, and tuning. Refer to the remaining topics in In this guide below.

In this guide

• TensorRT Capabilities: build and runtime model, supported precisions, and links to specialized topics

• Engine Compatibility: version and hardware compatibility, checks, and cross-platform engine deployment

• C++ API Documentation: C++ workflow for ONNX import, engine build, deserialization, and inference

• Python API Documentation: Python workflow for parsing ONNX, building engines, and executing inference

• Sample Support Guide: Sample Explorer, build and run instructions, and cross-compiling

• Advanced Topics: refitting, precision control, tensor formats, engine tools, weight streaming, and multi-device inference

• Working with Quantized Types: explicit quantization, Q/DQ networks, PTQ, and QAT workflows

• Accuracy Considerations: reduced-precision trade-offs, determinism, and numerical debugging

• Working with Dynamic Shapes: optimization profiles, shape tensors, and data-dependent outputs

• Extending TensorRT with Custom Layers: Plugin V3 authoring, registration, and advanced plugin patterns

• Working with Loops: ILoop networks for recurrent and iterative subgraphs

• Working with Conditionals: IIfConditional networks for data-dependent branching

• Working with DLA: Deep Learning Accelerator deployment, formats, and standalone mode

• TensorRT API Capture and Replay: record and replay engine-building API sequences for debugging

• Working with Transformers: fused attention, KV cache, MoE, and transformer-specific optimizations

For ONNX export paths and deployment workflows, refer to Quick Start Guide. For measure-then-optimize performance guidance, refer to Performance Best Practices.