TensorRT Backend Guide
The TensorRT backend provides highly optimized inference using NVIDIA’s TensorRT engine. It offers the best performance for production deployments on NVIDIA GPUs and seamlessly integrates TensorRT Model Optimizer for advanced quantization workflows.
Existing ONNX models
TensorRTBackend accepts an existing ONNX file through OnnxModule and builds an engine without PyTorch export. Export settings such as use_dynamo and opset_version do not apply to this path.
Only TensorRTBackend and ONNXRuntimeBackend support tuning OnnxModule. Configure an explicit backend list, as shown in ONNX Model Tuning.
Overview
The TensorRT backend:
- High Performance: Maximum inference speed on NVIDIA GPUs
- Dynamic Shapes: Supports optimization profiles for variable input sizes
- Quantization: INT8, FP8, INT4, FP16/BF16 autocast, and mixed precision support
- CUDA Graphs: Cached per static profile by default, with normal execution for dynamic ranges or capture failure
- Model Optimizer Integration: Advanced quantization via TensorRT Model Optimizer
- Flexible Export: Supports both Dynamo and script-based ONNX export
Quick Start
Basic Usage
With FP16 Precision
With CUDA Graphs
Export a Tuned Artifact
After tuning, retrieve the same DeploymentArtifact contract used by the ONNX backend:
Each optimization profile is a dictionary keyed by executable input name, in engine input order.
Each input contains min_shape, opt_shape, and max_shape tuples. These exact ranges are
preserved alongside the overall input bounds; multiple profiles can leave gaps within those bounds.
Runtime options preserve use_cuda_graphs, max_cuda_graphs, and cuda_graph_cache_policy as plain values
for deployment adapters to interpret.
The exported file is the TensorRT plan. The profile metadata is embedded in the deployment record;
AITune’s profile sidecar remains part of its checkpoint, and is not needed to execute the exported plan.
model.artifact() constructs and validates the deployment record when called. The backend retains
the finalized tensor names and profile metadata after deactivation, so generation does not require
reloading the engine. The method also works after restoring and deploying a checkpoint.
Output bounds come from the shapes recorded while tuning. model.artifact() therefore validates each
custom TensorRT input profile against the recorded input bounds. It rejects a profile that extends an
input beyond those bounds because sound output bounds cannot be inferred in general. Include the profile
boundary shapes in the tuning samples when the engine must also be exported as a deployment artifact.
Configuration Options
TensorRTBackendConfig
use_dynamo
Use torch.dynamo for ONNX export (recommended).
When to use:
True(default): Better compatibility with modern PyTorch modelsFalse: Legacy models or when Dynamo export fails
workspace_size
Maximum memory workspace for TensorRT engine building.
Guidelines:
- Default: TensorRT chooses automatically
- Larger workspace → More optimization opportunities → Longer build time
- Recommended: 1-4GB for most models
opset_version
ONNX opset version for export.
Guidelines:
- Default: Latest stable opset
- Specify only if you need a particular opset for compatibility
optimization_level
TensorRT builder optimization level (0-5).
Levels:
0: No optimization3: Default (balanced)5: Maximum optimization (longer build time)
compatibility_level
Hardware compatibility level for the engine.
Options:
None: Optimized for current GPU- Specific level: Portable across compatible GPUs
timing_cache
Path to timing cache for faster subsequent builds.
Benefits:
- Faster engine rebuilds
- Reuse timing information across builds
- Especially useful during development
profiles
Optimization profiles for dynamic shapes.
See Optimization Profiles section for details.
device
Device for TensorRT engine.
quantization_config
For OnnxModule, use ONNXAutoCastConfig or ONNXQuantizationConfig. TorchQuantizationConfig requires a PyTorch module and raises an error for OnnxModule.
TensorRT backend supports multiple quantization methods through TensorRT Model Optimizer integration. Use ONNXAutoCastConfig for FP16/BF16 mixed precision, ONNXQuantizationConfig for ONNX INT8/FP8/INT4 quantization, and TorchQuantizationConfig for ModelOpt PyTorch quantization presets.
For current ModelOpt preset names and version-specific support, use the Model Optimizer documentation for the installed version instead of copying preset lists into AITune docs.
enable_tf32
Enable TF32 tensor cores on Ampere+ GPUs.
Benefits:
- Faster FP32 operations on Ampere and newer GPUs
- No accuracy loss for most models
- Recommended to keep enabled
use_cuda_graphs
CUDA graphs are enabled by default for engines with fixed input shapes and for optimization profiles
where min == opt == max for every dimension of every input. Profiles containing a dynamic range use
ordinary TensorRT execution, even when a request happens to match their optimum shape. Inputs used as
shape tensors also use ordinary execution because fixed dimensions do not guarantee fixed shape values.
Each eligible profile is captured lazily when admitted to the cache. Its graph, execution context, input buffers,
and output allocator are cached together while sharing the engine. Switching back to a previously used
static profile replays its cached graph without recapture while it remains in the cache. The cache holds
up to max_cuda_graphs entries (default: 8). The default aged LFU policy can execute an uncached profile
normally to preserve frequently used graphs and avoid repeated capture. The optional LRU policy always
admits uncached profiles, releasing the least recently used graph when full. All TensorRT profiles remain available.
Deactivation releases the cache and usage history; activation starts with an empty cache.
If capture fails, the backend logs a warning, releases the graph cache after execution finishes, and runs the current and subsequent calls without CUDA graphs. It does not retry capture on shape changes after a failure. Normal TensorRT execution errors still propagate. A newly loaded backend attempts capture again.
Set use_cuda_graphs=False to disable capture explicitly:
Benefits:
- Reduced CPU overhead
- Better performance for small models
- Reuse of cached graphs when switching between static profiles
Limitations:
- Requests that capture a graph are slower, including recapture after eviction
- Each cached static profile retains its own context and buffers, increasing device memory use
- Not beneficial for very large models
max_cuda_graphs
Maximum resident CUDA graphs per backend, as a positive integer (default: 8). This limits cached graphs,
not the number of TensorRT optimization profiles or their total memory usage. It is a count limit, not a
device-memory budget.
With LRU, cycling through nine static profiles in an eight-entry cache causes eviction and recapture on every request after warmup. Default LFU admission avoids admitting profiles on equal frequency, allowing uncached requests to execute normally instead of repeatedly replacing graphs.
cuda_graph_cache_policy
Choose "lfu" (default) or "lru". LRU always admits an uncached eligible profile, evicting the least
recently used graph when full. LFU uses recent request frequency to protect hot profiles:
- Count every eligible profile request, including requests executed without a cached graph.
- Halve all counts using integer division every 1,024 eligible requests, before counting that request. Aging follows request traffic, not elapsed time, so historical popularity fades as new requests arrive.
- Fill free cache slots immediately. When full, choose the least frequently used resident, breaking frequency ties by least recent use.
- Admit an incoming profile only if its count is strictly higher than the victim’s. Otherwise, execute normally on the base context without capturing or evicting a graph. Rejected requests still contribute to future admission. This does not disable CUDA graphs or indicate a capture failure.
Set cuda_graph_cache_policy="lru" to always admit the most recently requested profiles.
Saved configurations with an explicit policy retain that choice; configurations without the field use LFU.
Frequency history is bounded by the engine’s eligible profiles, independently of the graph cache size. LFU can protect recurring hot shapes from occasional requests for other shapes, while LRU responds immediately to changes in the working set. Neither policy accounts for graph memory size or capture cost. The CUDA graph CI benchmark compares normal execution, LRU, and LFU using identical request sequences, reporting latency, capture counts, and requests served by graphs. LFU is the default to avoid the repeated capture overhead observed when the working set exceeds cache capacity. It is not faster for every workload.
Optimization Profiles
Optimization profiles define the range of input shapes TensorRT will optimize for. They are essential for models with dynamic input sizes.
Profile Modes
SINGLE (Default)
Automatically generates a single profile from recorded samples:
- Min shape: Minimum observed across all samples
- Opt shape: Maximum observed across all samples
- Max shape: Maximum observed across all samples
SAMPLES_USED
Generates one exact profile per unique recorded input shape. For dynamic inputs, a wide fallback follows the exact profiles and covers the observed range. If find-max-batch-size is enabled, it also covers the discovered batch bound.
When the module defines explicit dynamic shapes, those definitions take precedence and produce one graph-derived profile instead of sample profiles.
Important: Increase max_num_samples_stored:
Use case: When you have distinct input shape categories that need separate optimization.
Custom Profiles
Define exact optimization profiles:
These profiles assume the module defines forward(input), so input is the path of its top-level tensor parameter:
Finding Input Paths
Input tensor paths are shown in tuning logs. Top-level paths are forward parameter names; nested paths include all dictionary keys, sequence indices, or attributes:
Use the string shown in the Path column as the profile key. For example, the top-level path input uses "input",
while a nested dictionary path uses inputs["tokens"].
Best Practices for Profiles
- Min < Opt < Max: Ensure min ≤ opt ≤ max for all dimensions
- Opt = Typical: Set opt to your most common input size
- Range Coverage: Ensure your runtime inputs fall within [min, max]
- Multiple Profiles: Use for distinct size categories, not slight variations
- Test Runtime Shapes: Verify your production shapes are covered
Troubleshooting
Issue: ONNX export fails
Solution: Try disabling Dynamo export:
Issue: Engine build fails due to memory
Solution: Reduce workspace size:
Issue: Runtime shape not supported
Error: Input shape X exceeds max profile shape Y
Solution: Update profiles to cover your runtime shapes:
Issue: Slow first inference
Cause: This is expected when using CUDA graphs (graph capture overhead).
Solution: Warmup with a few inference calls before measuring performance.
Issue: INT8 accuracy drop
Solution: Try different quantization algorithms:
Best Practices
- Use FP16: Use
ONNXAutoCastConfig(precision="fp16")for FP16 mixed precision without a full quantization pass - Enable TF32: Keep
enable_tf32=Trueon Ampere+ GPUs - Profile Carefully: Ensure optimization profiles cover all runtime shapes
- Timing Cache: Use timing cache during development for faster iteration
- CUDA Graphs: Enable for latency-sensitive small models
- Workspace Size: Start with 1-2 GB and, increase if the build fails
- Quantization: Validate accuracy with a representative test set
Next Steps
- Learn about Torch-TensorRT JIT Backend
- Learn about Torch-TensorRT AOT Backend
- Explore Tune Strategies
- Review Deployment Guide