Skip to navigation

TensorRT Optimization Profiles

View as MarkdownOpen in Claude

Introduction

TensorRT optimization profiles optimize the performance of a TensorRT engine. They are used to profile the performance of a model at different input shapes and batch sizes.

By default, a single profile is generated from the graph spec that supports the minimum and maximum shapes of the input tensors.

Using samples for profile generation

Set the number of samples to use for profile generation

You can set the number of samples to use for profile generation by setting the max_num_samples_stored in the aitune.torch.config module. By default, it is set to 1 as samples are stored for each backend, model module, and each batch size.

from aitune.torch.config import config as global_config
global_config.max_num_samples_stored = float("inf")
# or you can set it to a specific number of samples to use for profile generation
global_config.max_num_samples_stored = 100

Use the TensorRTProfileMode.SAMPLES_USED mode

TensorRTProfileMode.SAMPLES_USED creates an exact profile for each recorded shape. When input shapes vary, it also adds a wide fallback spanning their range, including the discovered maximum batch size when find-max-batch-size is enabled.

When the module defines explicit dynamic shapes, those definitions take precedence and produce one graph-derived profile instead of sample profiles.

from aitune.torch.backend import TensorRTBackend, TensorRTBackendConfig
from aitune.torch.backend.tensorrt import TensorRTProfile, TensorRTProfileMode
backend = TensorRTBackend(TensorRTBackendConfig(profiles=TensorRTProfileMode.SAMPLES_USED))

Use the correct samples and the right batch sizes during tuning

The fallback accepts unrecorded shapes within its bounds, while the exact profiles are preferred for recorded shapes. Inputs outside the fallback bounds cannot run.

NOTE: As samples for a single parameter have different shapes, we are wrapping them in a DynamicShapeDataset to handle different shapes.

import aitune.torch as ait
from aitune.torch.dataloader import DynamicShapeDataset
data1 = torch.randn((3, 224, 224), device=device).to(dtype)
data2 = torch.randn((3, 448, 448), device=device).to(dtype)
global_config.max_num_samples_stored = 4 # 2 samples x 2 batch sizes
backend = TensorRTBackend(TensorRTBackendConfig(profiles=TensorRTProfileMode.SAMPLES_USED))
module = ait.Module(model, "toy-model", strategy=ait.OneBackendStrategy(backend).enable_find_max_batch_size(False))
ait.tune(module, DynamicShapeDataset([data1, data2]), batch_sizes=[2, 8], device=device) # 4 exact profiles + fallback
module(data1.repeat(8, 1, 1, 1))
module(data2.repeat(8, 1, 1, 1))

See tests/functional/pytorch/027_aitune_torch_toy_model_tensorrt_backend.py for a full example.

Using your own profiles

You can use your own profiles by setting the profiles argument in the TensorRTBackendConfig class. These profiles assume the module defines forward(input), so input is the path of its top-level tensor parameter:

backend = TensorRTBackend(TensorRTBackendConfig(profiles=[
TensorRTProfile().add_input_shape("input", (3, 224, 224), (3, 224, 224), (3, 224, 224)),
TensorRTProfile().add_input_shape("input", (3, 448, 448), (3, 448, 448), (3, 448, 448)),
]))

Getting the input paths

Profile keys are Python-like tensor paths rooted at their real forward parameter names. For forward(input), the key is "input". A nested dictionary path uses a key such as inputs["tokens"]. You can get the exact keys from the Path column in tuning logs for the default single-profile mode.

INFO - 🎯 Tuning module: `toy-model` (all graphs)
INFO - ------------------------------------------------------------
INFO - πŸš€ Tuning graph `0` for module `toy-model`:
INFO - number of parameters: 0
INFO - number of layers: 0
INFO - precisions:
INFO - graph_spec:
INFO - input_spec:
Tensors:
╒═══════════════╀═════════════════╀═══════════════════════════════╀══════════════════╀══════════════════╀═══════════════╕
β”‚ Access Path β”‚ Semantic Path β”‚ Shape β”‚ Min Shape β”‚ Max Shape β”‚ Dtype β”‚
β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════β•ͺ═══════════════════════════════β•ͺ══════════════════β•ͺ══════════════════β•ͺ═══════════════║
β”‚ input β”‚ input β”‚ ['batch0', 3, 'dim2', 'dim3'] β”‚ [2, 3, 224, 224] β”‚ [8, 3, 448, 448] β”‚ torch.float32 β”‚
β•˜β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•§β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•§β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•§β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•§β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•§β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•›