TensorRT Optimization Profiles
Introduction
TensorRT optimization profiles optimize the performance of a TensorRT engine. They are used to profile the performance of a model at different input shapes and batch sizes.
By default, a single profile is generated from the graph spec that supports the minimum and maximum shapes of the input tensors.
Using samples for profile generation
Set the number of samples to use for profile generation
You can set the number of samples to use for profile generation by setting the max_num_samples_stored in the aitune.torch.config module. By default, it is set to 1 as samples are stored for each backend, model module, and each batch size.
Use the TensorRTProfileMode.SAMPLES_USED mode
TensorRTProfileMode.SAMPLES_USED creates an exact profile for each recorded shape. When input shapes vary, it also adds a
wide fallback spanning their range, including the discovered maximum batch size when find-max-batch-size is enabled.
When the module defines explicit dynamic shapes, those definitions take precedence and produce one graph-derived profile instead of sample profiles.
Use the correct samples and the right batch sizes during tuning
The fallback accepts unrecorded shapes within its bounds, while the exact profiles are preferred for recorded shapes. Inputs outside the fallback bounds cannot run.
NOTE: As samples for a single parameter have different shapes, we are wrapping them in a DynamicShapeDataset to handle different shapes.
See tests/functional/pytorch/027_aitune_torch_toy_model_tensorrt_backend.py for a full example.
Using your own profiles
You can use your own profiles by setting the profiles argument in the TensorRTBackendConfig class.
These profiles assume the module defines forward(input), so input is the path of its top-level tensor parameter:
Getting the input paths
Profile keys are Python-like tensor paths rooted at their real forward parameter names. For forward(input),
the key is "input". A nested dictionary path uses a key such as inputs["tokens"]. You can get
the exact keys from the Path column in tuning logs for the default single-profile mode.