IOptimizationProfile
- class tensorrt.IOptimizationProfile
Optimization profile for dynamic input dimensions and shape tensors.
When building an
ICudaEngine
from anINetworkDefinition
that has dynamically resizable inputs (at least one input tensor has one or more of its dimensions specified as -1) or shape input tensors, users need to specify at least one optimization profile. Optimization profiles are numbered 0, 1, …The first optimization profile that has been defined (with index 0) will be used by the
ICudaEngine
whenever no optimization profile has been selected explicitly. If none of the inputs are dynamic, the default optimization profile will be generated automatically unless it is explicitly provided by the user (this is possible but not required in this case). If more than a single optimization profile is defined, users may set a target how much additional weight space should be maximally allocated to each additional profile (as a fraction of the maximum, unconstrained memory).Users set optimum input tensor dimensions, as well as minimum and maximum input tensor dimensions. The builder selects the kernels that result in the lowest runtime for the optimum input tensor dimensions, and are valid for all input tensor sizes in the valid range between minimum and maximum dimensions. A runtime error will be raised if the input tensor dimensions fall outside the valid range for this profile. Likewise, users provide minimum, optimum, and maximum values for all shape tensor input values.
IOptimizationProfile
implements__nonzero__()
and__bool__()
such that evaluating a profile as abool
(e.g.if profile:
) will check whether the optimization profile can be passed to an IBuilderConfig object. This will perform partial validation, by e.g. checking that the maximum dimensions are at least as large as the optimum dimensions, and that the optimum dimensions are always as least as large as the minimum dimensions. Some validation steps require knowledge of the network definition and are deferred to engine build time.- Variables
extra_memory_target – Additional memory that the builder should aim to maximally allocate for this profile, as a fraction of the memory it would use if the user did not impose any constraints on memory. This unconstrained case is the default; it corresponds to
extra_memory_target
== 1.0. Ifextra_memory_target
== 0.0, the builder aims to create the new optimization profile without allocating any additional weight memory. Valid inputs lie between 0.0 and 1.0. This parameter is only a hint, and TensorRT does not guarantee that theextra_memory_target
will be reached. This parameter is ignored for the first (default) optimization profile that is defined.
- get_shape(self: tensorrt.tensorrt.IOptimizationProfile, input: str) List[tensorrt.tensorrt.Dims]
Get the minimum/optimum/maximum dimensions for a dynamic input tensor. If the dimensions have not been previously set via
set_shape()
, return an invalidDims
with a length of -1.- Returns
A
List[Dims]
of length 3, containing the minimum, optimum, and maximum shapes, in that order. If the shapes have not been set yet, an empty list is returned.
- get_shape_input(self: tensorrt.tensorrt.IOptimizationProfile, input: str) List[List[int]]
Get the minimum/optimum/maximum values for a shape input tensor.
- Returns
A
List[List[int]]
of length 3, containing the minimum, optimum, and maximum values, in that order. If the values have not been set yet, an empty list is returned.
- set_shape(self: tensorrt.tensorrt.IOptimizationProfile, input: str, min: tensorrt.tensorrt.Dims, opt: tensorrt.tensorrt.Dims, max: tensorrt.tensorrt.Dims) None
Set the minimum/optimum/maximum dimensions for a dynamic input tensor.
This function must be called for any network input tensor that has dynamic dimensions. If
min
,opt
, andmax
are the minimum, optimum, and maximum dimensions, andreal_shape
is the shape for this input tensor provided to theINetworkDefinition
,then the following conditions must hold:len(min)
==len(opt)
==len(max)
==len(real_shape)
0 <=
min[i]
<=opt[i]
<=max[i]
for alli
if
real_shape[i]
!= -1, thenmin[i]
==opt[i]
==max[i]
==real_shape[i]
This function may (but need not be) called for an input tensor that does not have dynamic dimensions. In this case, all shapes must equal
real_shape
.- Parameters
input – The name of the input tensor.
min – The minimum dimensions for this input tensor.
opt – The optimum dimensions for this input tensor.
max – The maximum dimensions for this input tensor.
- Raises
ValueError
if an inconsistency was detected. Note that inputs can be validated only partially; a full validation is performed at engine build time.
- set_shape_input(self: tensorrt.tensorrt.IOptimizationProfile, input: str, min: List[int], opt: List[int], max: List[int]) None
Set the minimum/optimum/maximum values for a shape input tensor.
This function must be called for every input tensor
t
that is a shape tensor (t.is_shape
==True
). This implies that the datatype oft
isint32
, the rank is either 0 or 1, and the dimensions oft
are fixed at network definition time. This function must NOT be called for any input tensor that is not a shape tensor.If
min
,opt
, andmax
are the minimum, optimum, and maximum values, it must be true thatmin[i]
<=opt[i]
<=max[i]
for alli
.- Parameters
input – The name of the input tensor.
min – The minimum values for this shape tensor.
opt – The optimum values for this shape tensor.
max – The maximum values for this shape tensor.
- Raises
ValueError
if an inconsistency was detected. Note that inputs can be validated only partially; a full validation is performed at engine build time.