Optimization profile for dynamic input dimensions and shape tensors. More...
#include <NvInferRuntime.h>
Public Member Functions | |
| bool | setDimensions (char const *inputName, OptProfileSelector select, Dims const &dims) noexcept |
| Set the minimum / optimum / maximum dimensions for a dynamic input tensor. More... | |
| Dims | getDimensions (char const *inputName, OptProfileSelector select) const noexcept |
| Get the minimum / optimum / maximum dimensions for a dynamic input tensor. More... | |
| bool | setShapeValues (char const *inputName, OptProfileSelector select, int32_t const *values, int32_t nbValues) noexcept |
| Set the minimum / optimum / maximum values for an input shape tensor. More... | |
| int32_t | getNbShapeValues (char const *inputName) const noexcept |
| Get the number of values for an input shape tensor. More... | |
| int32_t const * | getShapeValues (char const *inputName, OptProfileSelector select) const noexcept |
| Get the minimum / optimum / maximum values for an input shape tensor. More... | |
| bool | setExtraMemoryTarget (float target) noexcept |
| Set a target for extra GPU memory that may be used by this profile. More... | |
| float | getExtraMemoryTarget () const noexcept |
| Get the extra memory target that has been defined for this profile. More... | |
| bool | isValid () const noexcept |
| Check whether the optimization profile can be passed to an IBuilderConfig object. More... | |
Protected Member Functions | |
| virtual | ~IOptimizationProfile () noexcept=default |
 Protected Member Functions inherited from nvinfer1::INoCopy | |
| INoCopy ()=default | |
| virtual | ~INoCopy ()=default |
| INoCopy (INoCopy const &other)=delete | |
| INoCopy & | operator= (INoCopy const &other)=delete |
| INoCopy (INoCopy &&other)=delete | |
| INoCopy & | operator= (INoCopy &&other)=delete |
Protected Attributes | |
| apiv::VOptimizationProfile * | mImpl |
Detailed Description
Optimization profile for dynamic input dimensions and shape tensors.
When building an ICudaEngine from an INetworkDefinition that has dynamically resizable inputs (at least one input tensor has one or more of its dimensions specified as -1) or shape input tensors, users need to specify at least one optimization profile. Optimization profiles are numbered 0, 1, ... The first optimization profile that has been defined (with index 0) will be used by the ICudaEngine whenever no optimization profile has been selected explicitly. If none of the inputs are dynamic, the default optimization profile will be generated automatically unless it is explicitly provided by the user (this is possible but not required in this case). If more than a single optimization profile is defined, users may set a target how much additional weight space should be maximally allocated to each additional profile (as a fraction of the maximum, unconstrained memory).
Users set optimum input tensor dimensions, as well as minimum and maximum input tensor dimensions. The builder selects the kernels that result in the lowest runtime for the optimum input tensor dimensions, and are valid for all input tensor sizes in the valid range between minimum and maximum dimensions. A runtime error will be raised if the input tensor dimensions fall outside the valid range for this profile. Likewise, users provide minimum, optimum, and maximum values for all shape tensor input values.
Constructor & Destructor Documentation
◆ ~IOptimizationProfile()
|
protectedvirtualdefaultnoexcept |
Member Function Documentation
◆ getDimensions()
|
inlinenoexcept |
Get the minimum / optimum / maximum dimensions for a dynamic input tensor.
If the dimensions have not been previously set via setDimensions(), return an invalid Dims with nbDims == -1.
- Warning
- The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
◆ getExtraMemoryTarget()
|
inlinenoexcept |
Get the extra memory target that has been defined for this profile.
This defaults to 1.0F.
- Returns
- the valid value set by setExtraMemoryTarget or 1.0F.
◆ getNbShapeValues()
|
inlinenoexcept |
Get the number of values for an input shape tensor.
This will return the number of shape values if setShapeValues() has been called before for this input tensor. Otherwise, return -1.
- Warning
- The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
◆ getShapeValues()
|
inlinenoexcept |
Get the minimum / optimum / maximum values for an input shape tensor.
If the shape values have not been set previously with setShapeValues(), this returns nullptr.
- Warning
- The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
◆ isValid()
|
inlinenoexcept |
Check whether the optimization profile can be passed to an IBuilderConfig object.
This function performs partial validation, by e.g. checking that whenever one of the minimum, optimum, or maximum dimensions of a tensor have been set, the others have also been set and have the same rank, as well as checking that the optimum dimensions are always as least as large as the minimum dimensions, and that the maximum dimensions are at least as large as the optimum dimensions. Some validation steps require knowledge of the network definition and are deferred to engine build time.
- Returns
- true if the optimization profile is valid and may be passed to an IBuilderConfig, else false.
◆ setDimensions()
|
inlinenoexcept |
Set the minimum / optimum / maximum dimensions for a dynamic input tensor.
This function must be called three times (for the minimum, optimum, and maximum) for any network input tensor that has dynamic dimensions. If minDims, optDims, and maxDims are the minimum, optimum, and maximum dimensions, and networkDims are the dimensions for this input tensor that are provided to the INetworkDefinition object, then the following conditions must all hold:
(1) minDims.nbDims == optDims.nbDims == maxDims.nbDims == networkDims.nbDims (2) 0 <= minDims.d[i] <= optDims.d[i] <= maxDims.d[i] for i = 0, ..., networkDims.nbDims-1 (3) if networkDims.d[i] != -1, then minDims.d[i] == optDims.d[i] == maxDims.d[i] == networkDims.d[i]
This function may (but need not be) called for an input tensor that does not have dynamic dimensions. In this case, the third argument must always equal networkDims.
- Parameters
-
inputName The input tensor name select Whether to set the minimum, optimum, or maximum dimensions dims The minimum, optimum, or maximum dimensions for this input tensor
- Returns
- false if an inconsistency was detected (e.g. the rank does not match another dimension that was previously set for the same input), true if no inconsistency was detected. Note that inputs can be validated only partially; a full validation is performed at engine build time.
- Warning
- If run on DLA, minimum, optimum, and maximum dimensions must to be the same.
- The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
◆ setExtraMemoryTarget()
|
inlinenoexcept |
Set a target for extra GPU memory that may be used by this profile.
- Parameters
-
target Additional memory that the builder should aim to maximally allocate for this profile, as a fraction of the memory it would use if the user did not impose any constraints on memory. This unconstrained case is the default; it corresponds to target == 1.0. If target == 0.0, the builder aims to create the new optimization profile without allocating any additional weight memory. Valid inputs lie between 0.0 and 1.0. This parameter is only a hint, and TensorRT does not guarantee that the target will be reached. This parameter is ignored for the first (default) optimization profile that is defined.
- Returns
- true if the input is in the valid range (between 0 and 1 inclusive), else false.
◆ setShapeValues()
|
inlinenoexcept |
Set the minimum / optimum / maximum values for an input shape tensor.
This function must be called three times for every input tensor t that is a shape tensor (t.isShape() == true). This implies that the dimensions of t are fixed at network definition time and the volume does not exceed 64. This function must not be called for any input tensor that is not a shape tensor.
Each time this function is called for the same input tensor, the same nbValues must be supplied (either 1 if the tensor rank is 0, or dims.d[0] if the rank is 1). Furthermore, if minVals, optVals, maxVals are the minimum, optimum, and maximum values, it must be true that minVals[i] <= optVals[i] <= maxVals[i] for i = 0, ..., nbValues - 1. Execution of the network must be valid for the optVals.
Shape tensors are tensors that contribute to shape calculations in some way. While input shape tensors can be type kINT32 or kINT64, the values used to set the minimum, optimium, and maximum values must fit in int32_t.
Examples:
- A shape tensor used as the second input to IShuffleLayer can contain a -1 wildcard. The corresponding minVal[i] should be -1.
- A shape tensor used as the stride input to ISliceLayer can contain any valid strides. The values could be positive, negative, or zero.
- A shape tensor subtracted from zero to compute the size input of an ISliceLayer can contain any non-positive values that yield a valid slice operation.
Tightening the minVals and maxVals bounds to cover only values that are necessary may help optimization.
- Parameters
-
inputName The input tensor name select Whether to set the minimum, optimum, or maximum input values. values An array of length nbValues containing the minimum, optimum, or maximum shape tensor elements. For multidimensional tensors, the array is in row-major order. nbValues The length of the value array, which must equal the number of shape tensor elements (>= 1)
- Returns
- false if an inconsistency was detected (e.g. nbValues does not match a previous call for the same tensor), else true. As for setDimensions(), a full validation can only be performed at engine build time.
- Warning
- If run on DLA, minimum, optimum, and maximum shape values must to be the same.
- The string inputName must be null-terminated, and be at most 4096 bytes including the terminator.
Member Data Documentation
◆ mImpl
|
protected |
The documentation for this class was generated from the following file:
