Algorithm Selector
- class tensorrt.IAlgorithmIOInfo
This class carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and output along with IAlgorithmVariant denotes the variation of algorithm and can be used to select or reproduce an algorithm using IAlgorithmSelector.select_algorithms().
- Variables
tensor_format –
TensorFormat
[DEPRECATED] TensorFormat of the input/output of algorithm. This is deprecated since the strides, data type, and vectorization information is sufficient to uniquely identify tensor formats.dtype –
DataType
DataType of the input/output of algorithm.strides –
Dims
strides of the input/output tensor of algorithm.vectorized_dim –
int
the index of the vectorized dimension or -1 for non-vectorized formats.components_per_element –
int
the number of components per element. This is always 1 for non-vectorized formats.
- __init__(*args, **kwargs)
- class tensorrt.IAlgorithmVariant
provides a unique 128-bit identifier, which along with the input and output information denotes the variation of algorithm and can be used to select or reproduce an algorithm, using IAlgorithmSelector.select_algorithms() see IAlgorithmIOInfo, IAlgorithm, IAlgorithmSelector.select_algorithms() note A single implementation can have multiple tactics.
- Variables
implementation –
int
implementation of the algorithm.tactic –
int
tactic of the algorithm.
- __init__(*args, **kwargs)
- class tensorrt.IAlgorithmContext
Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorithm. see IAlgorithm
- Variables
name –
str
name of the algorithm node.num_inputs –
int
number of inputs of the algorithm.num_outputs –
int
number of outputs of the algorithm.
- __init__(*args, **kwargs)
- get_shape(self: tensorrt.tensorrt.IAlgorithmContext, index: int) List[tensorrt.tensorrt.Dims]
Get the minimum / optimum / maximum dimensions for a dynamic input tensor.
- Parameters
index – Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the outputs.
- Returns
A List[Dims] of length 3, containing the minimum, optimum, and maximum shapes, in that order. If the shapes have not been set yet, an empty list is returned.`
- class tensorrt.IAlgorithm
Application-implemented interface for selecting and reporting the tactic selection of a layer. Tactic Selection is a step performed by the builder for deciding best algorithms for a layer.
- Variables
algorithm_variant –
IAlgorithmVariant&
the algorithm variant.timing_msec –
float
The time in milliseconds to execute the algorithm.workspace_size –
int
The size of the GPU temporary memory in bytes which the algorithm uses at execution time.
- __init__(*args, **kwargs)
- get_algorithm_io_info(self: tensorrt.tensorrt.IAlgorithm, index: int) tensorrt.tensorrt.IAlgorithmIOInfo
A single call for both inputs and outputs. Incremental numbers assigned to indices of inputs and the outputs.
- Parameters
index – Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the outputs.
- Returns
A
IAlgorithmIOInfo&
- class tensorrt.IAlgorithmSelector(self: tensorrt.tensorrt.IAlgorithmSelector) None
Interface implemented by application for selecting and reporting algorithms of a layer provided by the builder. note A layer in context of algorithm selection may be different from ILayer in INetworkDefiniton. For example, an algorithm might be implementing a conglomeration of multiple ILayers in INetworkDefinition.
To implement a custom algorithm selector, ensure that you explicitly instantiate the base class in
__init__()
:class MyAlgoSelector(trt.IAlgorithmSelector): def __init__(self): trt.IAlgorithmSelector.__init__(self)
- __init__(self: tensorrt.tensorrt.IAlgorithmSelector) None
- report_algorithms(self: tensorrt.tensorrt.IAlgorithmSelector, contexts: List[tensorrt.tensorrt.IAlgorithmContext], choices: List[tensorrt.tensorrt.IAlgorithm]) None
Called by TensorRT to report choices it made.
Note: For a given optimization profile, this call comes after all calls to select_algorithms. choices[i] is the choice that TensorRT made for algoContexts[i], for i in [0, num_algorithms-1]
For example, a possible implementation may look like this:
def report_algorithms(self, contexts, choices): # Prints the time of the chosen algorithm by TRT from the # selection list passed in by select_algorithms for choice in choices: print(choice.timing_msec)
- Parameters
contexts – The list of all algorithm contexts.
choices – The list of algorithm choices made by TensorRT corresponding to each context.
- select_algorithms(self: tensorrt.tensorrt.IAlgorithmSelector, context: tensorrt.tensorrt.IAlgorithmContext, choices: List[tensorrt.tensorrt.IAlgorithm]) List[int]
Select Algorithms for a layer from the given list of algorithm choices.
Note: TRT uses its default algorithm selection to choose from the list returned by the user. If the returned list is empty, TRT’s default algorithm selection is used unless strict type constraints are set. The list of choices is valid only for this specific algorithm context.
For example, the simplest implementation looks like this:
def select_algorithms(self, context, choices): assert len(choices) > 0 return list(range(len(choices)))
- Parameters
context – The context for which the algorithm choices are valid.
choices – The list of algorithm choices to select for implementation of this layer.
- Returns
A
List[int]
indicating the indices from the choices vector that TensorRT should choose from.