Algorithm Selector

class tensorrt.IAlgorithmIOInfo

This class carries information about input or output of the algorithm. IAlgorithmIOInfo for all the input and output along with IAlgorithmVariant denotes the variation of algorithm and can be used to select or reproduce an algorithm using IAlgorithmSelector.select_algorithms().

Variables
  • tensor_formatTensorFormat [DEPRECATED] TensorFormat of the input/output of algorithm. This is deprecated since the strides, data type, and vectorization information is sufficient to uniquely identify tensor formats.

  • dtypeDataType DataType of the input/output of algorithm.

  • stridesDims strides of the input/output tensor of algorithm.

  • vectorized_dimint the index of the vectorized dimension or -1 for non-vectorized formats.

  • components_per_elementint the number of components per element. This is always 1 for non-vectorized formats.

__init__(*args, **kwargs)
class tensorrt.IAlgorithmVariant

provides a unique 128-bit identifier, which along with the input and output information denotes the variation of algorithm and can be used to select or reproduce an algorithm, using IAlgorithmSelector.select_algorithms() see IAlgorithmIOInfo, IAlgorithm, IAlgorithmSelector.select_algorithms() note A single implementation can have multiple tactics.

Variables
  • implementationint implementation of the algorithm.

  • tacticint tactic of the algorithm.

__init__(*args, **kwargs)
class tensorrt.IAlgorithmContext

Describes the context and requirements, that could be fulfilled by one or more instances of IAlgorithm. see IAlgorithm

Variables
  • namestr name of the algorithm node.

  • num_inputsint number of inputs of the algorithm.

  • num_outputsint number of outputs of the algorithm.

__init__(*args, **kwargs)
get_shape(self: tensorrt.tensorrt.IAlgorithmContext, index: int) List[tensorrt.tensorrt.Dims]

Get the minimum / optimum / maximum dimensions for a dynamic input tensor.

Parameters

index – Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the outputs.

Returns

A List[Dims] of length 3, containing the minimum, optimum, and maximum shapes, in that order. If the shapes have not been set yet, an empty list is returned.`

class tensorrt.IAlgorithm

Application-implemented interface for selecting and reporting the tactic selection of a layer. Tactic Selection is a step performed by the builder for deciding best algorithms for a layer.

Variables
  • algorithm_variantIAlgorithmVariant& the algorithm variant.

  • timing_msecfloat The time in milliseconds to execute the algorithm.

  • workspace_sizeint The size of the GPU temporary memory in bytes which the algorithm uses at execution time.

__init__(*args, **kwargs)
get_algorithm_io_info(self: tensorrt.tensorrt.IAlgorithm, index: int) tensorrt.tensorrt.IAlgorithmIOInfo

A single call for both inputs and outputs. Incremental numbers assigned to indices of inputs and the outputs.

Parameters

index – Index of the input or output of the algorithm. Incremental numbers assigned to indices of inputs and the outputs.

Returns

A IAlgorithmIOInfo&

class tensorrt.IAlgorithmSelector(self: tensorrt.tensorrt.IAlgorithmSelector) None

Interface implemented by application for selecting and reporting algorithms of a layer provided by the builder. note A layer in context of algorithm selection may be different from ILayer in INetworkDefiniton. For example, an algorithm might be implementing a conglomeration of multiple ILayers in INetworkDefinition.

To implement a custom algorithm selector, ensure that you explicitly instantiate the base class in __init__() :

class MyAlgoSelector(trt.IAlgorithmSelector):
    def __init__(self):
        trt.IAlgorithmSelector.__init__(self)
__init__(self: tensorrt.tensorrt.IAlgorithmSelector) None
report_algorithms(self: tensorrt.tensorrt.IAlgorithmSelector, contexts: List[tensorrt.tensorrt.IAlgorithmContext], choices: List[tensorrt.tensorrt.IAlgorithm]) None

Called by TensorRT to report choices it made.

Note: For a given optimization profile, this call comes after all calls to select_algorithms. choices[i] is the choice that TensorRT made for algoContexts[i], for i in [0, num_algorithms-1]

For example, a possible implementation may look like this:

def report_algorithms(self, contexts, choices):
    # Prints the time of the chosen algorithm by TRT from the
    # selection list passed in by select_algorithms
    for choice in choices:
        print(choice.timing_msec)
Parameters
  • contexts – The list of all algorithm contexts.

  • choices – The list of algorithm choices made by TensorRT corresponding to each context.

select_algorithms(self: tensorrt.tensorrt.IAlgorithmSelector, context: tensorrt.tensorrt.IAlgorithmContext, choices: List[tensorrt.tensorrt.IAlgorithm]) List[int]

Select Algorithms for a layer from the given list of algorithm choices.

Note: TRT uses its default algorithm selection to choose from the list returned by the user. If the returned list is empty, TRT’s default algorithm selection is used unless strict type constraints are set. The list of choices is valid only for this specific algorithm context.

For example, the simplest implementation looks like this:

def select_algorithms(self, context, choices):
    assert len(choices) > 0
    return list(range(len(choices)))
Parameters
  • context – The context for which the algorithm choices are valid.

  • choices – The list of algorithm choices to select for implementation of this layer.

Returns

A List[int] indicating the indices from the choices vector that TensorRT should choose from.