Refitter

class tensorrt.Refitter(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger: tensorrt.tensorrt.ILogger) None

Updates weights in an ICudaEngine .

Variables
  • error_recorderIErrorRecorder Application-implemented error reporting interface for TensorRT objects.

  • loggerILogger The logger provided when creating the refitter.

  • max_threadsint The maximum thread that can be used by the Refitter.

  • weights_validationbool The flag to indicate whether to validate weights in the refitting process.

Parameters
  • engine – The engine to refit.

  • logger – The logger to use.

__del__(self: tensorrt.tensorrt.Refitter) None
__init__(self: tensorrt.tensorrt.Refitter, engine: tensorrt.tensorrt.ICudaEngine, logger: tensorrt.tensorrt.ILogger) None
Parameters
  • engine – The engine to refit.

  • logger – The logger to use.

get_all(self: tensorrt.tensorrt.Refitter) Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]

Get description of all weights that could be refitted.

Returns

The names of layers with refittable weights, and the roles of those weights.

get_all_weights(self: tensorrt.tensorrt.Refitter) List[str]

Get names of all weights that could be refitted.

Returns

The names of refittable weights.

get_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str) tuple

Gets the dynamic range of a tensor. If the dynamic range was never set, returns the range computed during calibration.

Parameters

tensor_name – The name of the tensor whose dynamic range to retrieve.

Returns

Tuple[float, float] A tuple containing the [minimum, maximum] of the dynamic range.

get_missing(self: tensorrt.tensorrt.Refitter) Tuple[List[str], List[tensorrt.tensorrt.WeightsRole]]

Get description of missing weights.

For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any unsupplied Weights in the combination are considered missing.

Returns

The names of layers with missing weights, and the roles of those weights.

get_missing_weights(self: tensorrt.tensorrt.Refitter) List[str]

Get names of missing weights.

For example, if some Weights have been set, but the engine was optimized in a way that combines weights, any unsupplied Weights in the combination are considered missing.

Returns

The names of missing weights, empty string for unnamed weights.

get_named_weights(self: tensorrt.tensorrt.Refitter, weights_name: str) tensorrt.tensorrt.Weights

Get weights associated with the given name.

If the weights were never set, returns null weights and reports an error to the refitter errorRecorder.

Parameters

weights_name – The name of the weights to be refitted.

Returns

Weights associated with the given name.

get_tensors_with_dynamic_range(self: tensorrt.tensorrt.Refitter) List[str]

Get names of all tensors that have refittable dynamic ranges.

Returns

The names of tensors with refittable dynamic ranges.

get_weights_location(self: tensorrt.tensorrt.Refitter, weights_name: str) tensorrt.tensorrt.TensorLocation

Get location for the weights associated with the given name.

If the weights were never set, returns TensorLocation.HOST and reports an error to the refitter errorRecorder.

Parameters

weights_name – The name of the weights to be refitted.

Returns

Location for the weights associated with the given name.

get_weights_prototype(self: tensorrt.tensorrt.Refitter, weights_name: str) tensorrt.tensorrt.Weights

Get the weights prototype associated with the given name.

The dtype and size of weights prototype is the same as weights used for engine building. The size of the weights prototype is -1 when the name of the weights is None or does not correspond to any refittable weights.

Parameters

weights_name – The name of the weights to be refitted.

Returns

weights prototype associated with the given name.

refit_cuda_engine(self: tensorrt.tensorrt.Refitter) bool

Refits associated engine.

If False is returned, a subset of weights may have been refitted.

The behavior is undefined if the engine has pending enqueued work. Provided weights on CPU or GPU can be unset and released, or updated after refit_cuda_engine returns.

IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead.

Returns

True on success, or False if new weights validation fails or get_missing_weights() != 0 before the call.

refit_cuda_engine_async(self: tensorrt.tensorrt.Refitter, stream_handle: int) bool

Enqueue weights refitting of the associated engine on the given stream.

If False is returned, a subset of weights may have been refitted.

The behavior is undefined if the engine has pending enqueued work on a different stream from the provided one. Provided weights on CPU can be unset and released, or updated after refit_cuda_engine_async returns. Freeing or updating of the provided weights on GPU can be enqueued on the same stream after refit_cuda_engine_async returns.

IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead. The weights updating task should use the the same stream as the one used for the refit call.

Parameters

stream – The stream to enqueue the weights updating task.

Returns

True on success, or False if new weights validation fails or get_missing_weights() != 0 before the call.

set_dynamic_range(self: tensorrt.tensorrt.Refitter, tensor_name: str, range: List[float]) bool

Update dynamic range for a tensor.

Parameters
  • tensor_name – The name of the tensor whose dynamic range to update.

  • range – The new range.

Returns

True if successful, False otherwise.

Returns false if there is no Int8 engine tensor derived from a network tensor of that name. If successful, then get_missing() may report that some weights need to be supplied.

set_named_weights(*args, **kwargs)

Overloaded function.

  1. set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights) -> bool

    Specify new weights of given name. Possible reasons for rejection are:

    • The name of weights is empty or does not correspond to any refittable weights.

    • The size of the weights is inconsistent with the size returned from calling get_weights_prototype() with the same name.

    • The dtype of the weights is inconsistent with the dtype returned from calling get_weights_prototype() with the same name.

    Modifying the weights before refit_cuda_engine() or refit_cuda_engine_async() returns will result in undefined behavior.

    arg name

    The name of the weights to be refitted.

    arg weights

    The new weights to associate with the name.

    returns

    True on success, or False if new weights are rejected.

  2. set_named_weights(self: tensorrt.tensorrt.Refitter, name: str, weights: tensorrt.tensorrt.Weights, location: tensorrt.tensorrt.TensorLocation) -> bool

    Specify new weights on a specified device of given name. Possible reasons for rejection are:

    • The name of weights is empty or does not correspond to any refittable weights.

    • The size of the weights is inconsistent with the size returned from calling get_weights_prototype() with the same name.

    • The dtype of the weights is inconsistent with the dtype returned from calling get_weights_prototype() with the same name.

    It is allowed to provide some weights on CPU and others on GPU. Modifying the weights before refit_cuda_engine() or refit_cuda_engine_async() returns will result in undefined behavior.

    arg name

    The name of the weights to be refitted.

    arg weights

    The new weights on the specified device.

    arg location

    The location (host vs. device) of the new weights.

    returns

    True on success, or False if new weights are rejected.

set_weights(self: tensorrt.tensorrt.Refitter, layer_name: str, role: tensorrt.tensorrt.WeightsRole, weights: tensorrt.tensorrt.Weights) bool

Specify new weights for a layer of given name. Possible reasons for rejection are:

  • There is no such layer by that name.

  • The layer does not have weights with the specified role.

  • The size of weights is inconsistent with the layer’s original specification.

Modifying the weights before refit_cuda_engine() or refit_cuda_engine_async() returns will result in undefined behavior.

Parameters
  • layer_name – The name of the layer.

  • role – The role of the weights. See WeightsRole for more information.

  • weights – The weights to refit with.

Returns

True on success, or False if new weights are rejected.

unset_named_weights(self: tensorrt.tensorrt.Refitter, weights_name: str) bool

Unset weights associated with the given name.

Unset weights before releasing them.

Parameters

weights_name – The name of the weights to be refitted.

Returns

False if the weights were never set, returns True otherwise.