tensorrt.lite

The lite package contains a

Default engine implementation

Engine

class tensorrt.lite.Engine(**kwargs)[source]

A TensorRT engine with self containted logger, runtime, context and memory

Members:
  • logger tensorrt.infer.Logger: Engine Logger
  • log_sev tensorrt.infer.LogSeverity: Verboseness of the logger
  • max_batch_size int: Maximum supported batch size
  • max_workspace_size int: Maximum workspace size
  • data_type tensorrt.infer.DataType: Operating data type of the engine
  • src_framework str: Parser used to create engine
  • runtime tensorrt.infer.Runtime: Engine runtime
  • engine tensorrt.infer.CudaEngine: TensorRT Engine
  • context tensorrt.infer.ExecutionContext: Engine execution context
  • profiler tensorrt.infer.Profiler: Engine profiler
  • input_dim [tensorrt.infer.DimsCHW]: Input layer dimensions
  • output_dim [tensorrt.infer.DimsCHW]: Output layer dimensions
  • input_names [str]: Input layer names
  • output_names [str]: Output layer names
  • d_input [pycuda.DeviceAllocation]: GPU Buffer allocations
  • d_output [pycuda.DeviceAllocation]: GPU Buffer allocations
  • preprocessors {str:function}: Dictionary of input layer names and preprocessing functions (or None)
  • postprocessors {str:function}: Dictionary of output layer names and postprocessing functions (or None)
  • bindings [int]: Buffer Pointers
- __init__(self, **kwargs)
- __del__(self)
- _create_engine(self, **kwargs)
- infer(self, *input_data)
- save(self, str)
- supported_data_format(self, *input_data)
- uniform_data_format(self, *input_data)
- convert_LCHW_to_LNCHW(self, input_data)
- transform_to_LNCHW(self, *input_data)
- verify_data_type(self, *input_data)
- format_data(self, output_data)
- apply_postprocessing(self, layer_data, layer_postprocessor)
- log_info(self, msg)
- log_error(self, msg)
- log_warn(self, msg)
apply_postprocessing(layer_data, layer_postprocessor)[source]

Apply the user specified postprocessing function to results

Takes outputs for a layer and the layer’s specified postprocessor function and processes each result (3D numpy array). Stores the result in the place of the 3D Numpy array.

Parameters:
  • layer_data list/numpy.ndarray (-) – Formated results for a layer
  • layer_postprocessor function (-) – Layer’s post processing function
Results:
  • list/numpy.ndarray: Post processed results
convert_LCHW_to_LNCHW(input_data)[source]

Converts data from LCHW format to LNCHW

Helper for transform_to_LNCHW to convert LCHW format to LNCHW

Parameters:input_data list List of lists of LCHW data (-) –
Returns:
  • list List of list of LNCHW data
format_data(output_data)[source]

Format results to same shape as input data

Format the results from inference from the operating data format (LNCHW) to the same data format that input data was given in.

Parameters:output_data [LNCHW data] (-) – List of inference results in LNCHW
Returns:List of results for each layer in the same format as the input data
Return type:
  • list
infer(*input_data)[source]

Run inference on a set of data

Runs inference on a set of data provided by the user. Data must be provided in a supportted data format:

  • CHW: Single 3D numpy array in form Channels x Height x Width
  • NCHW: Single 4D numpy array in form Batch Size x Channels x Height x Width note: If the batch size is larger than the supported max batch size, the function will attempt to split up the batch into smaller supported batches
  • ZNCHW: Single 5D numpy array in the form Number of Batces x Batch size x Channels x Height x Width note: if the batch size is larger than the supported max batch size, the function will error out
  • LNCHW: List of 4D numpy arrays in form Batch Size x Channels x Height x Width note: if the batch size is larger than the supported max batch size, the function will error out
  • LLCHW: List of lists of 3D numpy arrays in form Channels x Height x Width note: if the size of the inner lists are is larger than the supported max batch size, or the size of the inner lists are not uniform the function will error out
  • LCHW: List of 3D numpy arrays in form Channels x Height x Width note: If the size of the list is larger than the supported max batch size, the function will attempt to split up the list into smaller supported batches

Provide a seperate array for each input layer

If a preprocessor function table is registered with the engine at creation then before inference, each input data object (3D numpy array) will be passed into the user specified preprocessor function for the relevant input layer and inference will be run on the preprocessed data.

If a postprocessor function table is registered with the engine at creation then before inference, each output data object (3D numpy array) will be passed into the user specified postprocessor function for the relevant output layer and the function will return the postprocessed data.

Parameters:input_data,.. list/numpy.ndarray (-) – List or numpy array containing data in a supported format, multiple lists/np.ndarrays should be passed in if there are multiple input layers, one per layer in order of bindings
Returns:Results of inference arranged in the same format the input data was
Return type:
  • list/numpy.ndarray
log_error(msg)[source]

Helper for printing engine errors

Parameters:msg str (-) – What to print
Side-effects:
  • Prints message to console in the ERROR stream
Raises:
  • ValueError
log_info(msg)[source]

Helper for printing engine status

Parameters:msg str (-) – What to print
Side-effects:
  • Prints message to console in the INFO stream
log_warn(msg)[source]

Helper for printing engine warnings

Parameters:msg str (-) – What to print
Side-effects:
  • Prints message to console in the WARNING stream
save(path)[source]

Save the TensorRT Engine to a PLAN file

Saves the TensorRT Engine to a PLAN file that can be used later. Note: This saves the only the internal TensorRT engine in the class not the Engine object and all provided settings.

Parameters:path str (-) – Desired path to save file
supported_data_format(*input_data)[source]

Dectects wether the provided data is one of the supported types

Parameters:input_data tuple Tuple of lists of data for input layers (-) –
Returns:
  • None
Side-Effects:
  • Sets the input format detected (LLCHW, LNCHW, LCHW, ZNCHW, NCHW, CHW)
Raises:- ``Value Error`` – Unsupported data format
transform_to_LNCHW(*input_data)[source]

Converts supported data formats to LNCHW

Converts data from LLCHW, LCHW, ZNCHW, NCHW, CHW to LNCHW (List of batches)

For LLCHW, ZNCHW, LNCHW; batch size (Length of internal lists) must be smaller than the engine’s max batch size (default: 1)

Parameters:input_data tuple Tuple of lists of data for input layers (-) –
Returns:Data formated as a list of batches for inference
Return type:
  • list
Raises:- ``Value Error`` – Batch size too large
uniform_data_format(*input_data)[source]

Verifies that the data format is uniform accross all input layers and that it is uniform accross batches

Parameters:input_data tuple Tuple of lists of data for input layers (-) –
Returns:
  • None
Raises:- ``Value Error`` – If number of batches, batch size or CHW dims differ or if the batch size is too big
verify_data_type(input_data)[source]

Verifies if the data type is the expected type for the engine

Verifies if the data type is the expected type for the engine, will attempt to convert the data to the expected type.

Will provide a warning if a different data type is detected. May be the cause of incorrect results from the engine, if the data cannot be converted.

Parameters:input_data list (-) – List of LNCHW data
Returns:List of LNCHW data with the correct type
Return type:
  • list
Side-Effects:
  • If data type is changed, warning will be printed in the engine logger, make sure logger severity is to at least warning to see.