tensorrt.lite¶
The lite
package contains a
Default engine implementation
Engine¶
-
class
tensorrt.lite.
Engine
(**kwargs)[source]¶ A TensorRT engine with self containted logger, runtime, context and memory
- Members:
- logger
tensorrt.infer.Logger
: Engine Logger - log_sev
tensorrt.infer.LogSeverity
: Verboseness of the logger - max_batch_size
int
: Maximum supported batch size - max_workspace_size
int
: Maximum workspace size - data_type
tensorrt.infer.DataType
: Operating data type of the engine - src_framework
str
: Parser used to create engine - runtime
tensorrt.infer.Runtime
: Engine runtime - engine
tensorrt.infer.CudaEngine
: TensorRT Engine - context
tensorrt.infer.ExecutionContext
: Engine execution context - profiler
tensorrt.infer.Profiler
: Engine profiler - input_dim
[tensorrt.infer.DimsCHW]
: Input layer dimensions - output_dim
[tensorrt.infer.DimsCHW]
: Output layer dimensions - input_names
[str]
: Input layer names - output_names
[str]
: Output layer names - d_input
[pycuda.DeviceAllocation]
: GPU Buffer allocations - d_output
[pycuda.DeviceAllocation]
: GPU Buffer allocations - preprocessors
{str:function}
: Dictionary of input layer names and preprocessing functions (or None) - postprocessors
{str:function}
: Dictionary of output layer names and postprocessing functions (or None) - bindings
[int]
: Buffer Pointers
- logger
-
- __init__(self, **kwargs)
-
- __del__(self)
-
- _create_engine(self, **kwargs)
-
- infer(self, *input_data)
-
- save(self, str)
-
- supported_data_format(self, *input_data)
-
- uniform_data_format(self, *input_data)
-
- convert_LCHW_to_LNCHW(self, input_data)
-
- transform_to_LNCHW(self, *input_data)
-
- verify_data_type(self, *input_data)
-
- format_data(self, output_data)
-
- apply_postprocessing(self, layer_data, layer_postprocessor)
-
- log_info(self, msg)
-
- log_error(self, msg)
-
- log_warn(self, msg)
-
apply_postprocessing
(layer_data, layer_postprocessor)[source]¶ Apply the user specified postprocessing function to results
Takes outputs for a layer and the layer’s specified postprocessor function and processes each result (3D numpy array). Stores the result in the place of the 3D Numpy array.
Parameters: - layer_data list/numpy.ndarray (-) – Formated results for a layer
- layer_postprocessor function (-) – Layer’s post processing function
- Results:
list/numpy.ndarray
: Post processed results
-
convert_LCHW_to_LNCHW
(input_data)[source]¶ Converts data from LCHW format to LNCHW
Helper for transform_to_LNCHW to convert LCHW format to LNCHW
Parameters: input_data list List of lists of LCHW data (-) – Returns: list
List of list of LNCHW data
-
format_data
(output_data)[source]¶ Format results to same shape as input data
Format the results from inference from the operating data format (LNCHW) to the same data format that input data was given in.
Parameters: output_data [LNCHW data] (-) – List of inference results in LNCHW Returns: List of results for each layer in the same format as the input data Return type: list
-
infer
(*input_data)[source]¶ Run inference on a set of data
Runs inference on a set of data provided by the user. Data must be provided in a supportted data format:
- CHW: Single 3D numpy array in form Channels x Height x Width
- NCHW: Single 4D numpy array in form Batch Size x Channels x Height x Width note: If the batch size is larger than the supported max batch size, the function will attempt to split up the batch into smaller supported batches
- ZNCHW: Single 5D numpy array in the form Number of Batces x Batch size x Channels x Height x Width note: if the batch size is larger than the supported max batch size, the function will error out
- LNCHW: List of 4D numpy arrays in form Batch Size x Channels x Height x Width note: if the batch size is larger than the supported max batch size, the function will error out
- LLCHW: List of lists of 3D numpy arrays in form Channels x Height x Width note: if the size of the inner lists are is larger than the supported max batch size, or the size of the inner lists are not uniform the function will error out
- LCHW: List of 3D numpy arrays in form Channels x Height x Width note: If the size of the list is larger than the supported max batch size, the function will attempt to split up the list into smaller supported batches
Provide a seperate array for each input layer
If a preprocessor function table is registered with the engine at creation then before inference, each input data object (3D numpy array) will be passed into the user specified preprocessor function for the relevant input layer and inference will be run on the preprocessed data.
If a postprocessor function table is registered with the engine at creation then before inference, each output data object (3D numpy array) will be passed into the user specified postprocessor function for the relevant output layer and the function will return the postprocessed data.
Parameters: input_data,.. list/numpy.ndarray (-) – List or numpy array containing data in a supported format, multiple lists/np.ndarrays should be passed in if there are multiple input layers, one per layer in order of bindings Returns: Results of inference arranged in the same format the input data was Return type: list/numpy.ndarray
-
log_error
(msg)[source]¶ Helper for printing engine errors
Parameters: msg str (-) – What to print - Side-effects:
- Prints message to console in the ERROR stream
Raises: - ValueError
-
log_info
(msg)[source]¶ Helper for printing engine status
Parameters: msg str (-) – What to print - Side-effects:
- Prints message to console in the INFO stream
-
log_warn
(msg)[source]¶ Helper for printing engine warnings
Parameters: msg str (-) – What to print - Side-effects:
- Prints message to console in the WARNING stream
-
save
(path)[source]¶ Save the TensorRT Engine to a PLAN file
Saves the TensorRT Engine to a PLAN file that can be used later. Note: This saves the only the internal TensorRT engine in the class not the Engine object and all provided settings.
Parameters: path str (-) – Desired path to save file
-
supported_data_format
(*input_data)[source]¶ Dectects wether the provided data is one of the supported types
Parameters: input_data tuple Tuple of lists of data for input layers (-) – Returns: None
- Side-Effects:
- Sets the input format detected (LLCHW, LNCHW, LCHW, ZNCHW, NCHW, CHW)
Raises: - ``Value Error`` – Unsupported data format
-
transform_to_LNCHW
(*input_data)[source]¶ Converts supported data formats to LNCHW
Converts data from LLCHW, LCHW, ZNCHW, NCHW, CHW to LNCHW (List of batches)
For LLCHW, ZNCHW, LNCHW; batch size (Length of internal lists) must be smaller than the engine’s max batch size (default: 1)
Parameters: input_data tuple Tuple of lists of data for input layers (-) – Returns: Data formated as a list of batches for inference Return type: list
Raises: - ``Value Error`` – Batch size too large
-
uniform_data_format
(*input_data)[source]¶ Verifies that the data format is uniform accross all input layers and that it is uniform accross batches
Parameters: input_data tuple Tuple of lists of data for input layers (-) – Returns: None
Raises: - ``Value Error`` – If number of batches, batch size or CHW dims differ or if the batch size is too big
-
verify_data_type
(input_data)[source]¶ Verifies if the data type is the expected type for the engine
Verifies if the data type is the expected type for the engine, will attempt to convert the data to the expected type.
Will provide a warning if a different data type is detected. May be the cause of incorrect results from the engine, if the data cannot be converted.
Parameters: input_data list (-) – List of LNCHW data Returns: List of LNCHW data with the correct type Return type: list
- Side-Effects:
- If data type is changed, warning will be printed in the engine logger, make sure logger severity is to at least warning to see.