Class TrtInfer

• Defined in File core.hpp

Base Type

• public holoscan::inference::InferBase (Class InferBase)

class TrtInfer : public holoscan::inference::InferBase

Class to execute TensorRT based inference

Public Functions

TrtInfer(const std::string &model_path, const std::string &model_name, const std::vector<int32_t> &trt_opt_profile, int device_id, int device_id_dt, bool enable_fp16, bool is_engine_path, bool cuda_buf_in, bool cuda_buf_out)

Constructor.

~TrtInfer()

Destructor.

virtual InferStatus do_inference(const std::vector<std::shared_ptr<DataBuffer>> &input_data, std::vector<std::shared_ptr<DataBuffer>> &output_buffer, cudaEvent_t cuda_event_data, cudaEvent_t *cuda_event_inference)

Does the Core inference with TRT backend The provided CUDA data event is used to prepare the input data any execution of CUDA work should be in sync with this event. If the inference is using CUDA it should record a CUDA event and pass it back in cuda_event_inference.

Parameters

• input_data – Input DataBuffer

• output_buffer – Output DataBuffer, is populated with inferred results

• cuda_event_data – CUDA event recorded after data transfer

• cuda_event_inference – CUDA event recorded after inference

Returns

InferStatus

virtual std::vector<std::vector<int64_t>> get_input_dims() const

Get input data dimensions to the model.

Returns

Vector of values as dimension

virtual std::vector<std::vector<int64_t>> get_output_dims() const

Get output data dimensions from the model.

Returns

Vector of input dimensions. Each dimension is a vector of int64_t corresponding to the shape of the input tensor.

virtual std::vector<holoinfer_datatype> get_input_datatype() const

Get input data types from the model.

Returns

Vector of input dimensions. Each dimension is a vector of int64_t corresponding to the shape of the input tensor.

virtual std::vector<holoinfer_datatype> get_output_datatype() const

Get output data types from the model.

Returns

Vector of values as datatype per output tensor

inline virtual void cleanup()