Class InferBase

• Defined in File infer.hpp

Derived Types

• public holoscan::inference::OnnxInfer (Class OnnxInfer)

• public holoscan::inference::TorchInfer (Class TorchInfer)

• public holoscan::inference::TrtInfer (Class TrtInfer)

class InferBase

Base Inference Class.

Subclassed by holoscan::inference::OnnxInfer, holoscan::inference::TorchInfer, holoscan::inference::TrtInfer

Public Functions

virtual ~InferBase() = default

Default destructor.

inline virtual InferStatus do_inference(const std::vector<std::shared_ptr<DataBuffer>> &input_data, std::vector<std::shared_ptr<DataBuffer>> &output_buffer, cudaEvent_t cuda_event_data, cudaEvent_t *cuda_event_inference)

Does the Core inference The provided CUDA data event is used to prepare the input data any execution of CUDA work should be in sync with this event. If the inference is using CUDA it should record a CUDA event and pass it back in cuda_event_inference.

Parameters

• input_data – Input DataBuffer

• output_buffer – Output DataBuffer, is populated with inferred results

• cuda_event_data – CUDA event recorded after data transfer

• cuda_event_inference – CUDA event recorded after inference

Returns

InferStatus

inline virtual bool set_dynamic_input_dimension(const std::vector<std::string> &input_tensors, const std::map<std::string, std::vector<int>> &dims_per_tensor)

Updates the dimensions per tensor in case of dynamic inputs. Using the input Holoscan tensors and their dimension mapping, the internal input size vector is updated.

Parameters

• input_tensors – Vector of input Holoscan tensor names

• dims_per_tensor – Map storing the dimensions as values and Holoscan tensor names as keys.

Returns

true if the dynamic input dimensions were successfully updated, false otherwise

inline virtual std::vector<std::vector<int64_t>> get_input_dims() const

Get input data dimensions to the model.

Returns

Vector of values as dimension

inline virtual std::vector<std::vector<int64_t>> get_output_dims() const

Get output data dimensions from the model.

Returns

Vector of output dimensions. Each dimension is a vector of int64_t corresponding to the shape of the output tensor.

inline virtual std::vector<holoinfer_datatype> get_input_datatype() const

Get input data types from the model.

Returns

Vector of input dimensions. Each dimension is a vector of int64_t corresponding to the shape of the input tensor.

inline virtual std::vector<holoinfer_datatype> get_output_datatype() const

Get output data types from the model.

Returns

Vector of values as datatype per output tensor

inline virtual void cleanup()