public holoscan::GPUResidentOperator (Class GPUResidentOperator)

Class Documentation

class GPUResidentInferenceOp : public holoscan::GPUResidentOperator

GPU Resident Inference Operator class to perform model inference.

==Parameters==

backend: Backend to use for inference. Only TensorRT supported. "trt" for TensorRT (default: "trt").
config_file: Path to the config file.
in_tensor_names: Input tensors.
out_tensor_names: Output tensors.
batch_sizes: Batch sizes per model. Key is model name and value is the batch size.
model_path_map: Model to path map. Key is model name and value is the model path.
pre_processor_map: Model to Pre processed input data (to the model) map. Key is model name and value is the input tensor names to the model.
inference_map: Model to output tensor map. Key is model name and value is the output tensor names from the model.
tensor_to_buffersize: Map holding buffer size per tensor. Key is tensor name and value is the buffer size.
tensor_to_datatype: Map holding data type per tensor. Key is tensor name and value is the data type.
device_map: Device map. Key is model name and value is the GPU ID for inference.
dla_core_map: DLA core map. Key is model name and value is the DLA core index for inference.
temporal_map: Temporal map. Key is model name and value is the frame delay for model inference.
activation_map: Activation map. Key is model name and value is the activation state for model inference.
backend_map: Backend map. Key is model name and value is the backend type for inference.
parallel_inference: Parallel inference flag.
infer_on_cpu: Infer on CPU flag. Always false. Not configurable.
enable_fp16: Enable FP16 flag.
input_on_cuda: Input on CUDA flag. Always true. Not configurable.
output_on_cuda: Output on CUDA flag. Always true. Not configurable.
is_engine_path: Is engine path flag.
use_cuda_graphs: Use CUDA graphs flag. Always false. Not configurable.
dla_core: DLA core index. Default is -1.
dla_gpu_fallback: DLA GPU fallback flag.
dynamic_input_dims: Dynamic input dimensions flag.

Public Functions

HOLOSCAN_OPERATOR_FORWARD_ARGS_SUPER (GPUResidentInferenceOp, holoscan::GPUResidentOperator) GPUResidentInferenceOp()=default: Default constructor.

explicit GPUResidentInferenceOp(const std::string &config_file): Constructor with config file.

virtual void setup(OperatorSpec &spec) override: Setup the operator.

virtual void start() override: Start function of the operator.

virtual void compute(InputContext &op_input, OutputContext &op_output, ExecutionContext &context) override: Compute function of the operator.

virtual void stop() override: Stop function of the operator.

Previous Class ForwardOp

Next Class GXFCodeletOp

Class GPUResidentInferenceOp

Inheritance Relationships

Base Type

Class Documentation