holoscan::ops::InferenceOpParams

Compile-time-authored configuration for InferenceOp.

Configuration is supplied as one immutable aggregate when the Operator is instantiated on the authoring Graph (graph.op<InferenceOp>("name", ``InferenceOpParams``{...})). InferenceOp::setup() reads params() to dynamically declare one Input<Tensor>/Output<Tensor> port per configured tensor name (see inference.cpp).

#include <holoscan/ops/inference.hpp>

Static methods

from_yaml

static InferenceOpParams holoscan::ops::InferenceOpParams::from_yaml(
    std::string_view path
)

Loads an InferenceOpParams aggregate from a YAML file.

The YAML schema is 1:1 with the fields of this struct:

Every top-level key is optional except model_path_map, input_map, and output_map; absent scalar/bool keys fall back to the member’s in-class default shown above. The cuda_stream_pool and cuda_green_context_pool sub-maps are the only keys that construct real resource objects (holoscan::CudaStreamPool / holoscan::CudaGreenContextPool, respectively) rather than populating a plain-data field; when a sub-map is absent, the corresponding shared_ptr member stays null, matching a default-constructed InferenceOpParams.

Returns: The populated InferenceOpParams.

Throws: std::runtime_error if the file cannot be read/parsed, or a required key (model_path_map, input_map, output_map) is missing.

Parameters

Example

model_path_map: {model_a: /path/to/model_a.onnx}
backend: trt                      # default: "trt"
backend_map: {model_a: trt}       # optional per-model override
input_map: {model_a: [input_tensor]}
output_map: {model_a: [output_tensor]}
trt_opt_profile: {model_a: ["1,4,8"]}  # optional; "min,opt,max" per dynamic dim
is_engine_path: false
input_on_cuda: true
output_on_cuda: true
transmit_on_cuda: true
enable_cuda_graphs: true
enable_fp16: false
dla_core: -1
dla_core_map: {model_a: "0", model_b: "-1"}  # optional per-model DLA core override
dla_gpu_fallback: true
parallel_inference: false
max_batch_size: 256                          # TRT engine-build batch ceiling
max_memory: 10000000000                      # TRT engine-build workspace budget (bytes)
dynamic_input_dims: false
output_dynamic_dims: false
# Optional: constructs a real CudaStreamPool when present.
cuda_stream_pool:
  max_size: 4
# Optional: constructs a real CudaGreenContextPool when present.
cuda_green_context_pool:
  num_partitions: 2
  device_id: 0
  sms_per_partition: [8, 8]   # required when num_partitions >= 2; length must equal
                              # num_partitions; each entry is the SM count for that partition
                              # and must be a multiple of the device's structured_sm_quantum.

from_yaml_node

static InferenceOpParams holoscan::ops::InferenceOpParams::from_yaml_node(
    const YAML::Node &node,
    const std::filesystem::path &config_dir
)

Load an InferenceOpParams aggregate from an already-parsed YAML sub-node.

Use this when a single YAML file configures multiple operators — each operator’s parameters live under a named key at the file’s root. from_yaml_node() reads that subtree and produces the aggregate; config_dir is the directory that relative model_path_map paths resolve against (typically the parent of the YAML file that node was loaded from).

The subtree schema below the named key is identical to what from_yaml() accepts at the root — the same required + optional keys and the same behavior for absent keys.

from_yaml(path) is now a thin wrapper: it loads the file, uses the file’s parent as config_dir, and calls from_yaml_node(root, parent). Existing single-op YAMLs (where the file’s root is the InferenceOp’s section) keep working unchanged.

Returns: The populated InferenceOpParams.

Throws: std::runtime_error if a required key (model_path_map, input_map, output_map) is missing from node.

Parameters

Example

# multi_op_app.yaml — three operators in one file
primary_inference:                 # →
InferenceOpParams::from_yaml_node(root["primary_inference"], dir) model_path_map: {model_a:
models/model_a.onnx} input_map:  {model_a: [tensor_in]} output_map: {model_a: [logits]}
  enable_cuda_graphs: true
secondary_inference:               # → a second InferenceOpParams
  model_path_map: {model_b: models/model_b.onnx}
  input_map:  {model_b: [logits]}
  output_map: {model_b: [prediction]}
other_operator:                    # → some other operator's own from_yaml_node
  ...

Member variables