Inference

Holoscan Inference is the core module that provides holoscan::ops::InferenceOp, an inference operator backed with TensorRT for Holoscan pipelines. Give it one or more model files and the input and output tensor names each model consumes and produces; the operator builds (or loads a cached) TensorRT engine per model, declares one Holoscan port per configured tensor, and runs inference on the fastest dispatch strategy that is safe for your configuration. The backend library is HoloInfer, which owns the per-model TensorRT contexts and the multi-model dispatch manager.

The operator

InferenceOp is constructed from an InferenceOpParams aggregate declared in <holoscan/operators/inference/inference.hpp>. Three fields are required:

Every tensor named in input_map becomes an input port and every tensor in output_map becomes an output port, so upstream and downstream operators connect to inference by tensor name. The remaining fields tune the backend and the dispatch path:

The first time HoloInfer sees an .onnx source it builds a TensorRT engine and caches it next to the model. The cache filename encodes the GPU, compute capability, TensorRT version, precision, DLA core, and a hash of every other build-affecting option, so a stale engine is never reused after a configuration change.

Configure from YAML

InferenceOpParams has two loaders that share one schema, keyed 1:1 on the field names above:

• InferenceOpParams::from_yaml(path) reads the file’s root as one operator’s parameters — the convention used by every example config.
• InferenceOpParams::from_yaml_node(node, config_dir) reads an already parsed node, so one YAML file can configure several operators under distinct keys. Relative model paths resolve against config_dir.

# one InferenceOp: the file root is the parameter block
model_path_map:
  detector: models/detector.onnx
input_map:
  detector: [preprocessed]
output_map:
  detector: [boxes, scores]
enable_fp16: true  # TensorRT before 11; ignored by TensorRT 11
enable_cuda_graphs: true

#include <holoscan/operators/inference/inference.hpp>
auto params = holoscan::ops::InferenceOpParams::from_yaml(config_path);
auto infer = graph.op<holoscan::ops::InferenceOp>("inference", std::move(params));

The allocator field cannot be populated from YAML; assign it in code after loading when a non-default allocator is required. Both loaders report a missing required key with std::runtime_error.

Dispatch selection

At start() the operator inspects its configuration and selects one of seven dispatch kinds. There is no user-facing opt-in; the fast paths apply automatically when every model uses the trt backend, inputs are device-resident, and no model’s output feeds another model’s input:

When a configuration is unsafe for the requested path, the operator degrades one rung at a time and logs a fallback triggered — <REASON> warning that names the requested and selected paths and suggests a fix. Configuration-time rungs include a stream pool smaller than the model count (demote to sequential), chained models or host inputs (use kStandard), and dynamic-shape parallel dispatch with CUDA Graphs but no green-context pool (serialize). One runtime rung swaps a blocking-mode pool stream for a non-blocking one when CUDA Graphs require it.

Example

The module ships modules/inference/examples with eight YAML configurations that progress from a single model to concurrent dynamic-shape models with CUDA Graphs and context isolation:

./run build
./run launch --run-cmd './build-x86_64/modules/inference/examples/holoinfer_example --config modules/inference/examples/configs/01_single_model.yaml'

See modules/inference/examples for the per-scenario walkthrough.

Build, test, and consume

HOLOSCAN_BUILD_INFERENCE defaults to ON. InferenceOp requires CUDA and TensorRT 10.3 or newer at runtime; see the installation footnotes for platform coverage. To run PyTorch models next to the pipeline, install PyTorch as described in Installing PyTorch. Build and test through the SDK-level runner:

./run build --config "-DHOLOSCAN_BUILD_INFERENCE=ON -DHOLOSCAN_BUILD_TESTS=ON"
./run test --options "-L inference"

For installed consumers, use the main Holoscan package to load Inference as a component and link the operator and backend targets:

find_package(holoscan COMPONENTS inference REQUIRED)
target_link_libraries(my_pipeline PRIVATE holoscan::core holoscan::op_inference holoscan::infer)

Alternatively, load the dedicated Inference package and link its aggregate target:

find_package(holoscan-inference
    HINTS "/path/to/holoscan/lib/cmake/holoscan"
    REQUIRED)
target_link_libraries(my_pipeline PRIVATE holoscan::inference)

Replace the example path with the SDK’s shared package directory. The standalone package config is installed under lib/cmake/holoscan, so a bare find_package(holoscan-inference) may not find it through the usual package-name search. You can instead set holoscan-inference_DIR to <prefix>/lib/cmake/holoscan (for example, with -Dholoscan-inference_DIR=<prefix>/lib/cmake/holoscan).

Recap

• InferenceOp is a TensorRT-only inference operator; models, ports, and options are described by InferenceOpParams, in C++ or YAML.
• TensorRT engines are built once from ONNX and cached with a configuration-aware filename.
• The operator picks the fastest safe dispatch kind at start() and logs a named fallback whenever it must degrade.
• Build with HOLOSCAN_BUILD_INFERENCE; consume with find_package(holoscan COMPONENTS inference), or use the dedicated find_package(holoscan-inference) package with its package-directory hint.
• Next: Vision.