Overview

NVIDIA Holoscan SDK 5 is a sensor-processing framework for physical AI — systems such as robots, medical instruments, and industrial inspection lines that sense the physical world, run AI on what they sense, and act on the result within tight time budgets. It is built for pipelines that combine multiple sensors running at different rates and that must hold strict latency bounds from sensor input to actuator or display output.

This page is orientation: what the SDK is, how it is layered, and what ships today. For why the framework looks the way it does — and what changed for developers coming from earlier Holoscan releases — see Highlights.

Built for multi-rate sensor pipelines

Physical AI workloads share a common shape. A camera delivers frames at 60 Hz while an IMU delivers samples at a much higher rate; fusion has to combine them coherently; and a control or display output must react within a bounded time. In pipelines like these, correctness depends as much on when things run as on what they compute.

Holoscan SDK 5 makes that timing explicit and checkable. Each operator declares a temporal contract: what triggers it, which input samples participate in an activation, and what happens when data is missing or stale. The runtime validates the whole pipeline against those declarations before the first operator ever fires, instead of letting scheduling behavior emerge at runtime from queue depths and timing luck. The full treatment lives in Temporal contracts and Time and clocks.

The layered picture

The SDK is organized as layers you can adopt independently. From the bottom up:

A fast interprocess transport underneath

The foundation is HoloMQ, a zero-copy interprocess transport: shared-memory publish–subscribe over named channels, designed for low latency and aware of GPU memory. Payloads move between processes without copies.

The transport is useful on its own. A standalone process — a sensor daemon, a recorder, a UI — can publish into or subscribe to Holoscan channels through the ChannelRuntime client API without adopting anything else from the SDK. Inside a graph, by contrast, you never touch the transport directly: the runtime materializes channels from your port wiring. See Interprocess channels and Standalone channel clients.

An opinionated sensor-processing runtime on top

The Holoscan runtime is a periodic sensor-processing runtime, deliberately opinionated rather than a general-purpose graph-compute framework. An application is a graph of operators connected through statically typed ports (Input<T>/Output<T>), and every operator is three declarations: setup() describes structure, contract() describes time, and compute() does the work — intentionally oblivious to temporal concerns.

Every application then follows the same three-phase shape, compose → compile → run:

class MyPipeline final : public holoscan::Graph {
 protected:
  void compose() override {
    const auto camera = op<CameraSource>("camera");
    const auto fusion = op<WindowFusion>("fusion");
    add_flow(camera->frame, fusion->camera);  // typed ports, checked at wiring time
  }
};
int main() {
  MyPipeline graph;                                               // compose
  const holoscan::ExecutionPlan plan = holoscan::compile(graph);  // compile
  return holoscan::run(plan);                                     // run
}

On Linux, plain blocking run(plan) owns SIGINT/SIGTERM by default and admits transient ownership only when both process-task census reads observe exactly one live task. Multithreaded hosts should establish an early ManagedShutdownSignals domain or choose RunSignalPolicy::kCallerManaged; see Running and stopping.

Each phase has its own failure style: authoring mistakes throw while you compose, compile() returns rejected plans that explain what is infeasible, and a running pipeline fails closed. On the hot path the runtime is bounded by construction — preallocated pools and bounded queues rather than on-demand allocation — and runtime APIs report failures as values (holoscan::expected<T, Error>) rather than exceptions. Start with Core concepts for the model and Errors as values for the failure discipline.

Domain operator modules

Holoscan Modules are separately selectable libraries of operators that extend the core runtime with a domain capability. Holoscan SDK 5.x ships three core modules: Inference (InferenceOp, backed by TensorRT), Vision (nine CV-CUDA and NPP image and detection operators), and Visualization (HolovizOp, backed by the HoloViz renderer). Further domains — multimedia, networking, robotics, sensor bridge — are planned as additional modules in upcoming releases. You can always write your own operators (see Your first operator), and community modules and reference applications live on HoloHub.

What the SDK provides today

• A C++ API. One umbrella header, #include <holoscan/holoscan.hpp>, covers graph composition, operators and temporal contracts, typed parameters, tensors and payload schemas, channels, time and clocks, and logging. C++ is the API surface of this release; see API stability for what may still evolve. The generated C++ API reference documents the core public API.
• Core modules. Inference runs one or more TensorRT models behind a single operator; Vision provides GPU-resident Bayer demosaic, resizing, cropping, borders, remapping, normalization, type conversion, layout conversion, and non-maximum suppression; Visualization renders layered image and geometry output at a fixed display rate.
• Examples. A graded hello_* series (core runtime, logging, schemas, tensors, MatX and CCCL interop) plus fuller reference applications: camera_imu_control (camera + IMU fusion driving a control output), slow_fast_fusion, and external_clients. Holoscan by example walks through them, and the camera + IMU pipeline doubles as this guide’s running example.
• Benchmarks. Microbenchmarks in the source tree’s benchmark/ directory cover the channel layer (POD, schema, and tensor payloads, including interprocess paths), the core runtime, temporal contracts, graph composition, and time; they register under the benchmark CTest label. Measured figures — and the caveats that go with them — live in Channel benchmarks and Core runtime benchmarks.

Where to go next

• Why the framework looks like this — Highlights, then Relevant technologies for the surrounding stack.
• Install the SDK — SDK installation.
• Build a Jetson image with Yocto — Yocto on Jetson.
• Learn the model — Core concepts, then build up to your first pipeline.
• Coming from Holoscan SDK 4.x — Migrating from 4.x.
• Understand the performance story — Performance model.

Recap

• Holoscan SDK 5 is the sensor-processing framework for physical AI: multi-rate sensor pipelines with strict latency bounds, for robotics, medical imaging, and industrial inspection.
• It is layered: the HoloMQ zero-copy transport underneath, an opinionated periodic sensor-processing runtime on top, and separately selectable domain modules — Inference, Vision, and Visualization in 5.x.
• Every application follows compose → compile → run; timing is declared per operator in a temporal contract and validated before the pipeline starts.
• Today the SDK ships a C++ API, three core modules, a graded example set, and channel and runtime benchmarks.
• Next: Highlights.