Relevant Technologies

Holoscan SDK 5 brings together technologies for GPU-accelerated sensor processing around three goals: predictable performance, developer experience, and agent-assisted development. These technologies help pipelines minimize data movement and meet sensor-to-actuator latency requirements, let developers declare and validate pipeline behavior before execution, and give coding agents machine-readable feedback to check their changes without attached sensors.

Each goal rests on specific technologies. Overview shows how the SDK is layered and Highlights explains what changed in 5.0; this page names what the SDK is built from and why each piece was chosen.

Predictable performance: bounded latency on GPU-attached sensor pipelines

Latency in a sensor pipeline is lost to copies, allocations, host–device synchronization, and scheduling that emerges at run time. Each technology in this section removes one of those costs.

CUDA and stream-ordered execution

Operators can do host-only work or enqueue CUDA work. For operators that enqueue work on the runtime-assigned compute stream, the SDK manages stream ordering: committing a write records readiness, and a received Tensor is already ordered on that stream. Host-only operators run without a CUDA device query or switch. That is what lets a pipeline stay entirely on the device without a cudaStreamSynchronize on the hot path. CCCL provides stream-aware primitives for custom kernels, and CUDA green contexts partition SMs between pipeline stages when one stage must not starve another.

Start with Tensors 101; the stream discipline is in Performance considerations.

HoloMQ and iceoryx2: zero-copy interprocess transport

HoloMQ is the SDK’s typed, GPU-memory-aware transport, built on the iceoryx2 shared-memory publish–subscribe backend. Host, pinned-host, and device tensors can cross process boundaries without a payload copy when their backing is directly exportable and their declared placement permits it. Otherwise, supported paths use bounded staging pools and copy the payload; a path requiring direct export rejects backing it cannot export. Account for staging when measuring cross-process latency.

The same typed emit/receive surface serves in-process and cross-process edges. External processes can join a running pipeline through the standalone channel client API without adopting the rest of the SDK.

See Interprocess channels, Standalone channel clients, and the measured figures in Channel benchmarks.

Preallocated pools and bounded queues

The runtime is designed to keep heap allocation off the steady-state hot path. Tensor slots are borrowed from pools sized before execution, and queue depths are either pinned explicitly or derived at compile time from declared rates. The compile step can establish that pools and queues are bounded and consistent with declared rates; the latency a pipeline achieves on a given platform is still something to measure. Bounded memory is what makes latency predictable on embedded targets and what makes the runtime a tractable substrate for safety-qualified software. The model is described in Performance model and Tensors in depth.

TensorRT inference

Inference is usually the heaviest stage of a pipeline, and NVIDIA TensorRT optimizes trained models for low-latency execution on NVIDIA GPUs. The Holoscan inference module (HoloInfer) builds TensorRT 10 engines from ONNX models, or loads prebuilt engines, and executes them on the runtime’s CUDA streams so inputs and outputs stay on the device.

CV-CUDA and MatX: keeping the work around inference on the GPU

The speedup from accelerated inference is lost if pre- and post-processing fall back to the host. Holoscan Vision provides operators backed by CV-CUDA for resizing, cropping, remapping, normalization, type and layout conversion, and non-maximum suppression, consuming and producing CUDA-resident Holoscan tensors on the runtime stream. Vision also provides NPP-backed BayerDemosaicOp for Bayer-to-RGB or Bayer-to-RGBA conversion.

For numerical work that is not a standard vision primitive, you can use MatX, a C++20 library for GPU numerical computing, alongside the SDK. MatX tensors and holoscan::Tensor exchange data zero-copy through DLPack. The SDK packages do not include MatX: install it or fetch it in your project, then link against the matx::matx target:

target_link_libraries(my_application
  PRIVATE
  holoscan::core
  matx::matx
)

The hello_matx example (examples/hello_matx) shows the pattern: it uses an installed MatX when CMake finds one and otherwise downloads it with FetchContent. Hello tour walks through it.

Vulkan and CUDA interoperability in Holoscan Visualization

Display output is part of the latency budget, and a readback to host memory before rendering would break it. Vulkan, like CUDA, executes on the GPU, so the Holoviz module shares rendering resources between the two without staging through host memory. Image ingestion still copies or converts pixels into renderer-owned storage, and a CUDA output sink copies the rendered image into its output tensor; include those device copies in the latency budget. Resource sharing uses the external resource interoperability functions of the CUDA driver API together with the Vulkan external memory and external semaphore extensions. Shaders are compiled with Slang.

Developer experience: behavior you can read, check, and test without hardware

In earlier releases, timing, types, memory, and failure behavior were emergent properties you discovered on the bench. Holoscan 5 declares them, and the technologies in this section exist to make those declarations checkable early and failures legible.

Modern C++ with typed ports and errors as values

Ports are typed members (holoscan::Input<T> and holoscan::Output<T>), so connecting mismatched types is rejected when the graph is wired. Hot-path APIs return holoscan::expected<T, Error> and are marked [[nodiscard]], so an ignored failure path draws a compiler diagnostic rather than silently misbehaving. The result is that whole classes of mistakes move from the bench to the compiler. See Errors as values and Typed parameters.

The compile step and the execution plan

Every application follows compose → compile → run. compile() represents admission failures as a rejected ExecutionPlan rather than throwing. You can inspect its diagnostics, serialize the plan to JSON, and compare it with a previous plan before execution. compile() can still throw RuntimeError when graph composition cannot produce a coherent snapshot, executable factory reconstruction cannot be retained, or a process resource prevents construction of any plan result. Handle those exceptional boundaries separately from plan admission. See Compile and check and Plan tooling.

Temporal contracts and synthetic clocks

Each operator declares in contract() what triggers it, which input samples participate, and what happens when data is missing. Because timing is declared rather than tuned, it can be unit-tested with synthetic clocks and no sensors attached — you do not need hardware in the loop to know the pipeline runs at the rate you meant. See Temporal contracts, Time and clocks, and Testing.

FlatBuffers payloads and DLPack tensors

Operators, processes, and external tools need one payload vocabulary rather than ad-hoc serialization. Structured payloads are generated from FlatBuffers schemas, so a message’s layout is declared once and shared across process boundaries without a parse step. holoscan::Tensor is built on the DLPack standard, so a tensor is shared without copying with any DLPack-aware library that can address the memory it lives in: device tensors with MatX, CV-CUDA, CuPy, and PyTorch; host tensors with NumPy and PyTorch. See Schema payloads and Tensors in depth.

Structured diagnostics and logging

Compile-time rejections and run-time failures carry Diagnostic records with stable identities and severities, not free-form prose. The logger is built on spdlog and fmt for structured, low-overhead output. See Understanding failures and Logging.

Agent-assisted development: machine-readable APIs and validation

The same properties that make the API verifiable for a person make it usable by a coding agent. Treating agents as first-class users shaped the API; this section reads the technologies above from the agent’s point of view.

• Enumerable, declarative surfaces. Headers, port types, parameter schemas (HOLOSCAN_PARAM_SCHEMA), trigger types, error codes, and payload kinds are finite, closed sets that an agent can inspect directly rather than infer from examples.
• Machine-readable plans and diagnostics. plan.json() and diagnostics() with stable identities let an agent — or a script wrapped around one — pattern-match on which rule fired instead of scraping log text that may change between releases.
• Repeatable checks without sensors. The compile step validates wiring, pool sizes, queue depths, and temporal feasibility on the development machine, and synthetic clocks let operator timing be unit-tested there too. A rejected plan is ground truth about what to fix. Confirming throughput and latency on the target GPU still requires running on that hardware.

How to set up and get the most out of agent-assisted Holoscan development is covered in AI coding agents.

For readers coming from Holoscan SDK 4 or earlier

Several technologies that earlier releases were built on are no longer part of the stack and have direct replacements.

GXF

Holoscan 4 and earlier were built on GXF (Graph Execution Framework), an NVIDIA entity-component runtime with its own schedulers, allocators, and YAML graph format. Holoscan 5 runs on a native runtime and a compiled execution plan; there is no GXF underneath.

The mapping below helps interpret older code and documentation:

The full walk-through is in Migrating from 4.x.

UCX and fragments

Distributed 4.x applications split into fragments connected over UCX, which could span processes on one host or on several. In Holoscan 5, cross-process edges are HoloMQ channels over shared memory, so they connect processes on the same host, and operator code is the same on either side of the process boundary. Communication between hosts is not part of the runtime; a process that bridges a channel to a network protocol is written with the standalone channel client API.

NPP-based image operators

The 4.x FormatConverter and BayerDemosaic operators used NVIDIA Performance Primitives (NPP). In Holoscan 5, Holoscan Vision provides BayerDemosaicOp, also backed by NPP, for Bayer-to-RGB or Bayer-to-RGBA conversion. Its CV-CUDA-backed ConvertToOp changes element type with scaling, and ReformatOp changes image layout. These cover parts of the legacy FormatConverter workflow; they do not perform color-space conversion. Check the Vision operator’s supported formats and parameters when migrating each stage of an existing pipeline.

Optional platform integrations

Rivermax and GPUDirect RDMA

Rivermax and GPUDirect RDMA are not dependencies of the SDK. On systems with a ConnectX network adapter, GPUDirect RDMA lets network and sensor-bridge hardware write directly into GPU memory, which pairs well with the zero-copy pipeline described above. Enabling it is a platform setup step; see GPUDirect RDMA and Sensor bridge.

Where each technology lives

Not every technology on this page is part of every installation. The table below summarizes what each one does and how it reaches your application.

Recap

• Predictable performance comes from staying on the device: CUDA stream ordering, HoloMQ transport on iceoryx2 with direct export or bounded staging, preallocated pools, TensorRT inference, CV-CUDA, NPP, and MatX processing, and Vulkan–CUDA interop for display.
• Developer experience comes from declaring behavior and checking it early: typed ports and errors as values, the compile step and execution plan, temporal contracts with synthetic clocks, FlatBuffers and DLPack payloads, and structured diagnostics.
• Agent-assisted development falls out of the same design: enumerable surfaces, machine-readable plans, and checks that run without sensors.
• GXF and UCX fragments are 4.x-era technologies with replacements in Holoscan 5. Vision provides NPP-backed Bayer demosaic and selected CV-CUDA image operations; map legacy format-conversion stages individually. GPUDirect RDMA remains an optional platform integration.

Next: API stability.