Vision

Holoscan Vision is the core module for GPU-resident image and detection pipelines. Eight operators use CV-CUDA, while BayerDemosaicOp uses NPP because CV-CUDA has no Bayer demosaic operation. All nine use the Holoscan 5.x programming model: typed ports, temporal contracts, bounded output pools, typed image payloads, and explicit image metadata transformations.

Operators

All operators live in the holoscan::vision namespace. Image operators use Input<schema::ImageT> and Output<schema::ImageT> ports named input and output. Their Image.data field holds the CUDA tensor. RemapOp also has a map input carrying Tensor; NMS uses Tensor for boxes, scores, and mask. The eight image operators have no Tensor compatibility ports or payload template variants. Names such as CustomCropOp, CopyMakeBorderOp, ConvertToOp, and ReformatOp deliberately retain their CV-CUDA operation names so developers can find the corresponding backend documentation and behavior directly.

Unary operators process each input sample independently. By default, RemapOpParams::map_synchronization is RemapMapSynchronizationMode::kLatest: every image anchors an activation, and the operator selects the newest causal map within map_max_age. A selected map can be reused for later images within the same age bound. The default maximum age permits run-lifetime reuse, which suits static lens and rectification maps.

Image and map capture times must use the same clock domain for causal comparison. Publish the first eligible map before starting the image stream; images that arrive without an eligible map do not run later when a map becomes available. For frame-varying maps that require exact pairing, set map_synchronization to RemapMapSynchronizationMode::kExact. Exact mode uses a zero-width alignment window and requires identical image and map capture times. NonMaximumSuppressionOp always applies that exact requirement to boxes and scores.

Tensor contract

Image inputs use rank-3 HWC or rank-4 NHWC layouts; BayerDemosaicOp accepts only rank-3 HWC. ReformatOp additionally accepts CHW and NCHW inputs and produces the corresponding target layout. Image.width and Image.height must match tensor dimensions. The image encoding must agree with the tensor’s dtype, channels, and layout, and supplied plane descriptors must agree with its strides. CUSTOM encodings are accepted when the scalar storage satisfies the operator’s requirements.

Packed and chroma-subsampled encodings are unsupported and are rejected rather than reinterpreted as scalar image data.

The following table is the accepted matrix for scalar, byte-addressable DLPack element types:

NonMaximumSuppressionOp accepts S16 boxes shaped [N, W, 4] and F32 scores shaped [N, W]. It emits a contiguous U8 selection mask shaped [N, W]. Each box is encoded as (x, y, width, height).

The following storage rules apply:

• Interleaved inputs have tightly packed channels and pixels. Row and batch pitch may include padding.
• Planar ReformatOp inputs have tightly packed pixels. Row, channel, and batch pitch may include padding.
• NMS boxes and scores may use row pitch, but coordinates and scores are tightly packed within a row. Boxes require an 8-byte-aligned base address and batch pitch; scores require 4-byte alignment for both.
• Vision outputs use contiguous storage.
• Every byte stride and each tensor’s total strided byte span must not exceed INT32_MAX, as required by the underlying CV-CUDA operators.
• Every tensor participating in one invocation uses the same CUDA device.

A two-channel F32 RemapOp map may have a batch size of one or the image batch size. Its spatial dimensions may differ from both the input and output image: CV-CUDA samples the map over the destination grid when producing the configured output geometry. Image and detection batch capacities cannot exceed holoscan::vision::kMaximumBatchSize (65,535).

VisionElementType, used by ConvertToOpParams::output_type, maps to DLPack as follows. Every Vision dtype is scalar (lanes = 1).

NormalizeOp preserves the input dtype. To produce F32 values from a U8 image, place ConvertToOp configured with VisionElementType::kF32 before NormalizeOp; normalization alone does not cast the tensor.

For more about Holoscan tensors and output allocation, see Tensors in depth.

Bayer demosaic

BayerDemosaicOp uses NPP’s CFA-to-RGB(A) kernels because CV-CUDA has no corresponding Bayer demosaic operation. Its ImageT input carries a CUDA-device U8 or U16 tensor shaped [H, W, 1], with even width and height and tightly packed samples. Row pitch may include padding. The image dimensions, encoding, and supplied plane records must agree with the tensor.

bayer_grid_pos declares the CFA phase at the input ROI origin and defaults to kGBRG. Each named BAYER_RGGB, BAYER_BGGR, BAYER_GRBG, or BAYER_GBRG encoding must match the corresponding parameter value; a contradiction returns kInvalidArgument before allocation. For CUSTOM, MONO8, or MONO16, the parameter supplies the otherwise unnamed Bayer phase. A producer using these encodings must supply mosaic samples rather than an already reconstructed image.

Output has the same dimensions and element type, with contiguous RGB storage or RGBA when generate_alpha is true. alpha_value is unscaled: its default 255 remains 255 in U16 output; use 65535 for full-scale U16 alpha. U8 RGBA rejects values above 255 before allocation. When alpha generation is disabled, alpha_value is unused.

Output encoding is RGB8/RGBA8 or RGB16/RGBA16. ROI offsets, image header, exposure, gain, sample metadata, and provenance are preserved. color_space is cleared and significant_bits becomes unknown because interpolation changes sample values. Plane layout is rebuilt for the output tensor, and publication attaches its committed runtime-pool loan without copying pixels.

Execution and metadata

CV-CUDA operators wrap the existing input storage and the plan-owned output tensor loan as non-owning NVCV tensors; Bayer passes their pointers directly to NPP. Neither path stages pixel storage or synchronizes the host. Backend work uses the CUDA stream supplied by the operator’s ExecutionContext.

Unary outputs preserve capture time, frame identity, trace identity, and sample flags. RemapOp treats its map as configuration: it takes capture time, frame identity, and trace identity from the image, and unions image and map flags, including when a retained map is reused. NMS combines both boxes and scores: it unions flags, keeps frame identity only on agreement, retires conflicting trace roots, and preserves their common capture time. Mixed clock domains are rejected. Image outputs also preserve the image header, exposure, and gain. output.emit_tensor attaches the committed runtime-pool loan to the output image without copying pixels.

Image descriptors track the actual output geometry and storage. Plane offsets and row strides are rebuilt from output tensor strides. For NHWC/NCHW batches, the planes describe the first image; the tensor batch stride repeats that layout for the remaining images.

Geometric operations preserve color space. Unknown precision is represented by schema::kUnknownImageSignificantBits (255); zero still means full container width. An unscaled lossless cast retains source precision, expanding zero to the input container width when the dtype changes. For example, a 10-bit MONO16 image cast to F32 still reports 10 significant bits. This field does not specify a code range or valid-bit alignment: casting does not rescale white, shift bits, or normalize integer values to [0, 1]. Consumers must handle unknown precision explicitly instead of inferring it from the output dtype.

Author a graph

Vision operators are default-constructible. Their typed Parameters provide the structural fields used to derive fixed output-pool capacity. ResizeOp, CustomCropOp, and RemapOp use their fixed output dimensions directly; NormalizeOp and ReformatOp inherit their connected input’s maximum byte span, and ConvertToOp scales that input bound for its selected output dtype. CopyMakeBorderOp requires maximum output dimensions, while NMS requires a maximum box count. Complete configuration loading also requires every capacity-only maximum, including max_batch_size; direct C++ aggregate setup should set these fields to the application’s real bound instead of relying on their single-sample defaults. This example omits the source and sink operators:

#include <utility>
#include <holoscan/core/compile.hpp>
#include <holoscan/core/graph.hpp>
#include <holoscan/vision/copy_make_border_op.hpp>
#include <holoscan/vision/custom_crop_op.hpp>
#include <holoscan/vision/resize_op.hpp>
holoscan::Graph graph{"vision"};
auto resize = graph.op<holoscan::vision::ResizeOp>("resize");
resize.set_params(holoscan::vision::ResizeOpParams{
    .width = 1280,
    .height = 720,
    .max_batch_size = 4U,
    .interpolation = holoscan::vision::InterpolationMode::kLinear,
});
auto crop = graph.op<holoscan::vision::CustomCropOp>("crop");
crop.set_params(holoscan::vision::CustomCropOpParams{
    .x = 64,
    .y = 32,
    .width = 640,
    .height = 480,
    .max_batch_size = 4U,
});
auto border = graph.op<holoscan::vision::CopyMakeBorderOp>("border");
border.set_params(holoscan::vision::CopyMakeBorderOpParams{
    .max_width = 672,
    .max_height = 512,
    .max_batch_size = 4U,
    .top = 16,
    .bottom = 16,
    .left = 16,
    .right = 16,
});
graph.add_flow(resize->output, crop->input);
graph.add_flow(crop->output, border->input);

Python uses schema-derived operator proxies from the separately packaged holoscan_vision module. The Bayer phase accepts a stable name or its numeric value, and both ports carry holoscan.schema.Image:

import holoscan as hs
from holoscan_vision import operators as vision_ops
graph = hs.Graph("bayer")
demosaic = vision_ops.BayerDemosaicOp(
    graph,
    "demosaic",
    bayer_grid_pos="rggb",
    generate_alpha=True,
    alpha_value=255,
)

The other accepted names are "bggr", "gbrg", and "grbg"; numeric values 0 through 3 are equivalent. Invalid phases and values outside the uint16 range for alpha_value are rejected before the operator is attached to the graph. The narrower limit depends on the input: when generate_alpha=True, compute accepts the full range for U16 input but rejects values above 255 for U8 input before allocating the output.

Bind every runtime-pool CUDA tensor producer to a deployment device, including Vision outputs. This is a general Holoscan 5 requirement: the compiler does not guess device zero, and compilation fails if a CUDA pool output has no resolved device:

holoscan::CompileOptions options;
options.deployment
    .bind_tensor_output_device(holoscan::TensorOutputDevicePlacement{
        .operator_path = "resize",
        .output_port = "output",
        .device = holoscan::DeviceId{0},
    })
    .bind_tensor_output_device(holoscan::TensorOutputDevicePlacement{
        .operator_path = "crop",
        .output_port = "output",
        .device = holoscan::DeviceId{0},
    })
    .bind_tensor_output_device(holoscan::TensorOutputDevicePlacement{
        .operator_path = "border",
        .output_port = "output",
        .device = holoscan::DeviceId{0},
    });
const holoscan::ExecutionPlan plan = holoscan::compile(graph, std::move(options));

See Graph composition in depth for complete graph wiring. The source tree includes a complete image-processing and detection example at modules/vision/examples/vision_pipeline/cpp.

Build, test, and consume

Build and test through the SDK-level runner from the repository root:

./run build --config \
  "-DHOLOSCAN_BUILD_VISION=ON -DHOLOSCAN_BUILD_EXAMPLES=ON -DHOLOSCAN_BUILD_TESTS=ON"
./run test --options "-L vision"

The SDK development image provides CV-CUDA and NPP development files for shared builds. GPU tests report a CTest skip when no usable CUDA device is present.

Installed CMake consumers use the exported holoscan::vision target:

find_package(holoscan COMPONENTS vision REQUIRED)
target_link_libraries(my_pipeline PRIVATE holoscan::vision)

Standalone consumers can instead use:

find_package(holoscan-vision REQUIRED)
target_link_libraries(my_pipeline PRIVATE holoscan::vision)

Set holoscan-vision_DIR to <prefix>/lib/cmake/holoscan when the SDK is outside CMake’s default search locations. The Debian packages install it in /usr/lib/<triplet>/cmake/holoscan, which CMake searches by default.

HOLOSCAN_BUILD_VISION defaults to ON. CPack Debian generation packages Vision as the holoscan-vision payload by default, which depends on the matching holoscan-core version. Set HOLOSCAN_BUILD_VISION=OFF to omit the module from a source build.

Holoscan Vision uses CV-CUDA internally without exposing it in public headers. Published Vision Debian packages bundle the Vision shared library and the required CV-CUDA runtime libraries. To develop with CV-CUDA directly, install the matching Debian packages for your architecture and CUDA version from the CV-CUDA release assets.

Bayer demosaic also requires NPP, declared as a dependency of the Vision Debian package and Python wheel.

Recap

• Holoscan Vision provides nine C++ operators backed by CV-CUDA and NPP for Bayer demosaic, image transformation, normalization, format conversion, and NMS.
• Image payloads carry validated descriptors and CUDA tensors. Work runs on the HSDK runtime stream and outputs use bounded, plan-owned tensor pools.
• The accepted layout, channel, dtype, pitch, and device rules are explicit; validate producers against the tensor matrix before composing a pipeline.
• RemapOp reuses a causal map by default; configure map_max_age for the required freshness or select kExact for timestamp-paired maps.
• Build through ./run, or consume the installed holoscan::vision CMake target through the vision component of the holoscan package.