Operators

Video I/O Vendor Implementation Guide

View as Markdown

This guide describes how capture-card vendors subclass the Holoscan SDK VideoAcquisitionOperator and VideoTransmissionOperator to provide SDK-native multi-stream video capture and playout.

Architecture Overview

┌──────────────────────────────────────────────────────┐
│                   Application                        │
│  F.make_operator<ExampleVideoCapture>("cap", 4U,     │
│                   Arg("uri", "sdi://0"), ...);       │
└──────────────────────┬───────────────────────────────┘
                       │
        ┌──────────────▼──────────────────────┐
        │   VideoAcquisitionOperator (base)   │
        │   ─ setup(): registers output ports │
        │   ─ emit_capture_stream()           │
        │   ─ query_capture_capabilities()    │
        │   ─ note_acquired_frame() telemetry │
        └──────────────┬──────────────────────┘
                       │ subclass
        ┌──────────────▼──────────────────────┐
        │   ExampleVideoCapture (vendor)      │
        │   ─ start(): open device, alloc DMA │
        │   ─ compute(): dequeue + emit       │
        │   ─ stop(): release device          │
        │   ─ query_capture_capabilities()    │
        └─────────────────────────────────────┘

The base class handles:

ConcernBase provides
Port registrationsetup() registers signal, signal_1, … signal_N
Port name managementcapture_output_port_name() / transmit_input_port_name()
Emitting / receiving framesemit_capture_stream() / receive_transmit_stream()
Common parametersuri, width, height, frame_rate, pixel_format, color_space, transport, vendor_extensions
Capability reportingDefault query_capture_capabilities() from parameters
Telemetry countersnote_acquired_frame(), note_dropped_frame() (atomic)
Stream validationis_capture_stream_enabled(), bounds checking

The vendor subclass provides:

ConcernVendor implements
Device lifecyclestart(), stop()
Frame production / consumptioncompute()
Hardware capability queryquery_capture_capabilities() (optional override)
Vendor-specific parametersAdditional spec.param() calls in setup()
Registry enumeratorregister_video_acquisition_enumerator() (optional)

Step 1: Subclass the Base Operator

Acquisition (Capture)

1#pragma once
2 
3#include <holoscan/operators/video_io/video_acquisition_operator.hpp>
4 
5namespace example {
6 
7class ExampleVideoCapture : public holoscan::ops::VideoAcquisitionOperator {
8 public:
9  // Required: forward the default and num_streams constructors to the base.
10  HOLOSCAN_OPERATOR_FORWARD_ARGS_SUPER(ExampleVideoCapture, VideoAcquisitionOperator)
11 
12  ExampleVideoCapture() = default;
13 
14  explicit ExampleVideoCapture(uint32_t num_streams)
15      : VideoAcquisitionOperator(num_streams) {}
16 
17  // Required if you accept num_streams + Args (the common make_operator pattern).
18  HOLOSCAN_OPERATOR_FORWARD_TEMPLATE()
19  explicit ExampleVideoCapture(uint32_t num_streams, ArgT&& arg, ArgsT&&... args)
20      : VideoAcquisitionOperator(num_streams) {
21    add_arg(std::forward<ArgT>(arg));
22    (add_arg(std::forward<ArgsT>(args)), ...);
23  }
24 
25  void setup(OperatorSpec& spec) override;
26  void start() override;
27  void compute(InputContext& op_input, OutputContext& op_output,
28               ExecutionContext& context) override;
29  void stop() override;
30 
31  video_io::VideoCaptureCapabilities query_capture_capabilities() const override;
32 
33 private:
34  // Vendor SDK handle, DMA buffers, etc.
35  Parameter<uint32_t> dma_buffer_count_;
36  Parameter<bool> gpudirect_enabled_;
37 
38  void* device_handle_ = nullptr;
39};
40 
41}  // namespace example

Transmission (Playout)

1#pragma once
2 
3#include <holoscan/operators/video_io/video_transmission_operator.hpp>
4 
5namespace example {
6 
7class ExampleVideoPlayout : public holoscan::ops::VideoTransmissionOperator {
8 public:
9  HOLOSCAN_OPERATOR_FORWARD_ARGS_SUPER(ExampleVideoPlayout, VideoTransmissionOperator)
10 
11  ExampleVideoPlayout() = default;
12 
13  explicit ExampleVideoPlayout(uint32_t num_streams)
14      : VideoTransmissionOperator(num_streams) {}
15 
16  HOLOSCAN_OPERATOR_FORWARD_TEMPLATE()
17  explicit ExampleVideoPlayout(uint32_t num_streams, ArgT&& arg, ArgsT&&... args)
18      : VideoTransmissionOperator(num_streams) {
19    add_arg(std::forward<ArgT>(arg));
20    (add_arg(std::forward<ArgsT>(args)), ...);
21  }
22 
23  void setup(OperatorSpec& spec) override;
24  void start() override;
25  void compute(InputContext& op_input, OutputContext& op_output,
26               ExecutionContext& context) override;
27  void stop() override;
28 
29 private:
30  Parameter<std::string> output_standard_;
31  void* device_handle_ = nullptr;
32};
33 
34}  // namespace example

Step 2: Implement setup() — Register Vendor Parameters

Call the base setup() first. This registers all common parameters (uri, width, etc.) and the correct number of I/O ports based on num_streams(). Then add your vendor-specific parameters.

1void ExampleVideoCapture::setup(OperatorSpec& spec) {
2  // Base registers: output ports (signal, signal_1, ...), uri, width, height,
3  // frame_rate, pixel_format, color_space, transport, vendor_extensions, etc.
4  VideoAcquisitionOperator::setup(spec);
5 
6  // Vendor-specific parameters
7  spec.param(dma_buffer_count_,
8             "dma_buffer_count",
9             "DMA Buffer Count",
10             "Number of DMA ring buffers per channel.",
11             4U);
12  spec.param(gpudirect_enabled_,
13             "gpudirect_enabled",
14             "GPUDirect RDMA",
15             "Enable GPUDirect RDMA if supported.",
16             false);
17}

For transmission:

1void ExampleVideoPlayout::setup(OperatorSpec& spec) {
2  VideoTransmissionOperator::setup(spec);
3 
4  spec.param(output_standard_,
5             "output_standard",
6             "Output Standard",
7             "Video standard for output (e.g. 1080p60, 2160p30).",
8             std::string("1080p60"));
9}

Step 3: Implement Lifecycle — start() and stop()

start() is called once after initialize(). Open the device, allocate buffers, configure the hardware. stop() tears everything down.

1void ExampleVideoCapture::start() {
2  const std::string device_uri = uri_.get();
3  const uint32_t w = width_.get();
4  const uint32_t h = height_.get();
5  const float fps = frame_rate_.get();
6  const uint32_t buf_count = dma_buffer_count_.get();
7 
8  // Open vendor device (pseudo-code)
9  device_handle_ = example_sdk_open(device_uri.c_str());
10  if (!device_handle_) {
11    throw std::runtime_error("Failed to open device: " + device_uri);
12  }
13 
14  // Configure each stream (channel)
15  for (uint32_t i = 0; i < num_streams(); ++i) {
16    example_sdk_configure_channel(device_handle_, i, w, h, fps, buf_count);
17  }
18 
19  example_sdk_start_capture(device_handle_);
20}
21 
22void ExampleVideoCapture::stop() {
23  if (device_handle_) {
24    example_sdk_stop_capture(device_handle_);
25    example_sdk_close(device_handle_);
26    device_handle_ = nullptr;
27  }
28}

Step 4: Implement compute() — The Frame Loop

compute() is called by the scheduler on every tick. For acquisition, dequeue frames from the hardware and emit them via emit_capture_stream(). For transmission, receive frames via receive_transmit_stream() and queue them to the hardware.

Acquisition compute()

1void ExampleVideoCapture::compute(InputContext& op_input, OutputContext& op_output,
2                                ExecutionContext& context) {
3  for (uint32_t i = 0; i < num_streams(); ++i) {
4    if (!is_capture_stream_enabled(i)) {
5      continue;
6    }
7 
8    void* frame_ptr = nullptr;
9    size_t frame_size = 0;
10    int status = example_sdk_dequeue_frame(device_handle_, i, &frame_ptr, &frame_size);
11 
12    if (status == EXAMPLE_TIMEOUT) {
13      note_dropped_frame();
14      continue;
15    }
16    if (status != EXAMPLE_OK) {
17      note_dropped_frame();
18      HOLOSCAN_LOG_ERROR("Example dequeue failed on channel {}: {}", i, status);
19      continue;
20    }
21 
22    // Wrap the vendor buffer in a GXF Entity with a VideoBuffer or Tensor
23    auto entity = holoscan::gxf::Entity::New(&context);
24    // ... populate entity with frame data (vendor-specific) ...
25    // For GPUDirect: frame_ptr is already a device pointer
26    // For CPU DMA: memcpy or cudaMemcpyAsync to a device buffer
27 
28    emit_capture_stream(op_output, i, entity);
29    note_acquired_frame();
30 
31    // Return the buffer to the vendor SDK ring
32    example_sdk_requeue_buffer(device_handle_, i, frame_ptr);
33  }
34}

Transmission compute()

1void ExampleVideoPlayout::compute(InputContext& op_input, OutputContext& op_output,
2                                ExecutionContext& context) {
3  for (uint32_t i = 0; i < num_streams(); ++i) {
4    if (!is_transmit_stream_enabled(i)) {
5      continue;
6    }
7 
8    auto entity = receive_transmit_stream(op_input, i);
9    if (!entity) {
10      note_dropped_frame();
11      continue;
12    }
13 
14    // Extract the video buffer from the entity (vendor-specific)
15    // ... get pointer, size, format from the entity's VideoBuffer/Tensor ...
16 
17    int status = example_sdk_queue_output(device_handle_, i, gpu_ptr, frame_size);
18    if (status != EXAMPLE_OK) {
19      note_dropped_frame();
20      HOLOSCAN_LOG_ERROR("Example output queue failed on channel {}: {}", i, status);
21      continue;
22    }
23 
24    note_transmitted_frame();
25  }
26}

The default query_capture_capabilities() derives a minimal snapshot from configured parameters. Override it to query the actual hardware for supported modes, resolutions, frame rates, and features.

1video_io::VideoCaptureCapabilities ExampleVideoCapture::query_capture_capabilities() const {
2  video_io::VideoCaptureCapabilities cap;
3  cap.backend_id = "vendor.example";
4  cap.device_id = uri_.get();
5  cap.device_uri = uri_.get();
6 
7  // Query hardware for actual capabilities
8  int num_inputs = example_sdk_get_input_count(device_handle_);
9  cap.max_concurrent_inputs = static_cast<uint32_t>(num_inputs);
10  cap.transports.push_back(video_io::VideoTransport::kSdi);
11 
12  for (int ch = 0; ch < num_inputs; ++ch) {
13    video_io::VideoCaptureChannelCapabilities chan;
14    chan.channel_index = static_cast<uint32_t>(ch);
15    chan.transport = video_io::VideoTransport::kSdi;
16    chan.interface_label = "SDI In " + std::to_string(ch + 1);
17 
18    // Query supported resolutions
19    int num_modes = 0;
20    example_mode_t* modes = example_sdk_get_modes(device_handle_, ch, &num_modes);
21    for (int m = 0; m < num_modes; ++m) {
22      chan.resolutions.push_back({modes[m].width, modes[m].height});
23      chan.framerates.push_back({modes[m].min_fps, modes[m].max_fps});
24    }
25 
26    // Query pixel formats
27    int num_fmts = 0;
28    example_pixfmt_t* fmts = example_sdk_get_pixel_formats(device_handle_, ch, &num_fmts);
29    for (int f = 0; f < num_fmts; ++f) {
30      chan.pixel_formats.push_back({fmts[f].fourcc, fmts[f].description});
31    }
32 
33    chan.color_spaces.push_back(video_io::VideoColorSpaceKind::kBt709);
34    chan.color_spaces.push_back(video_io::VideoColorSpaceKind::kBt2020);
35    chan.gpudirect_rdma_supported = example_sdk_supports_gpudirect(device_handle_, ch);
36    chan.hardware_timestamp_supported = true;
37    chan.progressive_capture_supported = true;
38    chan.interlaced_capture_supported = true;
39 
40    cap.input_channels.push_back(std::move(chan));
41  }
42 
43  return cap;
44}

Step 6: Register a Capability Enumerator (Optional)

The registry allows applications to discover available hardware before creating operator instances. Register an enumerator at library load time (or from a static initializer) using your vendor backend_id.

1#include <holoscan/operators/video_io/video_io_registry.hpp>
2 
3namespace {
4 
5struct ExampleRegistrar {
6  ExampleRegistrar() {
7    holoscan::ops::video_io::register_video_acquisition_enumerator(
8        "vendor.example",
9        []() -> std::vector<holoscan::ops::video_io::VideoCaptureCapabilities> {
10          std::vector<holoscan::ops::video_io::VideoCaptureCapabilities> result;
11 
12          int num_devices = example_sdk_enumerate_devices();
13          for (int d = 0; d < num_devices; ++d) {
14            holoscan::ops::video_io::VideoCaptureCapabilities cap;
15            cap.backend_id = "vendor.example";
16            cap.device_id = example_sdk_get_device_serial(d);
17            cap.device_uri = "example://" + std::to_string(d);
18            cap.max_concurrent_inputs = example_sdk_get_input_count_by_index(d);
19            cap.transports.push_back(holoscan::ops::video_io::VideoTransport::kSdi);
20 
21            for (uint32_t ch = 0; ch < cap.max_concurrent_inputs; ++ch) {
22              holoscan::ops::video_io::VideoCaptureChannelCapabilities chan;
23              chan.channel_index = ch;
24              chan.transport = holoscan::ops::video_io::VideoTransport::kSdi;
25              chan.interface_label = "SDI In " + std::to_string(ch + 1);
26              cap.input_channels.push_back(std::move(chan));
27            }
28 
29            result.push_back(std::move(cap));
30          }
31          return result;
32        });
33  }
34};
35 
36static ExampleRegistrar s_registrar;
37 
38}  // namespace

Applications can then discover devices without instantiating any operator:

1auto devices = holoscan::ops::video_io::enumerate_video_acquisition_devices("vendor.example");
2for (const auto& dev : devices) {
3  HOLOSCAN_LOG_INFO("Found device {} with {} inputs",
4                    dev.device_id, dev.max_concurrent_inputs);
5}

Step 7: Wire Into an Application

Multi-Instance (Preferred — One Operator Per Channel)

The recommended deployment pattern: each operator instance owns one hardware channel. A shared device resource (vendor-specific) ensures safe access to the underlying SDK handle with per-channel reservation.

1// Shared device resource — single SDK handle with per-channel reservation
2auto dev = F.make_resource<example::ExampleDeviceResource>(
3    "example_dev0", Arg("device", "0"));
4 
5// One operator per channel on the same physical device
6auto ch0 = F.make_operator<example::ExampleVideoCapture>(
7    "sdi_ch0",
8    Arg("device_resource", dev),
9    Arg("channel_index", 0U),
10    Arg("uri", "sdi://0"),
11    Arg("width", 1920U),
12    Arg("height", 1080U),
13    Arg("frame_rate", 60.F),
14    Arg("rdma", true));
15 
16auto ch1 = F.make_operator<example::ExampleVideoCapture>(
17    "sdi_ch1",
18    Arg("device_resource", dev),
19    Arg("channel_index", 1U),
20    Arg("uri", "sdi://0"),
21    Arg("width", 1920U),
22    Arg("height", 1080U),
23    Arg("frame_rate", 60.F),
24    Arg("rdma", true));
25 
26auto proc0 = F.make_operator<InferenceOp>("proc0");
27auto proc1 = F.make_operator<InferenceOp>("proc1");
28 
29F.add_flow(ch0, proc0, {{"signal", "input"}});
30F.add_flow(ch1, proc1, {{"signal", "input"}});

Multi-Stream (Single Operator, Multiple Ports)

When the vendor SDK manages multiple channels through a single handle and does not support independent per-channel initialization, use one operator with num_streams > 1 to expose each channel on a separate output port.

1// 4 output ports: signal, signal_1, signal_2, signal_3
2auto capture = F.make_operator<example::ExampleVideoCapture>(
3    "quad_sdi",
4    4U,                              // <-- num_streams
5    Arg("uri", "sdi://0"),
6    Arg("transport", "sdi"),
7    Arg("width", 1920U),
8    Arg("height", 1080U),
9    Arg("frame_rate", 60.F));
10 
11auto proc0 = F.make_operator<InferenceOp>("proc0");
12auto proc1 = F.make_operator<InferenceOp>("proc1");
13auto proc2 = F.make_operator<InferenceOp>("proc2");
14auto proc3 = F.make_operator<InferenceOp>("proc3");
15 
16F.add_flow(capture, proc0, {{"signal", "input"}});
17F.add_flow(capture, proc1, {{"signal_1", "input"}});
18F.add_flow(capture, proc2, {{"signal_2", "input"}});
19F.add_flow(capture, proc3, {{"signal_3", "input"}});

YAML Configuration

1quad_sdi:
2  uri: "sdi://0"
3  transport: "sdi"
4  width: 1920
5  height: 1080
6  frame_rate: 60.0
7  pixel_format: "UYVY"
8  color_space: "bt709"
9  dma_buffer_count: 8
10  gpudirect_enabled: true
11  vendor_extensions:
12    vendor.example.genlock_source: "ref_in"
13    vendor.example.anc_capture: true

Port Naming Convention

num_streamsRegistered portsNotes
1 (default)signalBackward compatible with V4L2VideoCaptureOp
2signal, signal_1Index 0 is always signal, not signal_0
4signal, signal_1, signal_2, signal_3
Nsignal, signal_1, …, signal_{N-1}Max N = 128 (kVideoIoMaxStreams)

Use capture_output_port_name(i) / transmit_input_port_name(i) if you need the string programmatically. The base caches these in a vector for zero-allocation lookup in the hot path (emit_capture_stream / receive_transmit_stream).

num_streams vs. channel_indices — When to Use Which

These serve different purposes:

ParameterControlsSet by
num_streamsHow many I/O ports setup() registersConstructor argument
channel_indicesWhich hardware channels appear in capability reportsYAML / Arg()

Common patterns:

Scenarionum_streamschannel_indicesExplanation
Single SDI capture1{} (empty)One port, one default channel
4x SDI via single SDK handle4{0,1,2,3}Four ports, four reported channels
Singleton SDK that internally muxes 4 channels onto 1 output1{0,1,2,3}One port, but capability report shows all channels the SDK manages
4 separate operator instances, one per channel1 (each){} (each)Each operator uses channel_index instead

CMake Integration

1# Vendor operator library
2add_library(example_video_capture
3  example_video_capture.cpp
4)
5target_link_libraries(example_video_capture
6  PUBLIC
7    holoscan::core
8    holoscan::ops::video_io     # base classes + capabilities + registry
9  PRIVATE
10    example::sdk                # vendor SDK
11)

Checklist

  • Subclass VideoAcquisitionOperator and/or VideoTransmissionOperator
  • Forward all three constructor forms (default, num_streams, num_streams + args)
  • Call base setup() first in your override, then add vendor parameters
  • Implement compute() — use emit_capture_stream() / receive_transmit_stream()
  • Call note_acquired_frame() / note_transmitted_frame() on success
  • Call note_dropped_frame() on timeout or error
  • Implement start() to open device, stop() to close device
  • Override query_capture_capabilities() to return real hardware capabilities
  • Use backend_id = "vendor.&lt;yourname&gt;" for capability reporting
  • Register a capability enumerator for device discovery (optional)
  • Link against holoscan::ops::video_io
  • Test with num_streams = 1 (backward compat) and > 1 (multi-stream)
  • Validate port naming: signal, signal_1, …, signal_{N-1}

API Reference (Quick Summary)

VideoAcquisitionOperator (protected helpers)

MethodPurpose
emit_capture_stream(op_output, stream_index, entity)Emit a frame on port stream_index
capture_output_port_name(stream_index)Get port name string for index
note_acquired_frame()Increment acquired counter (atomic)
note_dropped_frame()Increment dropped counter (atomic)
is_capture_stream_enabled(stream_index)Check if index < num_streams()
build_capture_capabilities_from_parameters()Default capability snapshot from params

VideoTransmissionOperator (protected helpers)

MethodPurpose
receive_transmit_stream(op_input, stream_index)Receive a frame from port stream_index
transmit_input_port_name(stream_index)Get port name string for index
note_transmitted_frame()Increment transmitted counter (atomic)
note_dropped_frame()Increment dropped counter (atomic)
is_transmit_stream_enabled(stream_index)Check if index < num_streams()

Common Parameters (inherited from base setup())

ParameterTypeDefaultDescription
backend_idstring"generic"Vendor identifier for capability reporting
channel_indexuint32_t0Zero-based channel for single-channel mode
channel_indicesvector<uint32_t>{}Multi-channel index list
uristring""Device path, index, or stream URI
widthuint32_t0Requested width (0 = device default)
heightuint32_t0Requested height (0 = device default)
frame_ratefloat0.0Requested FPS (0 = device default)
pixel_formatstring"auto"Pixel format label or fourcc
color_spacestring"auto"Color space hint
transportstring"auto"Transport hint (sdi, hdmi, ethernet, …)
vendor_extensionsYAML::Node{}Arbitrary vendor key-value pairs