Introduction

Overview

Holoscan Sensor Bridge provides an FPGA-based interface for low-latency sensor data processing using GPUs. Peripheral device data is acquired by the Holoscan Sensor Bridge device FPGA and sent via UDP over Ethernet to the host system. In systems like IGX Thor or DGX Spark, ConnectX SmartNIC interfaces can greatly lower latency by writing that UDP data directly into GPU memory. Holoscan Sensor Bridge host software supports integrating received sensor data into Holoscan pipelines; examples are provided showing video processing and inference based on a Sony IMX274 camera. This configuration can be connected to IGX Thor, Jetson AGX Orin, Jetson AGX Thor, or DGX Spark compute systems. Additionally:

• Holoscan Sensor Bridge supports hosts with or without accelerated RDMA-capable NICs. For unaccelerated configurations, like Jetson AGX Orin Devkit support is provided using Linux socket-based Ethernet connections.
• Appropriately enabled host systems can support high-accuracy PTP timestamps; these can be used to record the time at which data was received, measure data and pipeline latency, and synchronize sensor behavior. IGX Thor, AGX Orin, AGX Thor, and DGX Spark systems have hardware PTP support in their network interfaces.

Software

Holoscan applications are graphs of operators.

Holoscan Sensor Bridge 3.0 uses Holoscan SDK 5.0.

Holoscan SDK 5.0 is a new, high-performance and modular version of Holoscan SDK, with a completely new runtime and API. It compiles each application graph into an execution plan before the application runs. It runs GPU work in CUDA Graph regions, with a CUDA stream for each partition, and it uses preallocated memory pools. Typed ports and typed parameters help you find connection errors before the application runs. The core, vision, visualization, and inference modules are separate packages, so you install only the modules that you use. Holoscan SDK 5.0 is not backward compatible with Holoscan SDK 4.x. You must port applications from Holoscan SDK 4.x to the new API. For the steps, see the Holoscan SDK 5 migration guide.

In Holoscan SDK 5.0, an application uses three steps:

1. Graph: The application adds operators to a Graph object. It connects an output port of one operator to an input port of another operator with the add_flow API.
2. Compile: The application calls compile with the graph. compile checks the graph and returns an execution plan.
3. Run: The application calls run_async with the execution plan. The connections and the contract of each operator specify when each operator can run.

Holoscan SDK 5 does not use the Application and compose model of Holoscan SDK 4. For the changes, see the migration guides in the Holoscan SDK user guide.

Holoscan Sensor Bridge host software leverages the Holoscan framework by providing operators that receive network data generated by Holoscan Sensor Bridge devices. For example, RoceReceiverOp provides a facility for receiving UDP messages from a Holoscan Sensor Bridge source and storing that data in GPU memory. When used with a video camera source, received data appears as a block of memory with CSI-2 Bayer image data. Holoscan includes off-the-shelf operators for

• Construction of RGBA images from Bayer video
• Image signal processing
• Inference
• Visualization, and
• Data integrity testing

These tools are all intended to facilitate real-time processing of video data. For more details, see IMX274 demo.

Different applications have different uses for received data. For example, a high-speed analog signal (e.g. 5G cellular antenna) has different requirements than a video camera. In general, application code works with APIs on sensor objects to provide appropriate configuration and controls for that device; that sensor object knows how to configure the network receiver operator as needed. If an application or device has requirements that are not met by the current implementation, Holoscan operators are provided as source code: users can adapt these to meet their requirements, and if desired, these modifications can be sent back to NVIDIA for inclusion in future versions of sensor bridge host software.

Applications

Holoscan SDK 5 applications build a graph of operators for their pipeline, compile it into an execution plan and run it. Holoscan Sensor Bridge application pipelines are usually focused on acquiring data from sensor equipment, performing processing on that data, generating appropriate outputs, and transmitting that output data to the appropriate target.

Sensor objects

APIs for configuring and monitoring specific sensors are presented on sensor objects. For example, NativeImx274Sensor is a sensor object with methods for initialization-time configuration, updates to runtime parameters (for example, the exposure), and monitoring device health. Holoscan applications use those APIs to affect device behavior: for example, an “automatic exposure operator” may receive a video frame as input and use that to determine the value sent to the camera’s set_exposure_reg method. Sensor objects are user-mode drivers for specific devices: they provide access to device functionality but do not include application-specific behavior.

Host requirements

Holoscan Sensor Bridge 3.0 uses Holoscan SDK 5.0. The host system must meet these requirements:

• Ubuntu 24.04 (on Jetson and IGX systems, the Ubuntu 24.04 base of JetPack 7.2.1 or IGX SW 2.0.1).
• CUDA 13, with an NVIDIA driver that is compatible with CUDA 13 (R580 or newer).
• Python 3.12 or newer.
• CMake 4.0 or newer, if you build Holoscan Sensor Bridge outside the container.

Holoscan Sensor Bridge 3.0 supports these platforms:

Holoscan SDK 5 does not support IGX Orin (IGX OS 1.x) or JetPack 6. To use these platforms, use Holoscan Sensor Bridge 2.7.

You can run the Holoscan Sensor Bridge software on other systems that NVIDIA Holoscan SDK 5 supports. For the best performance, an IGX Thor system using ConnectX SmartNICs is recommended. In systems without ConnectX NICs, such as the Jetson AGX Orin, performance is limited by the host OS network stack.