Holoscan Sensor Bridge Introduction

Holoscan Sensor Bridge provides an FPGA based interface for low-latency sensor data processing using GPUs. Peripheral device data is acquired by the Holoscan Sensor Bridge device FPGA and sent via UDP over Ethernet to the host system. In systems like IGX Devkit, ConnectX SmartNIC interfaces can greatly lower latency by writing that UDP data directly into GPU memory. Holoscan Sensor Bridge also supports the Jetson AGX Orin Devkit using socket based Ethernet connections. Holoscan Sensor Bridge host software supports integrating sensor equipment into Holoscan pipelines. Several examples are provided showing video processing and inference based on a Sony IMX274 camera attached to a Holoscan Sensor Bridge circuit board. This configuration can be connected to an IGX or Jetson AGX Orin edge compute system.

Holoscan applications are written by composing lists of operators. Connecting the output of one operator to the input of another operator (via the add_flow API) configures Holoscan’s pipeline and specifies when individual operators can run.

Holoscan Sensor Bridge host software leverages the Holoscan framework by providing operators that receive network data generated by Holoscan Sensor Bridge devices. For example, RoceReceiverOperator provides a facility for receiving UDP messages from a Holoscan Sensor Bridge source and storing that data in GPU memory. When used with a video camera source, received data appears as a block of memory with CSI-2 bayer image data. Holoscan includes off-the-shelf operators for construction of RGBA images from Bayer video, image signal processing, inference, and visualization, facilitating real-time processing of video data. For more details, see IMX274 demo.

Different applications have different uses for received data. For example, a high-speed analog signal (e.g. 5G cellular antenna) has different requirements than a video camera. In general, application code works with APIs on sensor objects to provide appropriate configuration and controls for that device; that sensor object knows how to configure the network receiver operator as needed. If an application or device has requirements that are not met by the current implementation, Holoscan operators are provided as source code: users can adapt these to meet their requirements, and if desired, these modifications can be sent back to NVIDIA for inclusion in future versions of sensor bridge host software.

Holoscan applications work by creating a HoloscanApplication subclass, overriding its configure method and using that to construct the appropriate pipeline for the application. Holoscan Sensor Bridge application pipelines are usually focused on acquiring data from sensor equipment, performing processing on that data, generating appropriate outputs, and transmitting that output data to the appropriate target.

APIs for configuring and monitoring specific sensors are presented on sensor objects. For example, Imx274Cam is a sensor object with methods for initialization-time configuration, updates to runtime parameters (e.g. exposure), and monitoring device health. Holoscan applications use those APIs to affect device behavior: for example, an “automatic exposure operator” may receive a video frame as input and use that to determine the value sent to the camera’s set_exposure method. Sensor objects are user-mode drivers for specific devices: they provide access to device functionality but do not include application specific behavior.

Holoscan Sensor Bridge software can be run on any system that is supported by NVIDIA Holoscan. For the best performance, an IGX system, using ConnectX SmartNICs are recommended. In systems without ConnectX NICs, such as the Jetson AGX Orin, performance is limited by the host OS network stack.