RealSense Camera

The Intel RealSense 435 camera is a stereo camera that computes depth with the help of an IR emitter. The RealSense camera codelet can be configured in several ways to work with various Isaac SDK GEMs:

  • As a regular color camera with GEMs that work with a single camera image–for example, with object detection.

  • As a depth camera providing color and depth images to GEMs like Superpixels.

  • as a monochrome stereo camera providing grayscale left and right IR camera images to GEMs like Stereo Visual Odometry.

RealsenseCamera Codelet

Isaac SDK provides the RealsenseCamera codelet to access the data streams from a RealSense camera via the librealsense SDK. On Jetson all dependencies are handled automatically by the Isaac SDK and do not require manual installation.

The codelet can be configured via various parameters. Most notably you can change the desired resolution via the rows and cols parameters and the desired framerate via the rgb_framerate and depth_framerate parameters. Please note that only certain modes are supported as mentioned in the codelet documentation. If you have multiple cameras connected you need to specify the desired device via the dev_index or serial_number parameters.

Sample Application

ISAAC provides a basic sample application to test your RealSense camera. You can run it with the following command:

bob@desktop:~/isaac$ bazel run //apps/samples/realsense_camera

Once the application is running, you can use Isaac Sight to see the images captured by the camera.

Troubleshooting

Notes on the Firmware

The RealSense camera may come with an outdated firmware version. You can check the firmware version of your camera by running the RealSense sample application as explained above. The application prints a message to the console if the firmware version is not the recommended version.

In case you need to update your firmware follow the instructions on the official website: https://dev.intelrealsense.com/docs/firmware-update-tool.

The recommended firmware version is 5.11.15.

Using a USB 3.0 Port with a USB 3.0 Cable

The RealSense camera requires a USB3.0+ connection. USB 3.0 ports and cables can be identified by blue-color components on the inside of ports and cable connectors. The Jetson Xavier USB-C ports are USB 3.0 ports.

Testing the RealSense camera with USB-C ports has revealed stability issues that are resolved using an unpowered USB hub. The AmazonBasics USB 3.1 Type-C to 4-Port Aluminum Hub Connector is known to work reliably with the RealSense camera.

The Accell USB C-to-A and C-to-C SuperSpeed+ USB 3.1 Gen 2 (10 Gbps) cables are known to work reliably with the RealSense camera.

x86_64 Linux Host Setup

Custom Linux kernel module patches and UDEV rules must be installed on a x86_64 Linux host for RealSense 435 camera to work correctly.

We recommend manual setup of Linux kernel modules because Isaac SDK uses 2.29 version of librealsense. To perform manual setup, please follow the Intel RealSense SDK setup guidelines.

Setting the Power Model

Running sudo nvpmodel -m 0 enables all cores, and increases the scaling minimum clock frequency. This change persists over reboot. Run sudo nvpmodel -q to display current settings. This increases processing power and may result in reduced frame drops and image blur, in some situations.