Camera Software Development Solution

This topic describes the NVIDIA® Jetson™ camera software solution, and explains the NVIDIA-supported and recommended camera software architecture for fast and optimal time to market. It outlines and explains development options for customizing the camera solution for USB, YUV, and Bayer camera support.

Camera Architecture Stack

The NVIDIA camera software architecture includes NVIDIA components that allow for ease of development and customization:

Camera architecture stack

The camera architecture includes the following NVIDIA components:

  • libargus: Provides a low-level API based on the camera core stack.

  • nvarguscamerasrc: NVIDIA camera GStreamer plugin that provides options to control ISP properties using the ARGUS API.

  • v4l2src: A standard Linux V4L2 application that uses direct kernel IOCTL calls to access V4L2 functionality.

NVIDIA provides OV5693 Bayer sensor as a sample. NVIDIA tunes this sensor for the NVIDIA® Jetson™ platform. The drive code, based on the media controller framework, is available at:

./kernel/nvidia/drivers/media/i2c/nv_ov5693.c

NVIDIA provides additional sensor support for Jetson Board Support Package (BSP) software releases. Developers must work with NVIDIA certified camera partners for any Bayer sensor and tuning support. The work involved includes:

  • Sensor driver development

  • Custom tools for sensor characterization

  • Image quality tuning

These tools and operating mechanisms are not part of the public Jetson Embedded Platform (JEP) Board Support Package release.

Camera API Matrix

The following table describes the camera APIs available for each camera configuration.

  Uses Jetson ISP (CSI interface) Does not use Jetson ISP† (CSI interface) USB (UVC)‡ (USB interface)
Camera API libargus
GStreamer (GST-nvarguscamerasrc) 		V4L2 V4L2
† libargus is the preferred path to access the cameera.

‡ Drivers are not provided by NVIDIA; you must write them. You can support peripheral bus devices such as:

  • Ethernet
  • Non-UVC USB

Note

The default OV5693 camera does not contain an integrated ISP. Use of the V4L2 API with the reference camera records “raw” Bayer data.

Approaches for Validating and Testing the V4L2 Driver

Once your driver development is complete, use the provided tools or application to validate and test the V4L2 driver interface.

For general GStreamer and multimedia operations, see Multimedia User Guide, available from the Jetson Download Center.

Applications Using libargus Low-Level APIs

The NVIDIA Multimedia API provides samples that demonstrate how to use the libargus APIs to preview, capture, and record the sensor stream.

The Multimedia API may be installed with NVIDIA® SDK Manager or as a standalone package.

Multimedia API documentation is in Jetson Linux API Reference, included in the Jetson Linux documentation package, and available for downloading separately from the Jetson Software Documentation page of the Jetson documentation site.

Applications Using GStreamer with the nvarguscamerasrc Plugin

Use the nvarguscamerasrc GStreamer plugin supported camera features with ARGUS API to:

  • Enable ISP processing for Bayer sensors

  • Perform format conversion part of ISP for Bayer Capture

For example, for a Bayer sensor with the format 1080p/30/BGGR:

  • Save the preview into the file:

    $ gst-launch-1.0 nvarguscamerasrc num-buffers=200 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12' ! omxh264enc ! qtmux ! filesink location=test.mp4 -e

  • Render the preview to an HDMI® screen:

    $ gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, format=(string)NV12, framerate=(fraction)30/1' ! nvoverlaysink -e

For more information about the ARGUS API, see the Libargus Camera API page of the Jetson Linux API Reference.

Applications Using GStreamer with V4L2 Source Plugin

Using the Bayer Sensor, YUV sensor, or USB camera, to output YUV images without ISP processing does not use the NVIDIA camera software stack.

For example, for a USB camera with the format 480p/30/YUY2:

  • Save the preview into a file (based on a software converter):

    $ gst-launch-1.0 v4l2src num-buffers=200 device=/dev/video0 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! videoconvert ! omxh264enc ! qtmux ! filesink location=test.mp4 -ev

  • Enter export DISPLAY=:0 if you are operating from a remote console. Then render the preview to a screen:

    $ gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! xvimagesink -ev

For a YUV sensor with the format 480p/30/UYVY:

  • Save the preview into a file (based on a hardware-accelerated converter):

    $ gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! omxh264enc ! qtmux ! filesink location=test.mp4 -ev

  • Render the preview to a screen:

    $ export DISPLAY=:0 //if you are operating from remote console
$ gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)30/1' ! xvimagesink -ev

Applications Using V4L2 IOCTL Directly

Use V4L2 IOCTL to verify basic functionality during sensor bringup.

For example, to capture from a Bayer sensor with the format 1080p/30/RG10:

$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video0 --stream-to=ov5693.raw

ISP Configuration

The Camera Core release package includes model ISP configuration files for reference sensors.

Note

CSI cameras, with integrated ISP and USB camera, can work in ISP bypass mode. Provided ISP support is available for Jetson Developer Kit (OV5693) RAW camera module.

Additional ISP support for other camera modules are supported using third party partners.

Infinite Timeout Support

This use case is different from typical camera use cases, in which the camera sensor streams frames continuously and the camera hardware and driver support finite timeouts while waiting for camera frames.

In this use case a camera sensor is triggered to generate a specified number of frames, after which it stops streaming indefinitely (the “infinite timeout”). Whenever the camera driver resumes streaming, it must resume capturing frames without timeout issues. The camera driver and hardware must always be ready to capture frames coming from the CSI sensor. The camera driver does not know when streaming will start again.

To support this use case, the camera sensor hardware module must support suspending and resuming streaming.

To enable this feature, run the camera server (i.e. the nvargus-daemon service) by setting the enableCamInfiniteTimeout environment variable to 1:

$ sudo service nvargus-daemon stop
$ sudo enableCamInfiniteTimeout=1 nvargus-daemon

Other References