This topic provides advanced information for developing USB cameras and Bayer and YUV image sensor with NVIDIA® Tegra® Board Support Package (BSP) releases.

Additionally, guidance is provided for the implementation of drivers suitable for use with this release. Implementation of a camera sensor driver enables acquisition of camera data over the CSI bus, in the native format, provided by the sensor. It is intended for use in enabling sensors that contain an ISP such as YUV output.

The Binary Support Package (BSP) supports two types of camera programming paths, depending on the camera and the application.

The Scalable Camera Framework (SCF) user mode library provides all the controls and data processing between the application and kernel-mode V4L2 drivers. In applications that use the Tegra ISP functionality, use the Scalable Camera Framework. Typically, for Bayer sensors, applications use the SCF library to convert RGB format to YUV format and to do various post processing tasks.

The architecture framework with the application and kernel mode V4L2 drivers is as follows:

In applications that support a direct V4L2 interface, use this interface to communicate to the NVIDIA V4L2 driver without having to use the SCF library. Use this path for a YUV sensor since this sensor has a built-in ISP and frame does not need extra processing.

The application uses the kernel mode V4L2 drivers as follows:

This topic describes how to bring up a Bayer sensor with the Tegra BSP. Bringing up the sensor requires customers to develop:

The provided examples use OmniVision OV5693 sensor, and the source code for OV5693 sensor is available.

A camera module installed on the target platform can consist of one or more devices. A typical rear camera module includes a complementary metal-oxide semiconductor (CMOS) sensor, combined with a Voice Coil Motor (VCM) focuser. A typical front camera module may include a single CMOS sensor.

A typical device-tree node for a camera module is as follows:

The information for moduleX: module (or moduleX: modules) is as follows:

The information for drivernodeX: device (or drivernodeX: devices) is as follows:

An imaging device is a component inside the camera module. It can be a:

You must add all the required information to the device-tree node to support the device operation.

For each device-tree node for a device, assign a device node that contains:

An example device-tree node for the OV5693 V4L2 sensor driver is as follows:

For the device tree node for the V4L2 sensor-device, define the required hardware resource for the device, as follows:

The property-value pairs that apply to the sensor mode for the V4L2 implementation are as follows:

The required information for an LC898212 focuser driver is as follows:

The focuser driver properties are as follows:

This topic is based on the Video for Linux 2 (V4L2) driver for the OmniVision OV5693 sensor (ov5693.c) at:

Examine this source content to obtain a complete understanding of the driver.

The sensor-specific macro values include:

The structure contains the private data specific to the sensor:

The sensor properties are as follows:

Depending on the I2C interface of the sensor, you must update the values of reg_bits and val_bits.

The regmap_config properties are as follows:

Check the vendor register programming table of your sensor to determine the size of reg_bits and val_bits.

The controls must be linked to the control handlers.

The following code example lists the controls and their initialized values. This list is looped through during the ctrls_init call to initialize each of the controls. Each control is then accessible through the ctrls handler in the private data. The set of controls defined for OV5693 are the standard ones used by the user-mode PCL V4L2 driver.

Three types of controls are defined for the OV5693 sensor.

Set up register writes for integer controls with the following functions:

Each control handler calls these functions to set up register writes for each of the controls.

The following functions are wrappers for the read-write interface of the I2C register. For OV5693, use the regmap interface. However, you can modify these functions for other interfaces.

The functions for power-related controls are as follows:

The ov5693_s_stream function is mainly for writing mode tables by making calls to register the write_table function. You set up mode tables in the ov5693_mode_tbls.h file. For details, see Mode Tables.

In addition to writing mode tables and enabling the stream through stream-enable register writes, ov5693_s_stream also writes the initial integer-control values to the register through direct calls to the integer-control handlers. For details, see the section Control Handlers.

If a test pattern is supported by the sensor, and you can create a register table for that pattern, you can add a test_mode flag to write the test-mode table.

Camera common is a set of functions that are common to camera drivers of the NVIDIA kernel, to which this driver refers. Camera common sets up most of the V4L2 framework, requiring linkage from the driver as follows:

For details on modifying and adding new modes, see Mode Tables.

The following code snippets set up the V4L2 subdevices and camera common for registering the sensor driver with the V4L2 framework. The s_stream pointer must point to the internal s_stream function of the sensor; you can leave the other pointers as is.

You do NOT need to modify this structure:

Match the name for the pointer for the core and video operations function with the two from above, as follows:

During the parsing of the device tree, of_device_id matches the compatible field with the one in the Device Tree.

This structure is required for camera common and you must set up the function pointers appropriately. Link the power_on, power_off, write_reg, and read_reg functions here:

This topic describes the control handlers that control the top-level structure and tells the function how many controls the handler is expected to handle.

The set-control function is used to call the desired control handler.

The V4L2 set-control function contains a switch statement to redirect set-control calls to their appropriate control handlers.

Setter-control handlers are the control handlers called by s_ctrl. They perform additional control-handling operations, such as writes to registers. The majority of these controls make calls to the control register setup functions. The exception is write_eeprom, which acts as a separate I2C device with its own I2C write interface.

For more information, see Setting Up Control Registers.

Note a special case for the gain-control handler, which contains a function call to:

The function takes a binary-coded decimal from user space as input and computes the gain-register value according to the vendor-provided gain-calculation formula. Because that formula can vary from sensor to sensor, you must rewrite this function in the V4L2 sensor driver for each of the sensors.

The input of this function is the gain value passed in from user space.

The output of this function is the actual gain-register value programmed over I2C. That value depends on the gain-calculation formula provided by the sensor vendor. It is usually found in the datasheets for the sensor. The goal of this function is to convert the decimal val input into the gain-register value with as little truncation as possible.

For the OV5693 formula on which the ov5693_calculate_gain (or _to_gain) function is based, see the OV5693 Software Reference Manual.

Sensor drivers controlling sensor exposure and frame rate must implement V4L2_CID_FRAME_LENGTH and V4L2_CID_COARSE_TIME controls. These controls are used to update sensor frame length and coarse integration time registers with the values calculated from user-mode camera stack.

If the sensor driver has special requirements or direct calculation of the exposure and frame rate is required, V4L2_CID_FRAME_RATE and V4L2_CID_EXPOSURE controls can be implemented instead. When these two controls are implemented, they receive unprocessed frame rate and exposure values in fixed point format. The driver must perform any necessary determination of final register settings.

When the driver implements V4L2_CID_FRAME_RATE and V4L2_CID_EXPOSURE controls, the received control values are in Q10.22 fixed point format. In Q10.22 fixed point format, the upper 10 bits are for storing integer values and the lower 22 bits are for fraction values. This is a workaround for lacking floating point in the kernel.

This formatting is achieved by multiplying the original value with 1 << 22. For your convenience, a FIXED_POINT_SCALING_FACTOR value is defined as follows:

Use caution before using these fixed point format values. The value must be divided back to original precision for the calculation. For best results, multiply first, and then divide the value with FIXED_POINT_SCALING_FACTOR last, to avoid unexpected truncation/rounding issues.

Following are the get-volatile controls:

Following are the other control-related functions:

The functions for initializing boot time.

The ov5693_ctrls_init function iterates through ctrl_config_list and registers each control as a new custom control with the V4L2 framework. s_ctrl is also called for each control to set it to its default value defined in ctrl_config_list.

For more information, see Configuration Controls.

Modifying this function is necessary if you have calls for initializing the value for the read-only controls. See the calls to otp_setup and fuse_id_setup in Control Handlers.

The ov5693_parse_dt function, which parses device trees, takes the Device Tree node and looks for the parameters, according to the sensor-related private data, required by the sensor driver.

The values include but are not limited to:

For details on how to set up device trees with the appropriate values, see the documentation. You must match the name list with the names in the Device Tree for their respective name-value pairs.

The port binding properties are as follows:

A few additional components are needed to setup the media controller for V4L2 use. The open call is a placeholder operation for satisfying the V4L2 sub-device internal operation requirements. The setup likely will not need to be change besides a name changed to match the devices.

Sub-device function operations must link to the open operation:

Media entity function operations must link to the V4L2 sub-device link validation method:

The probing function of the sensor driver are as follows.

The ov5693_probe function is the entry point of the driver. The function starts off by allocating memory for common data and sensor-private data. For more information, see Sensor-Private Data.

The function then initializes regmap:

Next, the function calls the other initialization functions, including:

Next, the function links the common data and sensor-private values:

For more information, see Macro Definitions.

All imager devices must register themselves as sub devices to media controller framework. VI acts as the master device which controls the binding, parsing the DT and establishes the media links once all the sub devices are registered.

Each sub-device can register to media controller framework by defining the entity and pads information.

The ov5693_remove function removes the sensor-device instance and calls on a device shutdown.

This function makes calls to various free, put, and destroy functions that match up with several calls in probe. It must remain largely the same.

After driver development is complete, you must add your device information to the system kernel device tree so it can be instantiated when the kernel boots. There are two ways of registering your device.

If your camera module has onboard EEPROM and has a valid camera ID programmed, Plugin Manager can be used. If your device module does not meet this requirement, use the main platform device tree instead by following the directions in Using Main Platform Device Tree File.

Plugin Manager automatically links devices when the system kernel boots. Plugin Manager uses the camera module ID returned by the boot loader to update the device tree entries with proper information at runtime. Plugin Manager allows a single system image to support multiple camera devices. To change camera modules, power down the device, replace the camera module, and then reboot. The new module works automatically.

For Plugin Manager support, add your DTSI file to the camera configuration DTSI file, and then update the camera plugin manager DTSI with proper override information.

Register your new device by updating main platform DTSI file to include your new device DTSI file. Because Tegra uses Plugin Manager by default, you must first unregister Plugin Manager support, then add your device information to the main device tree DTSI file.

When the driver is complete, before running any camera applications, you should first check to see if the camera device node(s) got populated correctly in the system.

If the driver is loaded properly, you should see a /dev/<video#> node. The device node is the file I/O interface with which the user-space driver accesses the driver.

Problems might occur in the probing process and require debugging. Typically, problems occur in the clock, GPIO, and regulator setup. See below for a few tips.

Your device name should be the same in both the Device Tree and your driver. This is how kernel binds the driver to your device.

In Device Tree:

In driver:

To debug those problems, first verify that the names that are being read through the parse_dt function matches those in the Device Tree.

Ensure that the values assigned to the Device Tree fields match the hardware they describe. Oftentimes, regulator names, GPIO numbers, and clock names are out-of-date in the Device Tree, causing probing to fail.

Ensure that the power_get calls runs to completion. For details, see Power Functions and Boot-Time Initialization.

Another common problem that occurs during probe is in control init.

Verify that default values are all linked correctly in the control configuration. Also, verify that the default macros are updated appropriately with the values in the mode table of the sensor.

For more information, see Sensor-Private Data and Macro Definitions.

If new controls have been added to the control configuration list, on rare occasions, ensure that they contain the appropriate control handlers and default values.

After probing succeeds, problems could still occur with the control setting. A common problem is in the values of the register writes.

Ensure that the control-register setup contains the correct register address and formats according to the mode table and data sheets. For more information, see Setting Up Control Registers.

For gain control, create a new calculate_gain (to_gain) function for calculating the gain value according to the gain formula in the datasheets.

Double-check the mode-specific settings in the Device Tree and update them if necessary. The most common issues that cause sensor timeout are due to the wrong values for the following settings:

Look closely in kernel logs for any i2c errors for the camera sensor. If there is an i2c error, check the power-on function to verify that the sensor power-on sequence is correct. The order of enabling power, clock, GPIO, and timing must comply with sensor specifications.

The modes tables of the register reside in a separate header file called sensor_mode_tbls.h and are included by the main driver. Those tables can be a list of reg_8 or reg_16 address-value pairs. Mode tables are separated by resolution. For the ov5693 driver, a common mode table also exists, which contains the common register values among all the resolutions.

The start and stop stream-register values must be in a separate mode table. When you separate them, delete the start stream-register values at the end of the resolution mode tables.

Also, a table for the color bars of the test pattern is used if the test_mode flag is enabled, activating the test-pattern mode of the sensor. That table is required only if test pattern is supported by the sensor.

At the end of the header of the mode tables is an enumeration of all the mode tables, as well as a list that maps the enumeration to table pointers.

The camera_common_frmfmt array list is a required table that sets up the format for the V4L2 framework. Each of the elements on that list contains the resolutions, the is_hdr flag, and the enumeration for the mode.