Sensor Driver Programming Guide
Implementing the camera sensor driver enables acquisition of camera data over the CSI bus, in the native format, provided by the sensor.
There are two types of camera programming paths, depending on the camera and the application.
• Camera Core Library Interface
• Direct V4L2 Interface
Camera Core User Mode Library Interface
The camera core user mode library provides all the controls and data processing between the application and kernel-mode V4L2 drivers. The typical usecases for using the camera core libray interface include:
• Applications that use the Tegra ISP functionality
• Applications that must convert RGB format to YUV format and perform various post-processing tasks for Bayer sensors
The Linux for Tegra architecture framework with the application and kernel mode V4L2 drivers is as follows:
The Android architecture framework with the application and kernel mode V4L2 drivers is as follows:
Direct V4L2 Interface
In applications that support a direct V4L2 interface, use this interface to communicate to the NVIDIA V4L2 driver without having to use the camera core user mode library. Use this path for capturing RAW data from sensor or for validating sensor drivers.
The application uses the kernel mode V4L2 drivers as follows:
Bringing up a Bayer pixel format sensor requires customers to develop:
• Device Tree in the kernel
• V4L2 sensor driver
The provided examples use:
• Sony IMX185 sensors
The source code is available for:
• Sony IMX185 sensor
Camera Modules and Device Tree
A camera module installed on the target platform can consist of one or more devices. A typical rear camera module includes a complementary metal-oxide semiconductor (CMOS) sensor. A typical front camera module may include a single CMOS sensor.
To add one or more camera modules to a device tree
1. Locate or create a tegra-camera-platform device node in the kernel source tree at:
• TX1:
<top>/hardware/nvidia/platform/t210/jetson/kernel-dts/jetson-platforms/tegra210-camera-imx185-a00.dtsi
• TX2:
<top>/hardware/nvidia/platform/t18x/common/kernel-dts/t18x-common-modules/tegra186-camera-li-mipi-adpt-a00.dtsi
Where <top> is the directory where you installed the board support package.
2. In a Tegra-camera-platform device node, create a module table (modules) with one or more modules.
Each module must contain its basic information and the definition of the devices that are inside that module.
Note: | All value fields in camera-related device nodes must use the string data type, except for the files that refer to other device nodes. |
A typical device-tree node for a camera module is as follows:
tegra-camera-platform {
compatible = "nvidia, tegra-camera-platform";
modules {
module0 {
badge = "imx185_bottom_liimx185";
position = "bottom";
orientation = "0";
drivernode0 {
pcl_id = "v4l2_sensor";
devname = "imx185 30-001a";
proc-device-tree = "/proc/device-tree/i2c@3180000/tca9546@70/i2c@0/imx185_a@1a";
}; };
};
};
Module Properties
The information for moduleX: module (or moduleX: modules) is as follows:
Property | Value |
badge | A unique name that identifies this module. Guidelines for naming the three parts of the badge_info property: • The first part is the camera board ID (camera_board_id) of the module. • The second part contains the position of the module, for example, rear or front. • The third part contains the last six characters of a part number, which you can find in the datasheet on the module from the vendor. If the part number is not available, use a unique identifier. If your system has multiple modules that are identical, create a unique name for each module. For example, if you have a rear facing camera, you can call the rear camera module imx185_rear_liimx185. |
position | The camera-facing information. The positions supported depends on the number of cameras in the system: • Two-camera system: rear and front. • Three-camera system: bottom, top, and center. • Six-camera system: bottomleft, bottomright, centerleft, centerright, topleft, and topright. |
orientation | An index based orientation for the sensor. When a device has a display with two cameras, typically these indices are used: • Rear facing: 0 • Front facing: 1 For additional camera layouts, each camera requires a unique index. |
Individual Imaging Device
An imaging device is a component inside the camera module. It can be a:
• Sensor
• Focuser
• Flash
The required information must be added to the device-tree node to support the device operation.
For each device-tree node for a device, assign a device node that contains:
• The name of the device
• The slave address for the device
The slave address is available in the device datasheet.
• A compatible string that identifies the node
Note: | Except for devices that refer to other device nodes, all value fields in camera-related device nodes must use the string data type. |
An example device-tree node for the IMX185 V4L2 sensor driver is as follows:
imx185_a@1a {
compatible = "nvidia,imx185";
reg = <0x1a>;
devnode = "video0";
physical_w = "15.0";
physical_h = "12.5";
sensor_model ="imx185";
post_crop_frame_drop = "0";
use_decibel_gain = "true";
delayed_gain = "true";
use_sensor_mode_id = "true";
mode0 {
mclk_khz = "37125";
num_lanes = "4";
tegra_sinterface = "serial_a";
discontinuous_clk = "no";
dpcm_enable = "false";
cil_settletime = "0";
dynamic_pixel_bit_depth = "12";
csi_pixel_bit_depth = "12";
mode_type = "bayer";
pixel_phase = "rggb";
active_w = "1920";
active_h = "1080";
readout_orientation = "0";
line_length = "2200";
inherent_gain = "1";
mclk_multiplier = "2";
pix_clk_hz = "74250000";
min_gain_val = "0"; /* dB */
max_gain_val = "48"; /* dB */
min_hdr_ratio = "1";
max_hdr_ratio = "1";
min_framerate = "1.5";
max_framerate = "30";
min_exp_time = "30";
max_exp_time = "660000";
embedded_metadata_height = "1";
};
…
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
liimx185_imx185_out0: endpoint {
csi-port = <0>;
bus-width = <4>;
remote-endpoint = <&liimx185_csi_in0>;
};
};
};
Device Properties
For the device tree node for the V4L2 sensor-device, define the required hardware resource for the device, as follows:
Property | Value |
compatible | Specifies the device identifier. The Linux kernel uses this keyword to bind the device driver to a specific device. |
reg | Specifies the I2C slave address. |
mclk | Specifies the name of the input clock for the device. • On TX1: default MCLK is clk_out_3 • On TX2: default MCLK is extperiph1 |
XXX-gpio | Specifies the general-purpose input/output (GPIO) pins for the device, where XXX is the GPIO pin for the camera. • On TX1, the default GPIO pins for camera include: • S4 - Camera0 Reset • S7 - Camera0 Power Down • S5 - Camera1 Reset • T0 - Camera1 Power Down • On TX2, the default GPIO pins for camera include: • R5 - Camera0 Reset • R0 - Camera0 Power Down • R1 - Camera1 Reset • L6 - Camera1 Power Down |
XXX-supply | Specifies the regulator for the device, where XXX is the actual regulator name defined somewhere else in the device tree. The -supply suffix is mandatory. The following are the defined regulators: • vana-supply = <&en_vdd_cam_hv_2v8>; // analog 2.8v • vdig-supply = <&en-vdd-cam_1v2>; // digital 1.2v • vif-supply = <&en-vdd-cam>; // interface 1.8v • vvcm-supply = <&en_vdd_vcm_2v8>; // analog 2.8v for vcm |
XXX-reg | Specifies the name of the regulator, where XXX is the regulator name for the sensor driver. The value of this field is the regulator name with the suffix -reg. The following are the defined regulators: • avdd-reg = “vana”; • dvdd-reg = “vdig” • iovdd-reg = “vif” • vcmvdd-reg = “vvcm” |
physical_w | Specifies the physical width (in millimeters) of the sensor. |
physical_h | Specifies the physical height (in millimeters) of the sensor. |
sensor_model* | Specifies which sensor is in this module |
post_crop_frame_drop* | Specifies number of frames to be dropped after applying sensor crop settings. For more information, refer to the IMX185 Sensor Driver documentation. |
use_decibel_gain* | Use this option for sensors that use decibels to represent the analog gain. When set to true, the analog gain value received by the driver is expressed in decibels (dB), based on the following conversion in the user mode library: dB = 20 * log(Analog Gain) For more information, refer to the IMX185 Sensor Driver documentation. |
delayed_gain* | Used to delay the gain setting. When set to true, the user mode driver delays sending the updated gain value to the driver by one frame. Default is “false”. For more information, refer to the IMX185 Sensor Driver documentation. |
user_sensor_mode_id* | When set to true, the user mode driver uses TEGRA_CAMERA_CID_SENSOR_MODE_ID control to select a specific sensor mode and bypasses the default mode selection logic. Default is “false”. For more information, refer to the IMX185 Sensor Driver documentation. |
* Optional Properties, in this table, can be specified in different DTSI files. For example, clock, GPIO, and regulator properties can be specified in the platform DTSI file and the rest of the properties can be specified in the sensor DTSI files. |
Property-Value Pairs
The mandatory property-value pairs that apply to the sensor mode for the V4L2 implementation are as follows:
Property | Value |
modeX | Specifies the sensor-mode information, that is, the X: 0-based index. |
ports | Provides the media controller graph binding information. For more information, see Port Binding. |
mclk_khz | Specifies the standard MIPI driving clock. Unit in KHz. |
num_lanes | Specifies the number of lane channels the sensor is programmed to output. |
tegra_sinterface | Specifies the base Tegra serial interface to which the lanes are connected. |
discontinuous_clk | Specifies the indication that the sensor is programmed to use a discontinuous clock on MIPI lanes. |
cil_settletime | Specifies the value of the THS-Settle time of the MIPI lane. A 0 value attempts to auto-calibrate according to the mclk_multiplier parameter. If you are not using auto-calibration, you can calculate the values with the formula: 85ns + 6*UI < (cli_settletime + 5) < 145ns + 10*UI. Where UI is the unit interval and is equal to the duration of the HS state on the Clock Lane. Consult the MIPI Alliance Specification for D-PHY for details. |
dpcm_enable | Specifies whether to enable dpcm compression for this mode. Set to true or false. |
active_h | Specifies the height of the pixel-active region. |
active_w | Specifies the width of the pixel-active region. |
pixel_t Deprecated | This property is deprecated and replaced with these properties: • mode_type • csi_pixel_bit_depth • pixel_phase |
mode_type | Specifies the sensor mode type. Possible values include: • yuv • bayer • bayer_wdr_pwl - Wide Dynamic Range mode that uses the piece-wise linear function to compress multi exposure fused pixel data. Multi exposure fusion is also performed on the sensor. For more information, consult the IMX185 Sensor Driver documentation. |
csi_pixel_bit_depth | Specifies the bit depth of the final sensor output on the CSI bus. “mode_type = bayer_wdr_pwl” represents the bit depth output after piece-wise linear function based compression is applied. For more information, consult to the IMX185 Sensor Driver documentation. |
dynamic_pixel_bit_depth* | Specifies the true dynamic range of the signal. For non WDR mode types, this is equal to csi_pixel_bit_depth. For “mode type = bayer_wdr_pwl”, specifies the bit depth of the multi exposure fused output before piece wise linear function based compression has been applied. For more information, consult the IMX185 Sensor Driver documentation. |
pixel_phase* | Specifies the sensor pixel phase. Possible values include: • uyvy • vyuy • yuyv • yvyu • rggb • bggr • grbg • gbrg For more information, consult to the IMX185 Sensor Driver documentation. |
readout_orientation Applies to: Android | Specifies the readout orientation that is based on the orientation of the camera module. Change this property if you would like to program a different readout order for this mode. Possible values: • 0 • 90 • 180 • 270 |
line_length | Specifies the pixel line width horizontal timing size for the sensor mode for calibrating the features in the camera stack. This value must be equal or larger than active_w. |
mclk_multiplier | Specifies the multiplier to MCLK for timing the capture sequence of the hardware. Use the following equation to calculate this value: mclk_multiplier = desired ISP clock frequency / mclk. This value must be larger than pixel_clk_hz / mclk to prevent ISP underrun. |
pix_clk_hz | Specifies the sensor pixel clock for calculating the exposure, frame rate, and so forth. This value is calculated based on input clock (mclk) and PLL settings from sensor mode table. Please refer to sensor data sheet for how to calculate this value. |
inherent_gain | Specifies the gain obtained inherently from the mode, that is, pixel binning. Set to 1 if you do not know this value. |
min_gain_val | Specifies the minimum gain limit for the mode. Floor to 6 decimal places. Usually is 1.0 (1x gain). |
max_gain_val | Specifies the maximum gain limit for the mode. Floor to 6 decimal places. If supported by the sensor, it can be increased to include digital gain. |
min_exp_time | Specifies the minimum exposure time limit for the mode, in microseconds. Ceiling function to integer. The equation to calculate this value is as follows: min_exp_time = (minimum coarse integration time) * line_ length / pix_clk_hz * 1000000 minimum coarse integration time is the minimum exposure intervals. Unit in lines. |
max_exp_time | Specifies the maximum exposure time limit for the mode in microseconds. Ceiling function to integer. The equation to calculate this value is as follows: min_exp_time = (maximum coarse integration time) * line_length / pix_clk_hz * 1000000 minimum coarse integration time is the maximum exposure intervals. Unit in lines. |
min_hdr_ratio | Specifies the minimum high-dynamic-range (HDR) ratio limit for the mode (for interleave HDR sensors). For non-HDR sensors, set min_hdr_ratio to an empty string. |
max_hdr_ratio | Specifies the maximum HDR ratio limit for the mode (for HDR sensors). |
min_framerate | Specifies the minimum frame-rate limit for the mode in frames per second (fps). Floor to 6 decimal places. The equation to calculate this value is as follows: min_framerate = pix_clk_hz / (line_length * maximum frame length) |
max_framerate | Specifies the maximum frame-rate limit for the mode in fps. Floor to 6 decimal places. The equation to calculate this value is as follows: max_framerate = pix_clk_hz / (line_length * minimum frame length) |
embedded_metadata_height | Specifies how many extra embedded metadata rows for each frame. Set to 0 to disable embedded metadata support. |
num_control_point* | Valid for “mode_type = bayer_wdr_pwl”. Specifies the number of control points used in the piece-wise linear function used to compress multi exposure fused pixel data. Multi exposure fusion is also performed on the sensor. Up to 16 control points are supported by the camera core user mode library interface. For more information, consult the IMX185 Sensor Driver documentation. |
control_point_x_[0..15]* | Valid for “mode_type = bayer_wdr_pwl”. Specifies the x-coordinate of the N control point from the piece-wise linear compression function. The value of the x-coordinate cannot exceed 2 to the power of dynamic_pixel_width_depth. |
control_point_y_[0..15]* | Valid for “mode_type = bayer_wdr_pwl”. Specifies the y-coordinate of the N control point from the piece-wise linear compression function. The value of the y-coordinate cannot exceed 2 to the power of csi_pixel_width_depth. |
* Alternative to pixel_t when WDR sensor is used. |
Example Piece-wise Linear Compression Function
An example piece-wise linear compression function is as follows:
• Input signal has 16-bit depth
• Output signal has 12-bit depth
• csi_pixel_bit_depth = “12”
• dynamic_pixel_bit_depth = “16”
The control point properties are as follows:
num_control_point = "4";
control_point_x_0 = "0";
control_point_x_1 = "2048";
control_point_x_2 = "16384";
control_point_x_3 = "65536";
control_point_y_0 = "0";
control_point_y_1 = "2048";
control_point_y_2 = "2944";
control_point_y_3 = "3712";
Port Binding
vi {
num-channels = <1>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
liimx185_vi_in0: endpoint {
csi-port = <2>;
bus-width = <2>;
remote-endpoint = <&liimx185_csi_out0>;
};
};
};
};
nvcsi {
num-channels = <1>;
#address-cells = <1>;
#size-cells = <0>;
channel@0 {
reg = <0>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
liimx185_csi_in0: endpoint@0 {
csi-port = <0>;
bus-width = <4>;
remote-endpoint = <&liimx185_imx185_out0>;
};
};
port@1 {
reg = <1>;
liimx185_csi_out0: endpoint@1 {
remote-endpoint = <&liimx185_vi_in0>;
};
};
};
};
The port binding properties are as follows:
Function | Description |
port | Specifies the mediapad port connection. All imager devices have one media pad for binding the connection with VI. |
csi-port | Defines the CSI port sensor connection. For imager devices, such as focuser or flash, this field is not required. |
bus-width | Defines the bus width by identifying the number of CSI lanes connected to sensor. |
remote-endpoint | Defines the label for binding two ports. The binding expects one port to be for the sink and the other one is for the source. |
To verify the port binding result
• Execute the command:
sudo media-ctl -p -d /dev/media0
The output returned is similar to the following:
Media controller API version 0.1.0
Media device information
------------------------
driver tegra-vi4
model NVIDIA Tegra Video Input Device
serial
bus info
hw revision 0x3
driver version 0.0.0
Device topology
- entity 1: 150c0000.nvcsi-0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
<- "imx185 30-001a":0 [ENABLED]
pad1: Source
-> "vi-output, imx185 30-001a":0 [ENABLED]
- entity 2: imx185 30-001a (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev1
pad0: Source
[fmt:SRGGB12/1920x1080 field:none]
-> "150c0000.nvcsi-0":0 [ENABLED]
- entity 3: vi-output, imx185 30-001a (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Sink
<- "150c0000.nvcsi-0":1 [ENABLED]
V4L2 Kernel Driver
This topic is based on the Video for Linux 2 (V4L2) driver for the Sony IMX185 sensor (imx185.c) at:
kernel/kernel-4.4/drivers/media/i2c/imx185.c
Examine this source content to obtain a complete understanding of the driver.
For information on V4L2 implementation, consult:
Macro Definitions
The sensor-specific macro values include:
• The minimum and maximum values for each control.
• The default value for each control.
• The macro values required for sensor timing or general functionality.
Sensor-Private Data
The structure contains the private data specific to the sensor:
struct imx185 {
struct camera_common_power_rail power;
int numctrls;
struct v4l2_ctrl_handler ctrl_handler;
struct i2c_client *i2c_client;
struct v4l2_subdev *subdev;
struct media_pad pad;
u32 frame_length;
s32 group_hold_prev;
bool group_hold_en;
s64 last_wdr_et_val;
struct regmap *regmap;
struct camera_common_data *s_data;
struct camera_common_pdata *pdata;
struct v4l2_ctrl *ctrls[];
};
The sensor properties are as follows:
Property | Value |
power | Holds generic power controls, include regulators, clks, and GPIOs. |
numctrls | Holds the number of v4l2 controls for the sensor. |
ctrl_handlerr | Holds the required v4l2 handler to controls, needed for v4l2 control init. |
i2c_client | Holds a handle to i2c_client, used to access to sensor i2c client instance. |
subdev | Holds a handle for v4l2 sub-device, needed to run subdev operations (ops). |
pad | Holds media pad used for media controller initialization for a device to work as SINK or SOURCE. |
group_hold_prev | Holds previous state use by group hold control handler to check for change of state. |
group_hold_en | Holds group hold enable flag directly related to group hold control. |
frame_length | Holds the previous frame_length value. |
reg_map | Holds a register map setup for I2C read and write. |
s_data | Holds a handle to common data, see documentation for camera_common_data. |
pdata | Holds a handle to common platform data, populated by read Device Tree. |
ctrls | Holds handles to initialized v4l2 controls, dynamic array, MUST BE LAST FIELD. |
Configuring Regmap
Depending on the I2C interface of the sensor, you must update the values of reg_bits and val_bits.
regmap_config {
reg_bits = 16;
val_bits = 8;
};
The regmap_config properties are as follows:
Property | Value |
reg_bits | Specifies the number of bits needed to represent the I2C register offset. |
val_bits | Specifies the number of bits in the buffer to store data to be transferred over I2C. |
Check the vendor register programming table of your sensor to determine the size of reg_bits and val_bits.
Configuring Controls
The controls must be linked to the control handlers.
To link the controls to their control handlers
• Set up the function pointers as follows:
• Point g_volatile_ctrl to the internal get volatile control handler of the sensor.
• Point s_ctrl to the internal set control handler of the sensor.
static const struct v4l2_ctrl_ops imx185_ctrl_ops = {
.g_volatile_ctrl = imx185_g_volatile_ctrl,
.s_ctrl = imx185_s_ctrl,
};
The following code example lists the controls and their initialized values. This list is looped through during the ctrls_init call to initialize each of the controls. Each control is then accessible through the ctrls handler in the private data. The set of controls defined for IMX185 are the standard ones used by the user-mode library.
Note: | Additional controls require a change in the user mode library. |
Three types of controls are defined for the IMX185 sensor.
static struct v4l2_ctrl_config ctrl_config_list[] = {
/* Integer Control: setting integer values such as gain, coarse
* time, *and frame length.
*/
{
.ops = &imx185_ctrl_ops, //pointer to control ops
//function
.id = TEGRA_CAMERA_CID_GAIN, //id, defined in
//camera_common.h
.name = "Gain", //string name of control
.type = V4L2_CTRL_TYPE_INTEGER, //type of control
.flags = V4L2_CTRL_FLAG_SLIDER, //control flags
// The following three are the values that likely need to change
.min = 0 * FIXED POINT SCALING FACTOR, //control value lower bound
.max = 48 * FIXED POINT SCALING FACTOR, //control value upper bound
.def = 0 * FIXED POINT SCALING FACTOR, //default control value
.step = 3 * FIXED POINT SCALING FACTOR, / 10, //increment step size for //value
/* Menu Control: used as on/off switch for group hold and HDR.
* switch_ctrl_qmenu is used to define the states on/off. There
* shouldn't be a need to change these controls, unless a
* completely new one is being added.
*/
{
.ops = &imx185_ctrl_ops,
.id = TEGRA_CAMERA_CID_GROUP_HOLD,
.name = "Group Hold",
.type = V4L2_CTRL_TYPE_INTEGER_MENU,
.min = 0,
.max = ARRAY_SIZE(switch_ctrl_qmenu) - 1,
.menu_skip_mask = 0,
.def = 0,
.qmenu_int = switch_ctrl_qmenu,
},
};
Setting Up Control Registers
Set up register writes for integer controls with the following functions:
• addr depends on the sensor being used.
• val is the source from the control handler.
Each control handler calls these functions to set up register writes for each of the controls.
imx185_get_frame_length_regs
imx185_get_coarse_time_regs_shs1
imx185_get_coarse_time_regs_shs2
imx185_get_gain_reg
Read-Write Wrapper in the Register
The following functions are wrappers for the read-write interface of the I2C register. For IMX185, use the regmap interface. However, you can modify these functions for other interfaces.
imx185_read_reg
imx185_write_reg
imx185_write_table
Power Functions
The functions for power-related controls are as follows:
Function | Description |
imx185_power_on | Contains the power-on sequence. You must modify this function according to the specification sheets. Calls regulator_enable() to turn on regulators and use gpio_set_value() to toggle the GPIO pins. |
imx185_power_off | Contains the power-off sequence. You must modify this function according to the specification sheets. Calls regulator_enable() to turn on regulators and use gpio_set_value() to toggle the GPIO pins. |
imx185_power_put | Calls regulator_put () on all regulators. |
imx185_power_get | Calls camera_common_regulator_get() helper function or regulator_get() to acquire all regulators. |
The driver must call the power_get() function first to acquire all regulators prior to calling the power_on() and power_off() functions.
Additionally, the driver must all the power_put() function to release all regulators when the driver is being released.
Setting Up V4L2 Sub-device and Camera Common
The
imx185_s_stream function is mainly for writing mode tables by making calls to register the
write_table function. You set up mode tables in the
imx185_mode_tbls.h file. For details, see
Mode Tables.
In addition to writing mode tables and enabling the stream through stream-enable register writes,
imx185_s_stream also writes the initial integer-control values to the register through direct calls to the integer-control handlers. For details, see the section
Control Handlers.
Control[0].id = TEGRA_CAMERA_CID_GAIN;
Control[1].id = TEGRA_CAMERA _CID_FRAME_RATE;
Control[2].id = TEGRA_CAMERA _CID_EXPOSURE;
err = v4l2_g_ext_ctrls(&priv->ctrl_handler, &cotrl);
err |= imx185_set_gain(priv, control[0].value64);
……
If a test pattern is supported by the sensor, and you can create a register table for that pattern, you can add a test_mode flag to write the test-mode table.
Camera common is a set of functions that are common to camera drivers of the NVIDIA kernel, to which this driver refers. Camera common sets up most of the V4L2 framework, requiring linkage from the driver as follows:
struct camera_common_sensor_ops
struct camera_common_data
For details on modifying and adding new modes, see
Mode Tables.
The following code snippets set up the V4L2 sub-devices and camera common for registering the sensor driver with the V4L2 framework. The s_stream pointer must point to the internal s_stream function of the sensor; you can leave the other pointers as is.
static struct v4l2_subdev_video_ops imx185_subdev_video_ops = {
.s_stream = imx185_s_stream,
...
};
You do NOT need to modify this structure:
static struct v4l2_subdev_core_ops imx185_subdev_core_ops = {
...
};
Match the name for the pointer for the core and video operations function with the two from above, as follows:
static struct v4l2_subdev_ops imx185_subdev_ops = {
.core = &imx185_subdev_core_ops,
.video = &imx185_subdev_video_ops,
};
During the parsing of the device tree, of_device_id matches the compatible field with the one in the Device Tree.
static struct of_device_id imx185_of_match[] = {
{ .compatible = "nvidia, imx185", },
{ },
};
This structure is required for camera common and you must set up the function pointers appropriately. Link the power_on, power_off, write_reg, and read_reg functions here:
static struct camera_common_sensor_ops imx185_common_ops = {
.power_on = imx185_power_on,
.power_off = imx185_power_off,
.write_reg = imx185_write_reg,
.read_reg = imx185_read_reg,
};
Control Handlers
This topic describes the control handlers that control the top-level structure and tells the function how many controls the handler is expected to handle.
Set Control
The set-control function is used to call the desired control handler as follows.
Controls | Operation |
TEGRA_CAMERA_CID_GAIN | Updates the gain settings. Unit in ratios. See Gain Control. |
TEGRA_CAMERA_CID_FRAME_LENGTH or V4L2 CID_FRAME_RATE | Updates the frame length registers, unit in lines, or updates the frame rate settings, unit in FPS. |
TEGRA_CAMERA_CID_COARSE_TIME or TEGRA_CAMERA_CID_EXPOSURE | Updates the coarse integration time registers, unit in lines, or update exposure settings, unit in seconds. See Exposure Control. |
TEGRA_CAMERA_CID_COARSE_TIME_SHORT* | Updates the short coarse integration time registers. Only available for HDR sensors. Unit in seconds. |
TEGRA_CAMERA_CID_GROUP_HOLD* | Provides the sensor group hold (register hold) control for sensors supporting this feature. True or False. |
TEGRA_CAMERA_CID_HDR_ENABLE* | Toggles between normal and HDR sensor mode. Only available for HDR sensors. True of False. |
TEGRA_CAMERA_CID_OTP_DATA* | Reads the module One-Time Programmable (OTP) memory data. |
TEGRA_CAMERA_CID_FUSE_ID* | Reads the sensor fuse ID. |
TEGRA_CAMERA_CID_SENSOR_MODE_ID* | Used for selecting specific sensor mode. 0 based index. |
TEGRA_CAMERA_CID_VI_BYPASS_MODE** | Used for bypassing VI settings in kernel driver. Set to true by default when using user mode library. |
TEGRA_CAMERA_CID_OVERRIDE_ENABLE** | Used for overriding gain and exposure settings from default sensor mode table. |
* Optional ** Provided by NVIDIA V4L2 Media Controller Framework kernel driver. |
Gain Control
The gain control is used to control the per-frame gain setting. It takes a 32 Bit or 64 Bit input value from the user mode library.
Control | Input |
32 Bit (deprecated) | 64 Bit |
TEGRA_CAMERA_CID_GAIN | Specifies the gain values from the user mode library. It uses 24.8 format (24 Bit integer, 8 Bit fraction) as default. To change this format, adjust the min/max gain range in the driver. Use the minimum gain value to determine how many bits of a fraction to use. | Specifies the gain values from the user mode library. It uses 42.22 format (42 Bit integer, 22 Bit fraction). * See imx185_set_gain function in imx185.c file. |
Exposure Control
The exposure updated is achieved by using one of these controls:
• For 32-Bit input parameters use:
TEGRA_CAMERA_CID_FRAME_LENGTH + TEGRA_CAMERA_CID_COARSE_TIME (deprecated)
• For 64-Bit input parameters use:
TEGRA_CAMERA_CID_FRAME_RATE + TEGRA_CAMERA_CID_EXPOSURE
Note: | Use either set of controls but do not mix or use both sets at once. |
Control | Input |
TEGRA_CAMERA_CID_FRAME_LENGTH + TEGRA_CAMERA_CID_COARSE_TIME (deprecated) | Specifies the pre-processed 32 Bit value. The user mode library converts the frame rate to the sensor frame length and converts exposure to sensor coarse integration time. The driver uses these values to program the frame length and coarse integration time registers. These controls are deprecated. Use the controls below. |
TEGRA_CAMERA_CID_FRAME_RATE + V4L2 CID_EXPOSURE | Specifies the unprocessed 64 Bit value*. The user mode library passes the frame rate and exposure values to the driver without any pre-processing. the driver is responsible for converting these values into frame length and coarse integration time settings. Use this method if the sensor has special requirements that cannot be handled by the user mode library. See imx185_set_frame_rate and imx185_set_exposure functions in imx185.c file. |
|
Fixed Point Format
When the driver implements TEGRA_CAMERA_CID_FRAME_RATE and TEGRA_CAMERA_CID_EXPOSURE controls, the received control values are in Q42.22 fixed point format. In Q42.22 fixed point format the upper 42 Bits are for storing integer values and the lower 22 Bits are for fraction values. This is a workaround due to the lack of a floating point in the kernel.
This format is achieved by multiplying the original value with 1 << 22. The FIXED_POINT_SCALING_FACTOR value is defined as follows.
#define FIXED_POINT_SCALING_FACTOR (1ULL << 22)
Use these fixed-point format values with caution. The value must be divided back to the original precision for calculation. For best results, multiply first, then divide the value with the FIXED_POINT_SCALING_FACTOR last, to avoid unexpected precision loss and overflow issues.
For example, these equations can yield different results:
Gain8 = (u8) (Gain64 / FIXED_POINT_SCALING_FACTOR * 160 / 48) // incorrect, precision loss
Gain8 = (u8) (Gain64 * 160 / 48 / FIXED_POINT_SCALING_FACTOR)
// correct
V4L2 Set-Control Operation
The V4L2 set-control function contains a switch statement to redirect set-control calls to their appropriate control handlers.
Note: | Read-only controls, such as fuse_id and One-Time Programmable Read-Only Memory (OTP ROM), do not have a case statement in the control ID switch statement. |
imx185_s_ctrl {
...
switch (ctrl->id) {
case TEGRA_CAMERA_CID_GAIN:
...
case TEGRA_CAMERA_CID_EEPROM_DATA:
...
case TEGRA_CAMERA_CID_HDR_EN:
...
}
...
HDR_EN is a pure software control. No control handler writes to the hardware so simply break out of the switch statement.
Setter-Control Handlers (Writes)
Setter-control handlers are the control handlers called by s_ctrl. They perform additional control-handling operations, such as writes to registers. The majority of these controls make calls to the control register setup functions.
imx185_set_group_hold
imx185_set_gain
imx185_set_frame_length
imx185_set_coarse_time
imx185_set_coarse_time_short
imx185_write_eeprom
Get-Volatile Control
The imx185_g_volatile_ctrl function contains a switch statement for redirecting get-control calls to their appropriate control handlers.
Note: | For non-volatile controls, get-control simply returns the previously written value stored in the control handler. |
Boot-Time Initialization
The functions for initializing boot time are as follows.
Control Initialization
The imx185_ctrls_init function iterates through ctrl_config_list and registers each control as a new custom control with the V4L2 framework. s_ctrl is also called for each control to set it to its default value defined in ctrl_config_list.
For more information, see
Configuration Controls.
Modifying this function is necessary if you have calls for initializing the value for the read-only controls. See the calls to
otp_setup and
fuse_id_setup in
Control Handlers.
imx185_ctrls_init {
...
err = imx185_fuse_id_setup(priv);
...
}
Device Tree Parser
The imx185_parse_dt function, which parses device trees, takes the Device Tree node and looks for the parameters, according to the sensor-related private data, required by the sensor driver.
The values include but are not limited to:
mclk
pwdn-gpios
reset-gpios
af-gpios
avdd-reg
dvdd-reg
iovdd-reg
For details on how to set up device trees with the appropriate values, see the documentation. You must match the name list with the names in the Device Tree for their respective name-value pairs.
Media Controller Setup
A few additional components are needed to setup the media controller for V4L2 use. The open call is a placeholder operation for satisfying the V4L2 sub-device internal operation requirements. The setup likely will not need to be change besides a name changed to match the devices.
imx185_open
Sub-device function operations must link to the open operation:
static const struct v4l2_subdev_internal_ops imx185_subdev_internal_ops = {
.open = imx185_open,
};
Media entity function operations must link to the V4L2 sub-device link validation method:
static const struct media_entity_operations imx185_media_ops = {
.link_validate = v4l2_subdev_link_validate,
};
Sensor-Driver Probing
The probing function of the sensor driver are as follows.
Entry Point for Initialization During Boot
The
imx185_probe function is the entry point of the driver. The function starts off by allocating memory for common data and sensor-private data. For more information, see
Sensor-Private Data.
common_data = devm_kzalloc(&client->dev,
sizeof(struct camera_common_data), GFP_KERNEL);
priv = devm_kzalloc(&client->dev,
sizeof(struct imx185) + sizeof(struct v4l2_ctrl *) *
ARRAY_SIZE(ctrl_config_list),
GFP_KERNEL);
The function then initializes regmap:
priv->regmap = devm_regmap_init_i2c(client, &sensor_regmap_config);
Next, the function calls the other initialization functions, including:
// See the section Parser for Device Trees: stored in private
// data pdata field.
imx185_parse_dt(client);
imx185_power_get(priv);
// Link the appropriate subdev ops see
v4l2_i2c_subdev_init(&common_data->subdev, client, &subdev_ops);
imx185_ctrls_init(priv);
Next, the function links the common data and sensor-private values:
common_data->ops = & imx185_common_ops;
// Control handler linking
common_data->ctrl_handler = &priv->ctrl_handler;
// I2C client passed in to probe
common_data->i2c_client = client;
common_data->frmfmt = & imx185_frmfmt[0];
// Based on default data format definition, generally defaults
common_data->colorfmt = camera_common_find_datafmt(
IMX185_DEFAULT_DATAFMT);
// Power-handler linking
common_data->power = &priv->power;
// Sensor-private data linking
common_data->priv = (void *)priv;
common_data->numfmts = ARRAY_SIZE(imx185_frmfmt);
// Set up of port information for device
camera_common_parse_ports(...)
// Port information is also used to create the name for debugfs node setup
Setup of Default Values for Common Data
For more information, see
Macro Definitions.
common_data->def_mode = IMX185_DEFAULT_MODE;
common_data->def_width = IMX185_DEFAULT_WIDTH;
common_data->def_height = IMX185_DEFAULT_HEIGHT;
common_data->def_clk_freq = IMX185_DEFAULT_CLK_FREQ;
// I2C client passed in to probe
priv->i2c_client = client;
// Link to common data above
priv->s_data = common_data;
// Link to V4L2 subdevice handler
priv->subdev = &common_data->subdev;
// Link to subdevice dev to i2c_client dev (for media controller usage)
priv->subdev->dev = &client->dev;
Setup of Media Controller
All imager devices must register themselves as sub devices to media controller framework. VI acts as the master device which controls the binding, parsing the DT and establishes the media links once all the sub devices are registered.
Each sub-device can register to media controller framework by defining the entity and pads information.
• Entity: Media entity is the unit device represented by media controller framework for establishing connections. Each media entity can have multiple pads. Framework provides a list of known entity types and the corresponding media operations.
• Pad: Pad represents the device behaves as SINK or SOURCE port. Imager device must have a SOURCE port which is bound to VI. If imager device has a SINK port then the SOURCE port which binds the device SINK port must be represented in DT to complete the binding.
// Link subdevice internal and media entity operations
priv->subdev->internal_ops = & imx185_subdev_internal_ops;
// Setup subdevice flags and media entity type
priv->subdev->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
// Setup media controller pad flags
priv->pad.flags = MEDIA_PAD_FL_SOURCE;
priv->subdev->entity.type = MEDIA_ENT_T_V4L2_SUBDEV_SENSOR;
priv->subdev->entity.ops = & imx185_media_ops;
// Initialize and register subdevice and media entity with media controller framework.
media_entity_init(&priv->subdev->entity, 1, &priv->pad, 0);
v4l2_async_register_subdev(priv->subdev);
Removing Sensor Drivers
The imx185_remove function removes the sensor-device instance and is called on device shutdown.
This function makes calls to various free, put, and destroy functions that match up with several calls in probe. It must remain largely the same.
// Unregister v4l2 sub-device
v4l2 async unregister subdev(priv->subdev);
// Cleanup and unregister subdevice and media entity with
media controller framework.
media entity cleanup (&priv->subdev->entity);
// freeing the control handler.
v4l2_ctrl_handler free(&priv->ctrl_handler);
Kernel Configuration
After driver development is complete, add your new device information to the kernel configuration file, so it can be enabled for the kernel, as follows:
CONFIG_VIDEO imx185=y
• Android:
• TX1:
<top>/kernel/kernel-4.4/arch/arm64/configs/tegra21_android_defconfig
• TX2:
<top>/kernel/kernel-4.4/arch/arm64/configs/tegra18_android_defconfig
• Linux:
• TX1:
<top>/kernel/kernel-4.4/arch/arm64/configs/tegra21_defconfig
• TX2:
<top>/kernel/kernel-4.4/arch/arm64/configs/tegra18_defconfig
Device Registration
After driver development is complete, you must add your device information to the system kernel device tree so it can be instantiated when the kernel boots. There are two ways of registering your device.
Using Plugin Manager
If your camera module has onboard EEPROM and has a valid camera ID programmed, Plugin Manager can be used. If your device module does not meet this requirement, use the main platform device tree instead by following the directions in
Using Main Platform Device Tree File.
Plugin Manager automatically links devices when the system kernel boots. Plugin Manager uses the camera module ID returned by the boot loader to update the device tree entries with proper information at runtime. Plugin Manager allows a single system image to support multiple camera devices. To change camera modules, power down the device, replace the camera module, and then reboot. The new module works automatically.
For Plugin Manager support
1. Add your DTSI file to the camera configuration DTSI file.
2. Set the status of your device nodes to “disabled” as follows.
imx185_cam0: imx185_a@1a {
status = "disabled";
};
3. Update the camera plugin manager DTSI with proper override information.
fragment-imx185@0 {
ids = "LPRD-002001";
override@0 {
target = <&imx185_cam0>;
_overlay_ {
status = "okay";
};
};
…
};
When the kernel boots, if a specific camera board is found, the override data is applied to the specific device nodes, and they are made available for the system.
Prerequisites
To enable Plugin Manager support
1. Add your device DTSI to the camera lists in:
• TX1:
<top>/hardware/nvidia/platform/:qt210/jetson/kernel-dts/jetson-platforms/tegra210-jetson-cv-camera-modules.dtsi
• TX2:
<top>/hardware/nvidia/platform/t18x/common/kernel-dts/t18x-common-platforms/tegra186-quill-camera-modules.dtsi
2. Add the override information to:
• TX1:
<top>/hardware/nvidia/platform/t210/jetson/kernel-dts/jetson-plugin-manager/tegra210-jetson-cv-camera-plugin-manager.dtsi
• TX2:
<top>/hardware/nvidia/platform/t18x/common/kernel-dts/t18x-common-plugin-manager/tegra186-quill-camera-plugin-manager.dtsi
Using Main Platform Device Tree File
Register your new device by updating main platform DTSI file to include your new device DTSI file. Because Tegra uses Plugin Manager by default, you must first unregister Plugin Manager support, then add your device information to the main device tree DTSI file.
Prerequisites
• You have obtained the kernel source files. For more information, see
Synchronizing the Kernel in the Development Guide for the release.
To register a device using main-platform device tree files
1. Locate and edit the following file:
• TX1:
<top>/hardware/nvidia/platform/t210/jetson/kernel-dts/tegra210-jetson-cv-base-p25970-2180-a00.dts
• TX2:
<top>/hardware/nvidia/platform/t18x/quill/kernel-dts/tegra186-quill-p3310-1000-a00-00-base.dtsi
2. In the DTSI, remove the following line:
• TX1:
#include "jetson-plugin-manager/tegra210-jetson-cv-camera-plugin-manager.dtsi"
• TX2:
#include <t18x-common-plugin-manager/tegra186-quill-camera-plugin-manager.dtsi"
3. Locate and edit the following file:
• TX1:
<top>/hardware/nvidia/platform/t210/jetson/kernel-dts/tegra210-jetson-cv-base-p2597-2180-a00.dts
• TX2:
<top>/hardware/nvidia/platform/t18x/quill/kernel-dts/tegra186-quill-p3310-1000-a00-00-base.dts
4. In the DTS file, replace the following line:
• TX1:
#include "jetson-platforms/tegra210-jetson-cv-camera-modules.dtsi"
• TX2:
#include <t18x-common-platforms/tegra186-quill-camera-modules.dtsi>
With an #include statement specifying the DTSI file for your new device.
Note: | Use tegra210-jetson-cv-camera-modules.dtsi or tegra18x-quill-cv-camera-modules.dtsi as a model for generating your DTSI. In your file, change status = disable to status = okay. |
Verifying the V4L2 Sensor Driver
When the driver is complete, run v4l2-compliance and v4l2-ctl to ensure that your driver can capture raw data through the V4L2 interface.
For details on RAW memory format, see the Tegra X2 Technical Reference Manual “Video Input” topic.
To run a v4l2-compliance test
• Enter the command:
v4l2-compliance -d /dev/video0
Specify the device node path with option ‑d.
• If you have a single camera, the device path is /dev/video0.
• If you have multiple cameras, the number after video indicates the index of one of the camera available in the system. The program will use this camera for the test.
To run a v4l2-ctl test
• Enter the command:
v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG12 --stream-mmap -set-ctrl=sensor_mode=0 --stream-count=100 -d /dev/video0
You can use v4l2-ctl to capture RAW data. You must provide appropriate parameter values for the camera driver you are using.
Both v4l2-compliance and v4l2-ctl are available as open source projects. Documentation for these commands is available from LinuxTV at:
Debugging Tips
When the driver is complete, before running any camera applications, check to see if the camera device node(s) got populated correctly in the system.
If the driver is loaded properly, the following device node is displayed:
/dev/<video#>
The device node is the file I/O interface with which the user-space driver accesses the driver.
Problems may occur in the probing process and require debugging. Typically, problems occur in the clock, GPIO, and regulator setup.
To verify that driver name matches the name in the Device Tree
Your device name must be the same in both the Device Tree and your driver. This is how kernel binds the driver to your device.
In Device Tree:
compatible = "nvidia,imx185";
In driver:
static struct of_device_id imx185_of_match[] = {
{ .compatible = "nvidia,imx185", },
{ },
};
…
.driver = {
.name = "imx185",
.owner = THIS_MODULE,
.of_match_table = of_match_ptr(imx185_of_match),
},
To verify that all names match the Device Tree
Verify that the names that are being read through the parse_dt function matches those in the Device Tree.
To verify that the Device-Tree values match the hardware
Ensure that the values assigned to the Device Tree fields match the hardware they describe. Oftentimes, regulator names, GPIO numbers, and clock names are out-of-date in the Device Tree, causing probing to fail.
Use the Device Tree Compiler (dtc) to convert your .dtb file back to .dtsi format and examine the content. Ensure that the “status” field of your device node is set to the proper state.
To verify that functions run to completion
Ensure that the
power_get calls runs to completion. For details, see
Power Functions and
Boot-Time Initialization.
Another common problem that occurs during probe is in control init.
To verify that default values are correctly linked
Verify that default values are all linked correctly in the control configuration. Also, verify that the default macros are updated appropriately with the values in the mode table of the sensor.
If new controls have been added to the control configuration list, on rare occasions, ensure that they contain the appropriate control handlers and default values.
After probing succeeds, problems could still occur with the control setting. A common problem is in the values of the register writes.
To verify that control-register values are correct
Ensure that the control-register setup contains the correct register address and formats according to the mode table and data sheets. For more information, see
Setting Up Control Registers. For gain control, create a new calculate_gain (to_gain) function for calculating the gain value according to the gain formula in the datasheets.
To verify that mode-specific settings are correct
Double-check the mode-specific settings in the Device Tree and update them if necessary. The most common issues that cause sensor timeout are due to the wrong values for the following settings:
• The value for num_lanes must be 1, 2, or 4, depending on your sensor configuration.
• The tegra_sinterface setting must be set to the proper CSI port where the sensor is connected.
• The discontinuous_clk setting specifies for Tegra that the sensor is running in continuous clock mode (free running), or discontinuous clock mode (gated). If unsure, first set this value to “no”.
• Set the mclk_multiplier to equal or larger than pix_clk_hz / mclk to make sure the ISP is running fast enough to process the data from sensor.
Verify that i2c accesses are working properly
Closely examine the kernel logs for any I2C errors for the camera sensor. If there is an I2C error, check the power-on function to verify that the sensor power-on sequence is correct. The order of enabling power, clock, GPIO, and timing must comply with sensor specifications.
Mode Tables
The modes tables of the register reside in a separate header file called sensor_mode_tbls.h and are included by the main driver. Those tables can be a list of reg_8 or reg_16 address-value pairs. Mode tables are separated by resolution.
The start and stop stream-register values must be in a separate mode table. When you separate them, delete the start stream-register values at the end of the resolution mode tables.
static imx185_reg imx185_start[] = {
{0x3000, 0x00 },
{IMX185_TABLE_WAIT_MS, IMX185_WAIT_MS_START},
{0x3002, 0x00},
{0x3049, 0x02},
{IMX185_TABLE_WAIT_MS, IMX185_WAIT_MS_STREAM},
{IMX185_TABLE_END, 0x00 }
};
static imx185_reg imx185_stop[] = {
{0x3000, 0x01 },
{IMX185_TABLE_WAIT_MS, IMX185_WAIT_MS_STOP},
{IMX185_TABLE_END, 0x00 }
};
Also, a table for the color bars of the test pattern is used if the test_mode flag is enabled, activating the test-pattern mode of the sensor. That table is required only if test pattern is supported by the sensor.
static imx185_reg tp_colorbars[] = {
...
};
At the end of the header of the mode tables is an enumeration of all the mode tables, as well as a list that maps the enumeration to table pointers.
enum {
IMX185_MODE_1920X1080_CROP_30FPS,
...
};
static oimx185_reg *mode_table[] = {
[IMX185_MODE_1920X1080_CROP_30FPS] = imx185_1920x1080_crop_30fps,
...
};
The camera_common_frmfmt array list is a required table that sets up the format for the V4L2 framework. Each of the elements on that list contains the resolutions, the is_hdr flag, and the enumeration for the mode.
static const struct camera_common_frmfmt imx185_frmfmt[] = {
{{1920, 1080}, 0, IMX185_MODE_1920X1080_CROP_30FPS},
...
};
To add a register mode table
1. Obtain the address-value pairs for the mode table you would like to add.
Note: | If separate stream-on and stream-off mode tables exist, you can omit them from the mode table. |
2. Format the pairs according to the register structure and initialize a static array with the mode resolution as part of the name. For example:
static imx185_reg mode_####x####[] = {
{ 0x3000, 0x00 },
...
/* your addr and val pairs */
};
The mode table is now in place. End the table array with the following:
{ IMX185_TABLE_END, 0x00 }
3. Create a new enumeration in the list of enumerations and add it to the array of mode tables:
enum {
...
IMX185_MODE_####X####,
}
static imx185_reg *mode_table[] = {
...
[IMX185_MODE_####X####] = mode_####x####,
};
4. Add the new mode to the camera_common frmfmt array:
static const struct camera_common_frmfmt IMX185_frmfmt[] = {
...
{{####, ####}, 0, IMX185_MODE_####X####},
};